Sunday, June 24, 2012

Talking to a Bluetooth Arduino RGB Lamp from C# for Continuous Integration

I previously posted some C# code that I use to gather build status information in a Continuous Integration environment. No CI environment is complete without the Big Red Build light. In case I'm using a custom dual RGB LED lamp controlled by an Arduino.  This build light communicates through a SparkFun BlueSmirf Bluetooth adapter in it that appears as a COM port on a Windows PC after pairing. The BlueSmirf talks to the Arduino over it's RX/TX pins making it simple to communicate with on the Arduino using it's Serial libraries.  There are newer versions of the BlueSmirf that appear as HID devices for driverless communication but I still like the simplicity of the COM interface and haven't upgraded yet.

The ArduinoRGB.cs  C# class accepts a serial port as a constructor and provides a simple API for turning on the various RGB combinations along with programmable blink rates. The API supports multiple RGB lamps in a single device.  TheArduino Uno has 6 PWM ports, just enough to support 2-RGB LED lamps. I have some other devices that use PWM port expanders to support more lights and devices that use addressable LED strips. The firmware transparently supports the extra lights. I've tested 2 and 4 lamp devices with this code.

The communication protocol is all ASCII to make terminal testing easier. The device echo's every command back with a preceding '+' if it understands the command and a '-' if it doesn't.

Arduino Communication

ArduinoDualRGB.cs acts as a cover for the remote device. This code shares some of the same Java-esque characteristics of all my C# code.  It mostly passes StyleCop and Code Analysis!

///
/// Written by Joe Freeman joe@freemansoft.com
/// Arduino RGB adapter for Arduino build light firmware used for 2, 4 and 32 RGB lamp build lights.
/// 
/// Standard commands are
/// color: ~c#[red][green][blue];
/// where red, green and blue have values 0-15 representing brightness
/// blink: ~b#[red on][green on][blue on][red off][green off][blue off];
/// where on and off have values 0-15 representing the number of half seconds.
namespace BuildWatcher
{
    using System;
    using System.IO.Ports;
    using System.Text;
    using log4net;
 
    public class ArduinoDualRGB
    {
        /// <summary>
        /// log4net logger
        /// </summary>
        private static ILog log = log4net.LogManager.GetLogger(typeof(ArduinoDualRGB));
 
        /// <summary>
        /// command prefix
        /// </summary>
        private static byte standardPrefix = (byte)'~';
 
        /// <summary>
        /// last character of commands
        /// </summary>
        private static byte standardSuffix = (byte)';';
        
        /// <summary>
        /// the command to chagne a color
        /// </summary>
        private static byte colorCommand = (byte)'c';
        
        /// <summary>
        /// the command to change a blink rate
        /// </summary>
        private static byte blinkCommand = (byte)'b';
 
        /// <summary>
        /// Serial port we communicate with Arduino over
        /// </summary>
        private SerialPort device;
 
        /// <summary>
        /// Initializes a new instance of the <see cref="ArduinoDualRGB"/> class. a proxy for the Arduino controlled dual RGB unit
        /// </summary>
        /// <param name="device">Serial port the device is connected two.  Can be virtual com port for bluetooth</param>
        /// <param name="canReset">determines if the device can be reset through DTR or if is actually reset on connect</param>
        public ArduinoDualRGB(SerialPort device, bool canReset, int numLamps)
        {
            if (device == null)
            {
                throw new ArgumentNullException("device""Device is required");
            }
            else
            {
                this.device = device;
            }
 
            if (canReset)
            {
                //// can we reset with DTR like this?
                device.DtrEnable = true;
                //// the firmware starts with the string "initialized"
 
                System.Threading.Thread.Sleep(250);
                byte[] readBuffer = new byte["initialized".Length];
                for (int i = 0; i < readBuffer.Length; i++)
                {
                    readBuffer[i] = (byte)this.device.ReadByte();
                    log.Debug("read " + i);
                }
                log.Debug("Hardware initialized returned string: " + readBuffer);
            }
            else
            {
                string trashInBuffer = device.ReadExisting();
                if (trashInBuffer.Length > 0)
                {
                    log.Debug("Found some cruft left over in the channel " + trashInBuffer);
                }
            }
 
            TurnOffLights(numLamps);
        }
 
        /// <summary>
        /// Turns off the number of lamps specified
        /// </summary>
        /// <param name="numLamps">number of lamps to clear</param> 
        public void TurnOffLights(int numLamps)
        {
            for (int deviceNumber = 0; deviceNumber < numLamps; deviceNumber++)
            {
                this.SetColor(deviceNumber, 0, 0, 0);
                this.SetBlink(deviceNumber, 2, 0);
            }
        }
 
        /// <summary>
        /// sets the color of one of the lamps using RGB
        /// </summary>
        /// <param name="deviceNumber">Number of lights in a device 0-1</param>
        /// <param name="red">value of red 0-15</param>
        /// <param name="green">vlaue of green 0-15</param>
        /// <param name="blue">vlaue of 0-15</param>
        public void SetColor(int deviceNumber, int red, int green, int blue)
        {
            byte[] buffer = new byte[7];
            buffer[0] = standardPrefix;
            buffer[1] = colorCommand;
            buffer[2] = this.ConvertIntToAsciiChar(deviceNumber);
            buffer[3] = this.ConvertIntToAsciiChar(red);
            buffer[4] = this.ConvertIntToAsciiChar(green);
            buffer[5] = this.ConvertIntToAsciiChar(blue);
            buffer[6] = standardSuffix;
            this.SendAndWaitForAck(buffer);
        }
 
        /// <summary>
        /// Sets the blink rate of one of the lamps.  All bulbs in a lamp blink at the same rate and time
        /// </summary>
        /// <param name="deviceNumber">lamp number in device 0-1</param>
        /// <param name="onTimeHalfSeconds">blink on time 0-15</param>
        /// <param name="offTimeHalfSeconds">blink off time 0-15</param>
        public void SetBlink(int deviceNumber, int onTimeHalfSeconds, int offTimeHalfSeconds)
        {
            byte[] buffer = new byte[10];
            buffer[0] = standardPrefix;
            buffer[1] = blinkCommand;
            buffer[2] = this.ConvertIntToAsciiChar(deviceNumber);
            buffer[3] = this.ConvertIntToAsciiChar(onTimeHalfSeconds);
            buffer[4] = this.ConvertIntToAsciiChar(onTimeHalfSeconds);
            buffer[5] = this.ConvertIntToAsciiChar(onTimeHalfSeconds);
            buffer[6] = this.ConvertIntToAsciiChar(offTimeHalfSeconds);
            buffer[7] = this.ConvertIntToAsciiChar(offTimeHalfSeconds);
            buffer[8] = this.ConvertIntToAsciiChar(offTimeHalfSeconds);
            buffer[9] = standardSuffix;
            this.SendAndWaitForAck(buffer);
        }
 
        /// <summary>
        /// Converts a number ot it's hex ascii equivalent
        /// </summary>
        /// <param name="number">input between 0-15 </param>
        /// <returns>ASCII character Hex equivalent of the number </returns>
        public byte ConvertIntToAsciiChar(int number)
        {
            if (number < 0 || number > 15)
            {
                throw new ArgumentException("number out of single digit hex range " + number);
            }
 
            byte result;
            if (number > 9)
            {
                result = (byte)('A' + number - 10); // we start at 10
            }
            else
            {
                result = (byte)('0' + number);
            }
 
            return result;
        }
 
        /// <summary>
        /// Sends a message and waits on the return ack
        /// </summary>
        /// <param name="buffer">bytes to be sent to arduino</param>
        private void SendAndWaitForAck(byte[] buffer)
        {
            log.Debug("Sending: " + Encoding.UTF8.GetString(buffer, 0, buffer.Length));
            this.device.Write(buffer, 0, buffer.Length);
            System.Threading.Thread.Sleep(20);
            //// should handle timeout with exception catch block
            //// always replies with the command plus a + or - key.  '+' means command understood
            byte[] readBuffer = new byte[buffer.Length + 1];
            for (int i = 0; i < buffer.Length + 1; i++)
            {
                readBuffer[i] = (byte)this.device.ReadByte();
            }
 
            log.Debug("Received ack: " + Encoding.UTF8.GetString(readBuffer, 0, readBuffer.Length));
        }
    }
}

GitHub
The source is available on GitHub at https://github.com/freemansoft/build-monitors


Gathering TFS Build Information for Continuous Integration

I've moved to a new .Net based project where we use TFS as our build system. The project is pretty large with 150 developers and over 65 deployable applications, programs and NuGet packages.  The CI, systems integration and stable branch builds results in over 200 different builds on the TFS server.  I'm mostly interested in the Continuous Integration builds, 65 build,s that I'd like to monitor. I really don't want to manage 65 lights, even though that's an excuse for a new hardware project. In this case I'll treat the 65 builds as if they are a single composite build tied to a single status light.  I'll do the same with each of the branch / target types so that I have 3 sets of 65 builds to monitor.




The first step is to write some C# code that talks to TFS to get the status of the builds I'm interested in. Microsoft provides a .Net compatible library for communicating with TFS in the Microsoft.TeamFoundation set of libraries. This provides a simple interface that hides the web API. 

Our TFS server is behind a firewall tied to an Active Directory domain other than our internal domain so I must explicitly specify my credentials in the configuration file. 

We connect to the TFS server through the TfsTeamProjectCollection class which accepts the connection URL an a set of credentials.  We then use that TFS connection to get access to the BuildServer (IBuildServer) itself. A build server can have multiple teams on it each with it's own TeamProject . Each Team Project can have multiple build definitions, each with its own set of builds. In my case all my builds are on a single project in a single collection.

To recap, we retrieve our TFS information like this:
  • Connect TFS Team Project Collection using a URL 
    • Select the TFS Team Project by name
      • Create a query that finds the last two builds for all Build Definitions that m match our build definition pattern. Earlier versions first found all the Build Definitions and then queried TFS for the build results for each definition separately.  This resulted in a lot more queries but not much slower response time.
      • Send the results to the attached build light
    • Repeat for each set of builds / lamp 

GitHub

Source is available on GitHub at https://github.com/freemansoft/build-monitors Firmware sources for feedback devices is also available on github https://github.com/freemansoft/build-monitor-devices


Classes

I created a couple reusable classes to manage the data for this task.



BuildWatchDriver.cs The main driver class that reads App.config properties, configures communications and acts a run loop.

IBuildIndicatorDevice.cs A simple interface implemented by each of the physical device adapters. This is the interface to the build lights.

TfsBuildAdapter.cs This acts as a proxy for TFS including all of the utility methods and support for communicating with the TFS build server. There is one per set of builds that are monitored as a single set

TfsBuildConnection.cs This class is used to manage the network connection to TFS providing a place to store connection strings and the build server instances resulting from the authentication to the server.  One instance can be shared across TfsBuildAdapter instances as long as they run single threaded.

LastTwoBuildResults.cs A "bag" class that holds a build definition and the last two build results.  This provides a simple way of passing around and handling related data. One instance is created for each build definition in the build set.

ArduinoDualRGB.cs Driver class for an Arduino based build lamp. The firmware for the lamp is also available on GitHub.

CheapLaunchpadMSP430.cs Driver class a single lamp $10 build-light based on the TI Launchpad development board a common anode RGB LED, 3 50 ohm resistors and a gift card tin case.

Freemometer.cs Driver class for an analog gauge and LED output device.  Another blog article discusses it's construction out of an Arduino , Ikea Dekad clock and a hobby servo.

SimulatedDEvice.cs Dummy driver class that simulates a hardware device. This can be used by folks wanting to test the application who dont have any actual build light hardware.

Usage

Connection information is stored in App.Config.  The TFS URL includes the collection name so it is more than just http://domain/tfs.  We specify full credentials to allow cross domain authentication.


    <add key="Tfs.Url" value="https://tfs.yourdomain.net/tfs/yourcollection" />
    <add key="Tfs.Username" value="userid" />
    <add key="Tfs.Password" value="password" />
    <add key="Tfs.Domain" value="your_ad_domain" /> 

Connection information is stored in App.Config.  The TFS URL includes the collection name so it is more than just http://domain/tfs.  We specify full credentials to allow cross domain authentication.

Build definitions are retrieved by pattern using a query-spec.  We've spring wired the specification into our build adapters using Spring.Net.


  <!-- There is one build adapter for each light, each group of builds monitored as a group-->
  <!-- this is set up with lazy-init=true because we let the program first verify the server connection before bringing everyithing up -->
  <object id="myBuildAdapters" type="System.Collections.Generic.List&lt;BuildWatcher.Tfs.TfsBuildAdapter>" lazy-init="true" >
    <constructor-arg name="collection">
    <list element-type="BuildWatcher.Tfs.TfsBuildAdapter, BuildWatcher">
    <!--
      You should only enable as many build adapters as you have lights!
      constructor-args are
      TFS Build Server connection
      TFS Team Project name
      TFS Build definition pattern
    -->
    <!-- specified constructor argument names to make it more obvious what is going on. They are not required -->
    <object type="BuildWatcher.Tfs.TfsBuildAdapter, BuildWatcher">
      <constructor-arg name="connection" ref="myBuildServerConnection" />
      <constructor-arg name="teamProjectName" value="MSI" />
      <constructor-arg name="definitionNamePattern" value="CI_vNext*" />
    </object>
    <object type="BuildWatcher.Tfs.TfsBuildAdapter, BuildWatcher">
      <constructor-arg name="connection" ref="myBuildServerConnection" />
      <constructor-arg name="teamProjectName" value="MSI" />
      <constructor-arg name="definitionNamePattern" value="V_vNext*" />
    </object>
    <object type="BuildWatcher.Tfs.TfsBuildAdapter, BuildWatcher">
      <constructor-arg name="connection" ref="myBuildServerConnection" />
      <constructor-arg name="teamProjectName" value="MSI" />
      <constructor-arg name="definitionNamePattern" value="P_Main*" />
    </object>
    <object type="BuildWatcher.Tfs.TfsBuildAdapter, BuildWatcher">
      <constructor-arg name="connection" ref="myBuildServerConnection" />
      <constructor-arg name="teamProjectName" value="MSI" />
      <constructor-arg name="definitionNamePattern" value="V_RC*" />
    </object>
    </list>
    </constructor-arg>
  </object>


The actual query is in TfsBuildAdapter.cs using the IBuildDetailSpec.  


  IBuildDetailSpec buildDetailsQuerySpec;
  if (buildDefinitionPattern != null)
  {
      buildDetailsQuerySpec = this.Connection.BuildServer.CreateBuildDetailSpec(teamProject.Name, buildDefinitionPattern);
  }
  else
  {
    buildDetailsQuerySpec = this.Connection.BuildServer.CreateBuildDetailSpec(teamProject.Name);
  }
  //// Failure to set this property results in ALL of the build information
  //// being retrieved resulting in 10X+ call times
  //// You can retrieve subsets with something like
  //// buildDetailsQuerySpec.InformationTypes = new string[] { "ActivityTracking", "AgentScopeActivityTracking" };
  buildDetailsQuerySpec.InformationTypes = null;
  //// last and previous
  buildDetailsQuerySpec.MaxBuildsPerDefinition = 2;
  //// use start time descending because InProgress builds don't seem to sort correctly when using EndTimeDescending
  buildDetailsQuerySpec.QueryOrder = BuildQueryOrder.StartTimeDescending;
  IBuildQueryResult buildResults = this.Connection.BuildServer.QueryBuilds(buildDetailsQuerySpec);

Additional Features

The build monitor also supports a http server so that you can check the status of the builds using a web browser. This is also enabled via the App.config.  You can pick any port or trailing url you want.


<!-- ***********************************************************************
      This key causes starts a small build status web server on this port
      "+" means all hosts on all paths
      "*" means all hosts on a psecified path
      trailing "/" is required
      YOU MUST ENABLE self hosting if with this command as Administrator
      netsh http add urlacl url=http://+:8080/ user=machine\username
      You shoul disable self hosting if you don't run this app any more
      netsh http delete urlacl url=http://+:8080/
      Failure to do this will result in no web services but the app will still run
      *********************************************************************** -->
 <add key="HttpListener.ServiceUri" value="http://+:8080/" />


 


Last Edited 11/4/2012