Extreme Feedback aka Status Lights (Das Blinkenlights)

Monitoring Automated Builds
The Agile concept of continuous integration is useful no matter what the actual software development methodology.  This is especially true as teams grow, projects branch and activity increases.  We initially set up Hudson for our MAIN and FUTURE branches with one build per branch. Hudson notifies everyone about build failures via email.

That works fine but it developers eventually suffer e-mail overload so we needed a more noticeable way of telling folks when builds are broken.  (For us, broken means either compilation or unit tests fail).   We purchased two Delcom USB powered lights and mounted them on the wall where everyone can see them. A Groovy script reads the Hudson build status for our two main branches using the CruiseControl (cctray.xml) URL and then sends commands to the build lights. The cctray standard is limited and doesn't show how far along a build and doesn't differentiate between compilation and unit tests.

The Delcom lights that support RGB and blinking via commands over USB.  We currently use the older Generation 1 lights connected to one of the developer's Windows development machines. Newer 2nd Generation versions of these lights act as HID devices which means they should work on any system in a driverless fashion.   Each light shows two types of information using the color and blink/no-blink.

Color

• Green: previous build compilation and tests succeeded
• Red: previous build compilation or tests failed
• Yellow:  can't reach the CI server

Blink

• Blinking: a build is in progress.
• Solid: no buld in progress

The Delcom lights should work for most teams and I highly recommend them.  We wanted lots of status lights with a lower cost.

More Build Lights
Our development team has 35 developers operating on 3 branches across 650,00 lines of Java code plus about the same amount of Javascript, rule engine, xhtml and xml code.  We have Hudson continuous integration set up on all branches for compilation and unit test and have integration, deployment and automated UI builds.  Our full build takes about 40 minutes with unit tests.  The continuous integration builds whenever changes are made and runs all the unit tests and then deploys the main applications to verify deployment still works.  Integration test builds and Sonar metrics builds run once per day. There are over 20+ Hudson build jobs configured in the system. when you take into account all the branches and the types of builds done.

Rolling Our Own
This means we needed more lights so I prototyped a dual build light using an Arduio Uno and two RGB  Sparkfun LED breakout kits along with a wood box and two candle holders from Michael's crafts. The first picture shows the completed system with the USB cord coming out of the front in the lower left.

 

This picture shows the inside of the box.  You can see the two candle holders hot glued into the top of the box.  The two LED units are each in their own compartment in attempt to reduce bleeding from one light to another. The blue board in the bottom left is the Arduino processor.

The Arduio is a USB powered micro-controller board with a nice IDE and lots of pins to drive LEDs.  It emulates a serial port under Linux and MS Windows so almost any language can talk to it once you write the firmware for the board. The USB/Serial port defaults to 9600 baud. The Arduino has 6 PWM ports that make it simple to programatically set the brightness of the LEDs.  This lets us drive 2 RGB units or 3 RG units while still staying within the limits of the USB port. The Sparkfun LED breakout kits provide the buffering and amplification required to connect high current drain LEDs to micro-controllers without having to design any circuits or source any parts.  Each LED uses about 80mA.  The Arduino and LED kits are currently stocked at the local Microcenter, if you're lucky enough to have one nearby.


Each light requires 3 (RGB) signal connections, 5V power and Ground.  This picture shows the two RGB connections on the left side and the two power/ground connections on the right side for a total of 10 connections. I used some jumper cables I had from another project but you could solder them, make jumper cables or used headers and sockets to do the connections.

The GitHub firmware assumes the first RGB is on pins 9,10,11 and the second RGB is on 3,5,6

More RGB Lights with the PCA9635

I wanted to collapse down our 3 build light devices into a single unit so I built a lunch box build light with 4 indicators on it. This once again uses the Spark Fun LED breakout kits.  The covers are actually the central tube to a roll of large format plotter paper. I cut the tube in half, cut holes in the lunch box and then hot glued the tubes in.  The ends of the tubes are cut-up paper roll end protectors that came with the printer paper.

You can add more PWM ports using I2C expander chips like the PCA9635 which provides 16PWM outputs. The Arduino has a pretty straightforward I2C control library makes it simple to talk with the PCA9635. I used the Jee Labs Dimmer Plug which is essentially a breakout board for the PCA9635. Here is an Arduino prototyping shield with the dimmer installed on it.

Each of the lights needs a cable but the pinout on the dimmer plug means the you can't just plug 4 cables straight in. JeeArduino.

The GitHub firmware assumes the LEDs are configured on the plugs in the following RGB sets.
    {{0,1,2},{5,6,7},{8,9,10},{11,12,13},{3,4,14}}


Build Light Firmware
The Arduino is a stand alone computer that requires firmware to do what you want.  There are lots of LED examples floating around the Internet that can be used as a basis for this project. I wrote firmware that provides a tiny command interpreter that lets you set the colors, brightness and blink rate of individual LEDs. The code ramps the LED brightness to the desired levels instead of just going immediately there. This provides a smoother on and off transitions.  All commands follow this format

1.  The '~' character
2. A single command character
3. The RGB unit number, 0 or 1
4. The data for the command

The following commands are supported

• ~c#RGB;  set color    #=triplet, RGB is 0-F
• ~b#lllddd; set blink    #=triplet, lll is # # 1/2 secs lit and ddd is # of 1/2 secs dark
• ~q#;           query status #=triplet, results 3 LED x4 attrib
• ~h;             this help string

Example: 

• ~b1666111; blinks RGB LED 1 with 3 seconds on and 1/2 second off.
• ~b1600111; blinks R LED 1 with 3 seconds on and 1/2 second off

The firmware uses a PCA9635 if available and automatically falls back to the two RGB lights directly connected to the Arduino PWM ports. You can download this code from GitHub, load  it into the arduino IDE, compile and then upload to the Arduino.

From Hudson to Status Lights via Java
Here is Java program that drives these lights with data from Cruise Control or Hudson This program reads status of any system producing cc.xml file output and sends the status to various lights.  I've included drivers for the Ambient Orb, the Seeed Studio  LED Cube and an Arduino project I built.  It is pretty easy to add any Serial port based device.  The program uses rxtx as it's library.  I've only included the windows dll but it should work fine for unix machines also. The program includes the maven build and execution pom.xml.

Future Directions
We have 20+ CI builds so we'd really like to have 20+ CI lights.  Scrolling LED displays aren't exactly what we're looking for and aren't visible from all angles so were are thinking of using addressable RGB LED strips.

USB tethering in the project listed above is nice because it provides power and communications but restricts placement.  We're looking at converting to bluetooth communication.  The lights could then be AC powered and we could communicate using one of the Serial port emulating Bluetooth modules. This would make it possible to put the lights anywhere within Bluetooth range instead of being restricted by the USB cable lengths and power capabilities of the USB ports.