Sunday, December 16, 2012

Web control an addressable LED strip using ArduinoEthernet

Edited 12/18/2011: Added Perl script and made displayColors(); call fix in event handler.
Edited 12/19/2011: Added bulk update POST behavior and updated Perl script to demonstrate.
Edited 12/26/2011: Fixed quoting javascript quoting problem.
Edited 1/3/2012: Added support for Nokia 5110 style LCD and display bonjour name and IP.

Most of my build and status "blinken light" type projects are USB or Bluetooth (Serial) controlled requiring a direct connection to the machine feeding the project it's values. I wanted more standlone device that could be remotely updated. Microcenter has the ArduinoEthernet and Netduino Plus boards, both of which have onboard ethernet. I still can't get my head around the .Net MF framework and which libraries apply to the Netduino vs some other .Net board.  The RaspberryPi is probably the biggest newcomer in this space because of its aggressive pricepoint, large feature set and broader set of developer tools.   The Arduinos are primitive simpler which is what I was looking for here. I purchased the Gheo ArduinoEthernet with its embedded Wiznet W5100 Ethernet chip for $45. This may be a Microcenter price mistake because that is normally the Ethernet shield price on line.

Project Overview

This is a standalone Arduino Ethernet web Server that accepts POST information that tells it how to control a string of LED lights.  It also accepts text and positioning command to display strings on a Nokia 5110 LCD.

This requires a full Ethernet stack, a web server and the ability to do precise serial streams to talk to our 3-wire LED light strip.

I wanted to let the device use the built-in DHCP client for obtaining addresses so that I could use it on any network without a firmware update for static IP.  DHCP does make it hard for clients to find the web server since it won't be registered in DNS.  I've decided to use Bonjour as a simple network registration protocol so that clients can find the device.

Hardware

Arduino pin usage is documented in the source code.

ArduinoEthernet

The ArduinoEthernet http://arduino.cc/en/Main/ArduinoBoardEthernet provides the base platform and Ethernet components. It contains 32KB of program space and an on board Ethernet controller. The W5100 Ethernet controller is described as follows by its creator
The W5100 is a full-featured, single-chip Internet-enabled 10/100 Ethernet controller designed for embedded applications where ease of integration, stability, performance, area and system cost control are required. The W5100 has been designed to facilitate easy implementation of Internet connectivity without OS. The W5100 is IEEE 802.3 10BASE-T and 802.3u 100BASE-TX compliant. 
The W5100 includes fully hardwired, market-proven TCP/IP stack and integrated Ethernet MAC & PHY. Hardwired TCP/IP stack supports TCP, UDP, IPv4, ICMP, ARP, IGMP and PPPoE which has been proven in various applications for several years. 16Kbytes internal buffer is included for data transmission. No need of consideration for handling Ethernet Controller, but simple socket programming is required.

Pololu 3-wire 1 Meter Addressable LED Strip

This flexible LED strip uses a clock-less 3-wire interface instead of the more common clocked 4-2 wire interface. The three wires are 5V, GND and Data.  The data is essentially an asynchronous serial connection with a little over 2us bit timing.   It is like many other serial LED devices in that you have to send all of the data for each LED between the start of the strip and the one at the end of the chain. I generally update the entire strip when making a color change to any LED.   Pololu has a lot of data on the project page in addition to source code amples http://www.pololu.com/catalog/product/2540/resources.
 The timing window for this device is pretty tight so the Arduino is essentially fully occupied when updating the strip.

The strip comes with two ground lines, one power line and one signal line. I connected ground to the power source ground and to the Arduino round. I connected the 5V to an off-board power supply so that I wasn't tempted to feed 5v off the Arduino   I connected the asynchronous data line to digital pin 8 on my Arduino.

Sparkfun FTDI USB to Serial Adapter

The ArduinoEthernet does not come with USB circuitry on-board like an Uno or Leonardo.  I used an FTDI serial/USB adapter that came with my ProtoSnap kit.

Nokia 5110 Breakout Board

The Nokia style 5100 LCD display is supports 6 lines of text when using an pixel tall font.  I normally use the Sparkfun https://www.sparkfun.com/products/10168 board available at Microcenter. This demonstration uses one I got as part of a "look at the interesting and only partially documented stuff they have on DX" order. The Sparkfun one is prettier and has connectors on both ends. It's mounted here on a proto-shield I got at Microcenter. I powered the unit with 5V but only 3.3 volts to the back light LEDs. The signal lines seem to be able to handle 5V Arduino signals without any signal level adaptation.

Software

The ArduinoEthernet's on board Wiznet adapter handles the hard parts of the ethernet and includes a 16KB TX/RX buffer than lets it capture data while the Arduino is doing something else, like updating the LEDs.    The Arduino Ethernet library http://arduino.cc/en/Reference/Ethernet provides TCP/IP connectivity with DHCP address setup.  Arduino IDE 1.0.1 comes with these drivers in its library directory.  

 Webduino Web Server for Arduino

I could have talked over sockets to the ArduinoEthernet but instead decided to put small web server on the device so that any program could push data to the Arduino with a simple HTTP request.  I used the  Webduino https://github.com/sirleech/Webduino web server library because it really simplifies inbound data with a nice GET/POST processing library. I downloaded this library and copied it into the Arduino IDE library directory. 

Bonjour

Bonjour finds a simple way for dynamic IP address programs to find each other almost as if they had static IP addresses that have DNS entries. It is used find and bind printers, network storage and most Apple products. Consumers can use Bonjour to find the Arduino and connect to it without having to hard code its IP address or update DNS.

Anyone can write a program that uses Bonjour services.  Information can be found on the Apple developer website.  MAC users can see available Bonjour based network services with Safari.  Windows users can see available Bonjour services by installing the Windows Bonjour developer kit. It installs an IE browser plugin that shows you Bonjour devices on your network. 


This library is unnecessary if you use static IP address or if you wish to create a program that talks out, off the Arduino, instead of inbound, to the Arduino.

Pololu LED Strip Library

Sample AVR and Arudino libraries are available on the Pololu LED strip product page .  The strip retains its colors so the library only is invoked when colors need to be changed. This is a good thing because the library completely ties up the Arduino when updating the strip.

LED Control Program

I combined a couple programs  to create a web server that announces it self with zeroconf. It has a single web page that lets a user change the RGB values of all of the LEDs or a single LED with a single update.  The web page has Red, Green and Blue sliders and one LED selecton slider. The client side javascript calls every time a slider position changes with a little bit of time delay to reduce the number of calls that would otherwise be generated by a slider drag action.


The program updates the values on the strip in response to an HTTP POST request. The program registers a handler that reads the POST values and updates the LEDs.  The LED slider home position has a value of -1 which causes all of the LEDS to get updated with the current color.  Any other position only updates a single LED

The main loop() invokes the Bonjour and WebServer updates the routine so that they can respond to information received over the WizNet component.

Software Constraints

The Bonjour and Web server libraries only leave about 2K of programs space.  I combined the Bonjour and Webduino example program along with portions of the Pololu demo program.  My simple program doesn't do much which leaves about 1400 bytes left for future programming.


Arduino Firmware



//  This file created by joe at freemansoft inc. http://joe.blog.freemansoft.com
//  This program is written for the Arduino Ethernet http://arduino.cc/en/Main/ArduinoBoardEthernet available at microcenter
//  Note:  
//  This board uses the following pins
//      0 RX when programming/debugging
//      1 TX when programming/debugging
//      2 W5100 interrupt when bridged
//      4 SD Card SS (if not using Nokia 5110)
//      3 optional Nokia 5110 CLK
//      4 optoinal Nokia 5110 DIn
//      5 optoinal Nokia 5110 DC
//      6 optoinal Nokia 5110 Rst Reset (6 and 7 are twisted on LCD breakout board)
//      7 optoinal Nokia 5110 CE Chip Enable (6 and 7 are twisted on LCD breakout board)
//      8 LED Strip
//     10 SPI Wiznet SS
//     11 SPI
//     12 SPI
//     13 SPI
//  It demonstrates how to combine bonjour and the webduino project to create a web 
//  server that controls a TM1803 3-wire LED strip that I acquired from  http://www.pololu.com/catalog/product/2541
//  The strip used +5V, GND and signal. This program assumes the signal is on 8.   
//  Pololu has written a library available here https://github.com/pololu/pololu-led-strip-arduino
//  Tie the three grounds together, power supply, strip, arduino ethernet.
//  
//  Portions of this file originated from Arduino EthernetBonjour.
//  The updated code is available here https://github.com/neophob/EthernetBonjour
//  The original code is available here http://gkaindl.com
//
//  Portions of this file originated from the Webduino project https://github.com/sirleech/Webduino
//  Portions of this file originated from teh Sparkfun Web site supporting their Nokia LCD breakout board 
//    http://www.sparkfun.com/Code/Nokia_Example_Bitmap.pde
//
//  The web user interface demonstrates the POST API for this service.
// 
//  I had code in that logged via serial port but burned too much memory.
//
//  Host API form post can post any number of form elements in a single ost with the following conventions
//    r=  set all RED LEDs to this value
//    g=  set all GREEN LEDs to this value
//    b=  set all BLUDELEDs to this value
//    r#= set RED LED number '#' to this value
//    g#= set GREEN LED number '#' to this value
//    b#= set BLUE LED number '#' to this value
//    s#= set Nokia 5100 LCD text on line # to this value
//    c=  clear Nokia 5100 LCD value is ignored

#if defined(ARDUINO) && ARDUINO > 18
#include 
#endif
// raw arduino ethernet
#include 
// simpleconfig support using bonjour
#include 
#include 
// TM1803 67us/136us strip
#include 

// mac address make something up. odds of collision on your network are low
byte mac[] = { 0x46, 0x52, 0x45, 0x45, 0x4D, 0x4E };
// shortened the name to 12 characters so it fits on the 5110 LCD.  
// that is not required. You will want to name each device differently
#define BONJOUR_HOST_NAME "Arduino_LED"

// webduino on top of web server on port 80
// can pick a prefix but I'm not sure how you mention that prefix in the Bonjour library with the text area
// this prefix MUST match the URL in the embedded javascript
#define PREFIX "/"
WebServer webserver(PREFIX, 80);

// Create an ledStrip object on pin 8.
PololuLedStrip<8> ledStrip;
// Create a buffer for holding 60 colors.  Takes 90 bytes.
#define LED_COUNT 30
rgb_color colors[LED_COUNT];

// don't have space for the adafruit LCD libraries


/* 
 * his command is set as the default command for the server.  It
 * handles both GET and POST requests.  For a GET, it returns a simple
 * page with some buttons.  For a POST, it saves the value posted to
 * the red/green/blue variable, affecting the output of the speaker 
 *
 * this has to go before startup because it is referenced
 */
void stripCmd(WebServer &server, WebServer::ConnectionType type, char *, bool)
{
  if (type == WebServer::POST)
  {
    bool repeat;
    char name[5];     // we shortened all our post variable names to 3 digits including the LED #
    char  value[36];
    char * nameNumber;
    do
    {
      /* readPOSTparam returns false when there are no more parameters
       * to read from the input.  We pass in buffers for it to store
       * the name and value strings along with the length of those
       * buffers. */
      repeat = server.readPOSTparam(name, 6, value, 32);


      /* strcmp is a standard string comparison function.  It returns 0
       * when there's an exact match.  We're looking for a parameter
       * named "r", "g" or "b" here.  The character comparison saved 40 bytes
       *
       * These must exactly match the javascript post variable names
       */
      /* use the Ascii to Int function to turn the string
       * version of the color strength value into our unsigned char red/green/blue
       * variable 
       */
      if (name[0] == 'c'){
        LCDClear();        
      } else if (name[0]=='s' && name[1] != '\0'){
        nameNumber = &name[1];
        int row = atoi(nameNumber);
        gotoXY(0,row);
        LCDString("            ");  // bug: quick hard coded hack that knows screen width
        gotoXY(0,row);
        LCDString(value);                
      } else {
        // this exists to support the sliders in the web interface
        // just modify them all
        int valueAsInt = atoi(value);
        if (name[0]=='r' && name[1]=='\0') {
          for (int i = 0; i < LED_COUNT; i ++){
            colors[i].red = valueAsInt;
          }
        } else if (name[0]=='g' && name[1]=='\0') {
          for (int i = 0; i < LED_COUNT; i ++){
            colors[i].green = valueAsInt;
          }
        } else if (name[0]=='b' && name[1]=='\0') {
          for (int i = 0; i < LED_COUNT; i ++){
            colors[i].blue = valueAsInt;
          }
        } else {
        // this is the preferred interface if you want to set lots of lights.
        // r0,r1,r2,...,g0,g1,g2,...,b0,g1,g2,...
          nameNumber = &name[1];
          int webTarget = atoi(nameNumber);
          if (name[0] == 'r'){
            colors[webTarget].red = valueAsInt;
          } else if (name[0] == 'g'){
            colors[webTarget].green = valueAsInt;
          } else if (name[0] == 'b'){
            colors[webTarget].blue = valueAsInt;
          }
        }
      }
      
    } while (repeat);
    displayColors();
    
    // after procesing the POST data, tell the web browser to reload
    // the page using a GET method. 
    server.httpSeeOther(PREFIX);

    return;
  }

  /* for a GET or HEAD, send the standard "it's all OK headers" */
  server.httpSuccess();

  /* we don't output the body for a HEAD request */
  if (type == WebServer::GET)
  {
    /* store the HTML in program memory using the P macro */
    P(message) = 
      "\n"
        "LED Strip Controller\n"
        ""
        ""
        "\n"
        "\n"
        "\n"
      "\n"
      ""
        "
" "
" "
" "\n" "\n"; server.printP(message); } } void setup() { // start the ethernet loop Ethernet.begin(mac); // Initialize the Bonjour/MDNS library. You can now reach or ping this // Arduino via the host name "arduino_LED_strip.local", provided that your operating // system is Bonjour-enabled (such as MacOS X). // Always call this before any other method! // esentially the server name in bonjour EthernetBonjour.begin(BONJOUR_HOST_NAME); // Now let's register the service we're offering (a web service) via Bonjour! // To do so, we call the addServiceRecord() method. The first argument is the // name of our service instance and its type, separated by a dot. In this // case, the service type is _http. There are many other service types, use // google to look up some common ones, but you can also invent your own // service type, like _mycoolservice - As long as your clients know what to // look for, you're good to go. // The second argument is the port on which the service is running. This is // port 80 here, the standard HTTP port. // The last argument is the protocol type of the service, either TCP or UDP. // Of course, our service is a TCP service. // With the service registered, it will show up in a Bonjour-enabled web // browser. As an example, if you are using Apple's Safari, you will now see // the service under Bookmarks -> Bonjour (Provided that you have enabled // Bonjour in the "Bookmarks" preferences in Safari). EthernetBonjour.addServiceRecord("Arduino Adressable LED Strip._http", 80, MDNSServiceTCP); // register default command on the discoverable url http://a.b.c.d/strip_cmd webserver.setDefaultCommand(&stripCmd); // start the webduino processing webserver.begin(); // display the initial set of colors displayColors(); // space padding not required but it reminds me we have 12 characters per line LCDInit(); //Init the LCD LCDClear(); LCDString("Bonjour Name"); gotoXY(0,1); // clear reset to the home position LCDString(BONJOUR_HOST_NAME); gotoXY(0,3); LCDString("Server IP"); gotoXY(0,4); char buf[4]; for (byte thisByte = 0; thisByte < 4; thisByte++) { if (thisByte != 0){ LCDString("."); } // print the value of each byte of the IP address: itoa(Ethernet.localIP()[thisByte],buf,10); LCDString(buf); } } void loop() { // This actually runs the Bonjour module. // YOU HAVE TO CALL THIS PERIODICALLY, OR NOTHING WILL WORK! // Preferably, call it once per loop(). EthernetBonjour.run(); // start the webduino component. // YOU HAVE TO CALL THIS PERIODICALLY, OR NOTHING WILL WORK! // Preferably, call it once per loop(). webserver.processConnection(); // show colors based the current values // no need to do this because the handler does it now //displayColors(); } void displayColors(){ // Write the colors to the LED strip. // could be dupe copy to LEDs if was set to NO_TARGET ledStrip.write(colors, LED_COUNT); } /*********************************************************** 7-17-2011 Spark Fun Electronics 2011 Nathan Seidle This code is public domain but you buy me a beer if you use this and we meet someday (Beerware license). This code writes a series of images and text to the Nokia 5110 84x48 graphic LCD: http://www.sparkfun.com/products/10168 Do not drive the backlight with 5V. It will smoke. However, the backlight on the LCD seems to be happy with direct drive from the 3.3V regulator. Although the PCD8544 controller datasheet recommends 3.3V, the graphic Nokia 5110 LCD can run at 3.3V or 5V. No resistors needed on the signal lines. You will need 5 signal lines to connect to the LCD, 3.3 or 5V for power, 3.3V for LED backlight, and 1 for ground. ***********************************************************/ #define PIN_SCE 7 //Pin 3 on LCD #define PIN_RESET 6 //Pin 4 on LCD #define PIN_DC 5 //Pin 5 on LCD #define PIN_SDIN 4 //Pin 6 on LCD #define PIN_SCLK 3 //Pin 7 on LCD //The DC pin tells the LCD if we are sending a command or data #define LCD_COMMAND 0 #define LCD_DATA 1 //You may find a different size screen, but this one is 84 by 48 pixels #define LCD_X 84 #define LCD_Y 48 //This table contains the hex values that represent pixels //for a font that is 5 pixels wide and 8 pixels high static const byte ASCII[][5] = { {0x00, 0x00, 0x00, 0x00, 0x00} // 20 ,{0x00, 0x00, 0x5f, 0x00, 0x00} // 21 ! ,{0x00, 0x07, 0x00, 0x07, 0x00} // 22 " ,{0x14, 0x7f, 0x14, 0x7f, 0x14} // 23 # ,{0x24, 0x2a, 0x7f, 0x2a, 0x12} // 24 $ ,{0x23, 0x13, 0x08, 0x64, 0x62} // 25 % ,{0x36, 0x49, 0x55, 0x22, 0x50} // 26 & ,{0x00, 0x05, 0x03, 0x00, 0x00} // 27 ' ,{0x00, 0x1c, 0x22, 0x41, 0x00} // 28 ( ,{0x00, 0x41, 0x22, 0x1c, 0x00} // 29 ) ,{0x14, 0x08, 0x3e, 0x08, 0x14} // 2a * ,{0x08, 0x08, 0x3e, 0x08, 0x08} // 2b + ,{0x00, 0x50, 0x30, 0x00, 0x00} // 2c , ,{0x08, 0x08, 0x08, 0x08, 0x08} // 2d - ,{0x00, 0x60, 0x60, 0x00, 0x00} // 2e . ,{0x20, 0x10, 0x08, 0x04, 0x02} // 2f / ,{0x3e, 0x51, 0x49, 0x45, 0x3e} // 30 0 ,{0x00, 0x42, 0x7f, 0x40, 0x00} // 31 1 ,{0x42, 0x61, 0x51, 0x49, 0x46} // 32 2 ,{0x21, 0x41, 0x45, 0x4b, 0x31} // 33 3 ,{0x18, 0x14, 0x12, 0x7f, 0x10} // 34 4 ,{0x27, 0x45, 0x45, 0x45, 0x39} // 35 5 ,{0x3c, 0x4a, 0x49, 0x49, 0x30} // 36 6 ,{0x01, 0x71, 0x09, 0x05, 0x03} // 37 7 ,{0x36, 0x49, 0x49, 0x49, 0x36} // 38 8 ,{0x06, 0x49, 0x49, 0x29, 0x1e} // 39 9 ,{0x00, 0x36, 0x36, 0x00, 0x00} // 3a : ,{0x00, 0x56, 0x36, 0x00, 0x00} // 3b ; ,{0x08, 0x14, 0x22, 0x41, 0x00} // 3c < ,{0x14, 0x14, 0x14, 0x14, 0x14} // 3d = ,{0x00, 0x41, 0x22, 0x14, 0x08} // 3e > ,{0x02, 0x01, 0x51, 0x09, 0x06} // 3f ? ,{0x32, 0x49, 0x79, 0x41, 0x3e} // 40 @ ,{0x7e, 0x11, 0x11, 0x11, 0x7e} // 41 A ,{0x7f, 0x49, 0x49, 0x49, 0x36} // 42 B ,{0x3e, 0x41, 0x41, 0x41, 0x22} // 43 C ,{0x7f, 0x41, 0x41, 0x22, 0x1c} // 44 D ,{0x7f, 0x49, 0x49, 0x49, 0x41} // 45 E ,{0x7f, 0x09, 0x09, 0x09, 0x01} // 46 F ,{0x3e, 0x41, 0x49, 0x49, 0x7a} // 47 G ,{0x7f, 0x08, 0x08, 0x08, 0x7f} // 48 H ,{0x00, 0x41, 0x7f, 0x41, 0x00} // 49 I ,{0x20, 0x40, 0x41, 0x3f, 0x01} // 4a J ,{0x7f, 0x08, 0x14, 0x22, 0x41} // 4b K ,{0x7f, 0x40, 0x40, 0x40, 0x40} // 4c L ,{0x7f, 0x02, 0x0c, 0x02, 0x7f} // 4d M ,{0x7f, 0x04, 0x08, 0x10, 0x7f} // 4e N ,{0x3e, 0x41, 0x41, 0x41, 0x3e} // 4f O ,{0x7f, 0x09, 0x09, 0x09, 0x06} // 50 P ,{0x3e, 0x41, 0x51, 0x21, 0x5e} // 51 Q ,{0x7f, 0x09, 0x19, 0x29, 0x46} // 52 R ,{0x46, 0x49, 0x49, 0x49, 0x31} // 53 S ,{0x01, 0x01, 0x7f, 0x01, 0x01} // 54 T ,{0x3f, 0x40, 0x40, 0x40, 0x3f} // 55 U ,{0x1f, 0x20, 0x40, 0x20, 0x1f} // 56 V ,{0x3f, 0x40, 0x38, 0x40, 0x3f} // 57 W ,{0x63, 0x14, 0x08, 0x14, 0x63} // 58 X ,{0x07, 0x08, 0x70, 0x08, 0x07} // 59 Y ,{0x61, 0x51, 0x49, 0x45, 0x43} // 5a Z ,{0x00, 0x7f, 0x41, 0x41, 0x00} // 5b [ ,{0x02, 0x04, 0x08, 0x10, 0x20} // 5c \\ escape the backslash or a row vanishes ,{0x00, 0x41, 0x41, 0x7f, 0x00} // 5d ] ,{0x04, 0x02, 0x01, 0x02, 0x04} // 5e ^ ,{0x40, 0x40, 0x40, 0x40, 0x40} // 5f _ ,{0x00, 0x01, 0x02, 0x04, 0x00} // 60 ` ,{0x20, 0x54, 0x54, 0x54, 0x78} // 61 a ,{0x7f, 0x48, 0x44, 0x44, 0x38} // 62 b ,{0x38, 0x44, 0x44, 0x44, 0x20} // 63 c ,{0x38, 0x44, 0x44, 0x48, 0x7f} // 64 d ,{0x38, 0x54, 0x54, 0x54, 0x18} // 65 e ,{0x08, 0x7e, 0x09, 0x01, 0x02} // 66 f ,{0x0c, 0x52, 0x52, 0x52, 0x3e} // 67 g ,{0x7f, 0x08, 0x04, 0x04, 0x78} // 68 h ,{0x00, 0x44, 0x7d, 0x40, 0x00} // 69 i ,{0x20, 0x40, 0x44, 0x3d, 0x00} // 6a j ,{0x7f, 0x10, 0x28, 0x44, 0x00} // 6b k ,{0x00, 0x41, 0x7f, 0x40, 0x00} // 6c l ,{0x7c, 0x04, 0x18, 0x04, 0x78} // 6d m ,{0x7c, 0x08, 0x04, 0x04, 0x78} // 6e n ,{0x38, 0x44, 0x44, 0x44, 0x38} // 6f o ,{0x7c, 0x14, 0x14, 0x14, 0x08} // 70 p ,{0x08, 0x14, 0x14, 0x18, 0x7c} // 71 q ,{0x7c, 0x08, 0x04, 0x04, 0x08} // 72 r ,{0x48, 0x54, 0x54, 0x54, 0x20} // 73 s ,{0x04, 0x3f, 0x44, 0x40, 0x20} // 74 t ,{0x3c, 0x40, 0x40, 0x20, 0x7c} // 75 u ,{0x1c, 0x20, 0x40, 0x20, 0x1c} // 76 v ,{0x3c, 0x40, 0x30, 0x40, 0x3c} // 77 w ,{0x44, 0x28, 0x10, 0x28, 0x44} // 78 x ,{0x0c, 0x50, 0x50, 0x50, 0x3c} // 79 y ,{0x44, 0x64, 0x54, 0x4c, 0x44} // 7a z ,{0x00, 0x08, 0x36, 0x41, 0x00} // 7b { ,{0x00, 0x00, 0x7f, 0x00, 0x00} // 7c | ,{0x00, 0x41, 0x36, 0x08, 0x00} // 7d } ,{0x10, 0x08, 0x08, 0x10, 0x08} // 7e ~ ,{0x78, 0x46, 0x41, 0x46, 0x78} // 7f DEL }; void gotoXY(int x, int y) { LCDWrite(0, 0x80 | x); // Column. LCDWrite(0, 0x40 | y); // Row. ? } //This function takes in a character, looks it up in the font table/array //And writes it to the screen //Each character is 8 bits tall and 5 bits wide. We pad one blank column of //pixels on each side of the character for readability. void LCDCharacter(char character) { LCDWrite(LCD_DATA, 0x00); //Blank vertical line padding for (int index = 0 ; index < 5 ; index++) LCDWrite(LCD_DATA, ASCII[character - 0x20][index]); //0x20 is the ASCII character for Space (' '). The font table starts with this character LCDWrite(LCD_DATA, 0x00); //Blank vertical line inter-character padding } //Given a string of characters, one by one is passed to the LCD void LCDString(char *characters) { while (*characters) LCDCharacter(*characters++); } //Clears the LCD by writing zeros to the entire screen void LCDClear(void) { for (int index = 0 ; index < (LCD_X * LCD_Y / 8) ; index++) LCDWrite(LCD_DATA, 0x00); gotoXY(0, 0); //After we clear the display, return to the home position } //This sends the magical commands to the PCD8544 void LCDInit(void) { //Configure control pins pinMode(PIN_SCE, OUTPUT); pinMode(PIN_RESET, OUTPUT); pinMode(PIN_DC, OUTPUT); pinMode(PIN_SDIN, OUTPUT); pinMode(PIN_SCLK, OUTPUT); //Reset the LCD to a known state digitalWrite(PIN_RESET, LOW); digitalWrite(PIN_RESET, HIGH); LCDWrite(LCD_COMMAND, 0x21); //Tell LCD that extended commands follow LCDWrite(LCD_COMMAND, 0xB1); //Set LCD Vop (Contrast): Try 0xB1(good @ 3.3V) or 0xBF if your display is too dark LCDWrite(LCD_COMMAND, 0x04); //Set Temp coefficent LCDWrite(LCD_COMMAND, 0x14); //LCD bias mode 1:48: Try 0x13 or 0x14 LCDWrite(LCD_COMMAND, 0x20); //We must send 0x20 before modifying the display control mode LCDWrite(LCD_COMMAND, 0x0C); //Set display control, normal mode. 0x0D for inverse } //There are two memory banks in the LCD, data/RAM and commands. This //function sets the DC pin high or low depending, and then sends //the data byte void LCDWrite(byte data_or_command, byte data) { digitalWrite(PIN_DC, data_or_command); //Tell the LCD that we are writing either to data or a command //Send the data digitalWrite(PIN_SCE, LOW); shiftOut(PIN_SDIN, PIN_SCLK, MSBFIRST, data); digitalWrite(PIN_SCE, HIGH); }

Driving the Device

The firmware supports two different HTTP POST models.

  1. The single/all LED control as demonstrated by the web interface.
    1. There are 4 fields, 'r', 'g', 'b' and 'l' where 'l' is the led number.  LED = -1 means change all the LEDs to the r/g/b value.  LED = 0-29 means change that individual LED to the RGB value.
  2. An arbitrary number LEDs changed in a single post.  This is demonstrated in the PERL program.
    1. POST parameters are 'r#', 'g#' and 'b#" where the # is the LED number from 0-29.  This lets you individually set each channel for each LED.  Values not modified in the POST are left as they were.  

Sample Perl Code

# cpanm Net::Bonjour not needed  because we know the name
use HTTP::Request::Common;
use LWP;
use Time::HiRes; qw(usleep);

$ua = LWP::UserAgent->new;
#the Bonjour name we gave it in the arduino code
$url = 'http://arduino_led.local/';
$num_leds = 30;
# blank the lights using the "all led" command
$ua->post($url,
 [ r  => '0',
  g => '0',
  b => '0',
  l => '-1',
  c => 'NA'
 ],
); 

# now turn all of them on white using the bulk set command
# post accepts array ref or hash ref so lets create a hash ref
print "bam!  do them all in a single call (4 times)\n";
my $post_hash  ;
for ($color=3;$color>=0;$color--) {
 $post_hash{"s5"} = "Updating All";
 $actual_color=$color*64;
 for ($count = 0 ; $count < $num_leds; $count++){
  $post_hash{"r$count"} = $actual_color;
  $post_hash{"g$count"} = $actual_color;
  $post_hash{"b$count"} = $actual_color;
 }
 $ua->post($url,\%post_hash);
}

# turn on the lights one per time using the original single led command
# this is a LOT slower
# examples all use 
#  $ua->post($url,
#   [ r  => '0',
#    g => '0',
#    b => $actual_color,
#    l => $three
#   ],
#  ); 

# demo one led per post
for ($color=3;$color>=0;$color--) {
 $actual_color=$color*64;
 $i = 0;
 while ($i < $num_leds) {
  $one = $i+0;
  $two = $i+1;
  $three = $i+2;
  print "doing $one - $actual_color\n";
  %{$post_hash} = ();
  $post_hash{"s5"} = "Updating $one";
  $post_hash{"r$one"} = $actual_color;
  $post_hash{"g$one"} = '0';
  $post_hash{"b$one"} = '0';
  $ua->post($url,\%post_hash);
  print "doing $two - $actual_color\n";
  %{$post_hash} = ();
  $post_hash{"s5"} = "Updating $two";
  $post_hash{"r$two"} = '0';
  $post_hash{"g$two"} =  $actual_color;
  $post_hash{"b$two"} = '0';
  $ua->post($url,\%post_hash);
  print "doing $three - $actual_color\n";
  %{$post_hash} = ();
  $post_hash{"s5"} = "Updating $three";
  $post_hash{"r$three"} = '0';
  $post_hash{"g$three"} = '0';
  $post_hash{"b$three"} = $actual_color;
  $ua->post($url,\%post_hash);
  $i=$i+3;
 };
}

TFS Build Watcher 

I've updated the TFS Build Watcher to support this device to show build status using both the LED strip and the optional LED display.

Source code is available on github at https://github.com/freemansoft/build-monitors/tree/master/build-lights-net









Thanks for reading...

Sunday, December 9, 2012

Dell First Gen SSD vs Vertex 2 in older Core 2 Duo Laptop

A recent Microcenter sale on Dell g914j 50GB first generation Server SSDs seemed like a good deal at $35.  I found it that it gives a nice performance boost to a core 2 duo system.

Some folks claimed it wasn't worth the money because of poor performance vs current drives but I was looking for a cheap "pick me up" for a Dell D630, 4GB Core2Duo system. I benchmarked a Vertex 2 80GB against the Dell.

Here is the Vertex2

Here is the Dell which is actually made by Samsung
The Dell/Samsung has a 25% lower transfer rate but a 30% higher burst rate.  

Saturday, December 1, 2012

Using the Amped UA2000 Directional Antenna in a Noisy Office Environment

Wifi hotspots have changed the way people connect to the Internet with their own devices during the work day.  Many companies have restrictive Internet policies to protect the corporate network and to manage employee productivity.  Wi-fi hotspots are andy in these situations providing connectivity for  personal laptops and tablets. This also applies to contractors who need certain kinds of external connectivity when working with their laptops on-site



I work in a building where you can only get 3G or 4G reception when you are within 5 feed of the windows with "good" reception available only at certain locations. The closest spot is about 50 feed from my cube.  Hotspots are great because I can put my $35/mo Virgin Mobile Overdrive at the window and use my laptop wifi to communicate with it.  I get about 2Mbps-3-Mbs when wi-fi to the Overdrive in 4G mode.





This works great until about 9:00 AM when everyone comes in with their laptops, phones and hotspots.  My computer then loses and regains connectivity every couple minutes.  This make sense with the omnidirectional nature of my laptop antenna and the number of transmitting devices on overlapping and directly conflicting channels.




The Amped UA2000 USB Wi-Fi adapter has a directional antenna that increases the signal to noise ratio of the connection between your computer and the wi-fi hub (hotspot).  It also reduces the strength of the interfering signals from the hotspots that are out of line between your UA2000 and the hotspot.  Today I got 0.9Mbps with regular wifi and 5.8Mbs with the UA2000 in a quick speedtest.

UA2000 USB Connection

The UA2000 comes with two connectors. One is for communication and the other provides extra power to "Amp up" the adapter.  I want to improve my connection without making everyone else's worse so I only use the data connection, leaving the "Amped" connection unplugged. I achieved these results without using the "Amped" connector.

Windows Support

I had no problem using the UA2000 with Windows 7 under bootcamp on my MacBook using the drivers from the web site. Windows 7 can only support one wi-fi adapter at a time. You have to disable all other wi-fi adapters when using the UA2000.  You then have to re-enable the standard wi-fi adapter when you unplug the UA2000.

Mac OS/X 10.8 Support

Amped has OS/X 10.7 drivers on their web site that they say are valid for 10.8 but I was unable to get them working with Mac OS/X 10.8.  The UA2000 uses the RALinkTech RT2870 USB Wifi chipset.  I was able to get the UA2000 working with drivers available on the RaLinkTech web site. Make sure you install the drivers before plugging the UA2000 into the Mac.  The Amped utility is a customized version of the OEM utility so the screens and instructions are identical.


Cutting the TI Launchpad to Stuff it in a Smaller Case

I've been giving away TI Launchpad Blinken-Light projects to some of my software-only friends so they can have something new to play with.  The projects are basically a TI-Launchpad, an RGB LED and a USB cable all mounted in some kind of case.  I've burned the simple firmware available on GitHub so they can flash the LED over the serial port.


Gamestop a Nintendo DS gift card tin. The TI Launchpad mounts fine in the tin if you tern it sideways but you have to shorten the board if you want the USB connector to come out the back.

I used a Dremel cutting wheel to cut the board about one pin spacing below the last header pins. None of that circuitry is required to program or run the board.  That cuts off the reset switch, the two LEDs and the extra Vcc and GND pins.











Here is the cutoff board mounted in a Gamestop DS Gift Card tin.  I mounted the 10mm RGB LED to the case and drilled a hole in the top. The case is pretty thin so it is pretty easy to scratch the finish or dent metal in some hard to fix way.

Three small LEDs front mounted in the base would have been nicer.  Another neat trick would be to mount one of the little LCD screens in the location where the Nintendo DS screen would normally go.




Other Cases



I built most of my gift cases in the PS/2 gift card tin available from GameStop for $0.99 on closeout. It is big enough to hold the board without cutting where you can put the USB connector on the bottom tier under the overhang.


I used the PS/2 case originall because it was cheap and because I was afraid of cutting the board.  The PS/2 case is actually made up of three metal boxes. I had to cut through two of them to create enough room for the board. The metal work is a little sloppy but it's all hidden when the cover is put back on.  

It is a lot less work to just cut the board and use a simpler case.












The Launchpad will also mount sideways in an Altoids tin. You can cut a hole in the back corner of the case for the USB connector if you wish to power or control it that way.

Mounting to the Case


The Launchpad  doesn't have any mounting holes.  I normally just hot-glue the board to the case.  The fit is tight enough that a blob of glue on each corner will hold it in place well enough for many cable insertions and removals.

You can create a single mounting hole by drilling through the TI Launchpad rocket logo between the two switches. You can see it in the center top of the board in this picture. Some folks also drill a hole where the TI logo used to be on version 1.4 and earlier boards.  I haven't seen a PCB layout of the 1.5 boards.  They changed the way RX/TX are connected to the jumbers on those and that space may no longer be available.

An "Inexpensive" Gift

The whole package runs about $7.00 with the Altoids tin, $8.00 with the PS/2 case and $10.00 with one of the other tins. That makes it a pretty cheap unusual gift.  Of course it takes me about an hour to solder cut, re-cut, re-solder and fiddle with them to get them the way I way I want them with my limited manual skills.  Plastic cases are a lot easier to work with.

Sunday, November 25, 2012

Adding cheap LCD display and bluetooth shields to a Netduino

I purchased a cheap LCD keypad shield from DX that makes it simple to add a 16x2 LCD display to an Arduino. The board also includes a set of buttons that are connected to a single analog pin through a resistor ladder network.  It's a pretty slick way of minimizing the pin count.  The board is designed for 5V.  The Netduino is a 3.3v system.

The buttons resistor network is wired to the 5V supply so most of the buttons analog values are out of bounds for the 3.3V input.  Others have modified the board to use the 3.3v rail or dropped 5V to 3.3V through a resistor. I didn't do any of this because I don't need the buttons.

The LCD uses digital pins D4-D9 and can operate with 3.3V signals.  The Netduino MF libraries support the on-board devices but not external chips or interfaces.  The LCD Controller is a standard HD44780 which is supported by nice netmftoolbox supplemental library. I downloaded the netmftoolbox library from codeplex and unpacked into a 3rd party directory I keep on my dev machine. I then added two netmftoolbox DLLs to my Visual Studio project using the instructions on the netmftoolbox wiki.

Note the mounting hole offset on the LCD shielddue to the offset headers on the Bluetooth shield.

This program builds on the Bluetooth Shield code from a previous post. It assumes you have connected to the Netduino over Bluetooth Serial and can send commands to the Bluetooth module using something like putty.exe on a Windows machine.  Mac/linux users can use their built in tools.  The program echos anything sent to the LCD and back to the sending machine.  Note that the HD44780 thinks it has a 40 character display no matter what you tell the library so it will fill the display memory for columns 0-39 before wrapping to the next line.  The code supports a couple special characters to move the cursor to the beginning of the first line or the second line so that you don't have to blindly type off screen.

using System;
using System.Text;
using System.Threading;
using Microsoft.SPOT;
using Microsoft.SPOT.Hardware;
using SecretLabs.NETMF.Hardware;
using SecretLabs.NETMF.Hardware.Netduino;
using System.IO.Ports;
using Toolbox.NETMF.Hardware;

///
/// This code is described at http://joe.blog.freemansoft.com
/// Serial code was derived from http://blog.codeblack.nl/post/NetDuino-Getting-Started-with-RS232.aspx
/// 
/// This uses uses netmftoolbox available on codeplex.  It is a pretty slick contributed library.
/// Documentation on how to use it is available at http://netmftoolbox.codeplex.com/
/// http://netmftoolbox.codeplex.com/wikipage?title=How%20to%20use%20the%20classes%3f&referringTitle=Documentation
/// 
/// The Hd44780LCD controller actually has a 40 character wide buffer so we will just write 
/// into the extra space if we write off the edge of the 16character LCD.  So line overflow 
/// doesen't wrap to display boundry. A real program would know what it was writing and where cursor was.
/// 

namespace Netduino_COM2_Echo
{
    public class Program
    {
        static SerialPort serial;
        static Hd44780Lcd display;

        public static void Main()
        {
            initialize_SerialPort();
            initialize_LcdPanel();
            tellUser_WeAreReady();

            // we can toggle the LED including Sleep() because we use an event handler to handle input
            OutputPort led = new OutputPort(Pins.ONBOARD_LED, false);
            while (true)
            {
                led.Write(true);
                Thread.Sleep(250);
                led.Write(false);
                Thread.Sleep(250);
            }
        }

        static void serial_DataReceived(object sender, SerialDataReceivedEventArgs e)
        {
            // create a single byte array
            byte[] bytes = new byte[1];

            // as long as there is data waiting to be read
            while (serial.BytesToRead > 0)
            {
                // read a single byte
                serial.Read(bytes, 0, bytes.Length);
                // send the same byte back
                serial.Write(bytes, 0, bytes.Length);
                // just some kludgy code to show cursor management
                // there isn't any way to know where the cursor currently is
                if (bytes[0] == 0xd)
                {
                    // lets just use CR to go to the home position
                    display.ChangePosition(0, 0);
                }
                else if (bytes[0] == 0xa)
                {
                    // lets use LF to go to 2nd line home position
                    display.ChangePosition(1, 0);
                }
                else if (bytes[0] == 0xc)
                {
                    display.ClearDisplay();
                }
                else
                {
                    // display on LCD
                    display.Write(bytes);
                }
            }
        }

        static void initialize_SerialPort()
        {
            // initialize the serial port for COM2 (using D2 & D3)
            serial = new SerialPort(SerialPorts.COM2, 9600, Parity.None, 8, StopBits.One);
            // open the serial-port, so we can send & receive data
            serial.Open();
            // add an event-handler for handling incoming data
            serial.DataReceived += new SerialDataReceivedEventHandler(serial_DataReceived);

        }

        /// 
        /// Initialize the display shield
        /// 
        static void initialize_LcdPanel()
        {
            // default width is 16 but controller still acts like 40 char buffer
            // setting the display width won't cause it to wrap at 16 characters
            display= new Hd44780Lcd(
                Data4: Pins.GPIO_PIN_D4,
                Data5: Pins.GPIO_PIN_D5,
                Data6: Pins.GPIO_PIN_D6,
                Data7: Pins.GPIO_PIN_D7,
                RegisterSelectPin: Pins.GPIO_PIN_D8,
                ClockEnablePin: Pins.GPIO_PIN_D9
            );
            display.ClearDisplay();
        }

        /// 
        /// Dumb method that initializes the display in some "friendly way"
        /// 
        static void tellUser_WeAreReady() 
        {
            string greeting = "Type Here To See Text\n\r";
            byte[] greetingBytes = Encoding.UTF8.GetBytes(greeting);
            // Wait a little bet to let the bluetooth connection setup 
            // -- ok this magic worked so I'm keeping it!
            Thread.Sleep(250);
            serial.Write(greetingBytes, 0, greetingBytes.Length);
            display.Write(greeting);
            display.ChangePosition(1, 0);

        }
    }
}

Thanks for reading...

Saturday, November 24, 2012

Using a Cheap Bluetooth Shield on the Netduino

I found this $19 Bluetooth Shield on DX.com from Elec Freaks.  They claim it is Arduino compatible but the Wiki and spec sheet both say 3.3v only, in bold,  which means it doesn't work with a standard 5V Arduino. Netduino boards use Arduino shields and have 3.3V signal levels.

The shield comes with an HC-05 Bluetooth module which can act as either master or slave. You can set the mode by sending it AT style commands after configuring the two switches to set it to command mode.   It is a little simpler than the "you have x seconds from bootup to configure" that some other devices have. My board arrived in Slave Mode which works with a PC acting as Master.

The module has an on-board voltage regulator and a set of daughter card connectors to add additional shields. They used offset top and bottom connectors rather than a single pass through so the card extends off one side.  I'd probably stack this board on top of any other shields I was using.



A basic Netduino comes with two hardware COM ports.  The COM port attached to the USB connector is used for programming and debugging, leaving one available for any purpose.  That means you can't power your Netduino and communicate with it in your program "out of the box" unless you give up debug capability and console output.  The Netduino team does support swapping the two COM ports but it seems like a hassle on the forums. It would be nicer if they supported multiple channels on the USB connector like the TI Launchpad does.  My particular project is unaffected because I intend to power the device over the DC jack and communicate with it over Bluetooth. The USB port will only be used for development.
  • COM1 D0/D1 connected to the USB connector and used for debugging and programming.  Using this in a program means you have no way to control or debug the device.
  • COM2 D2/D3 usually available for an FTDI or other serial breakout board. I jumper'd the Bluetooth shield to COM2.
The unit was simple to pair with my Dell laptop using the pin 1234 as specified on the Wiki.  The Windows 7 Bluetooth stack installed two virtual COM ports, one Outgoing and one Incoming.  I was able to use putty.exe, under windows, to communicate with the device using the Outgoing port.  The Wiki says the Bluetooth shield defaults to 38400.  Mine defaulted to 9600 bps.

I lightly modified a Netduino program provided in this blog that shows you how to do serial communication with a Netduino.  My changes were mainly to use the 2nd COM port and to flash the LED to show that the program is running.

The program sets up communication on COM2 (D2/D3) at 9600 bits per second. It then prints out a welcome message and echos anything you send to the Bluetooth module from your computer over the virtual serial port.

using System;
using System.Text;
using System.Threading;
using Microsoft.SPOT;
using Microsoft.SPOT.Hardware;
using SecretLabs.NETMF.Hardware;
using SecretLabs.NETMF.Hardware.Netduino;
using System.IO.Ports;

namespace Netduino_COM2_Echo
{
    public class Program
    {
        static SerialPort serial;
 
        public static void Main()
        {
            string greeting = "Everything you type in this window will be echoed\n\r";
            byte[] greetingBytes = Encoding.UTF8.GetBytes(greeting);
            // initialize the serial port for COM2 (using D2 & D3)
            serial = new SerialPort(SerialPorts.COM2, 9600, Parity.None, 8, StopBits.One);
            // open the serial-port, so we can send & receive data
            serial.Open();
            // add an event-handler for handling incoming data
            serial.DataReceived += new SerialDataReceivedEventHandler(serial_DataReceived);

            // Wait a little bet to let the bluetooth connection setup 
            // -- ok this magic worked so I'm keeping it!
            Thread.Sleep(250);
            serial.Write(greetingBytes, 0, greetingBytes.Length);

            // we can toggle the LED including Sleep() because we use a multi threaded event handler
            OutputPort led = new OutputPort(Pins.ONBOARD_LED, false);
            while (true)
            {
                led.Write(true);
                Thread.Sleep(250);
                led.Write(false);
                Thread.Sleep(250);
            }

            // original example code said wait forever...
            // Thread.Sleep(Timeout.Infinite);
        }
 
        static void serial_DataReceived(object sender, SerialDataReceivedEventArgs e)
        {
            // create a single byte array
            byte[] bytes = new byte[1];
 
            // as long as there is data waiting to be read
            while (serial.BytesToRead > 0)
            {
                // read a single byte
                serial.Read(bytes, 0, bytes.Length);
                // send the same byte back
                serial.Write(bytes, 0, bytes.Length);
            }
        } 

    }
}

Thanks for reading...

Monday, November 19, 2012

Installing Oracle Java and Tomcat on Ubuntu Server 12

Ubuntu Server comes with only a command line console so the usual GUI tools are unavailable.  You instead get to Java and Tomcat by hand. Oracle installations require the addition of another apt repository because they are license restricted from the Linux point of view.

Install Java

Enter the following commands into the VMWare visible console.
  1. Ubuntu server doesn't include add-apt-repository so first install that
    sudo apt-get install software-properties-common
    Enter "Y" if prompted.
  2. Add the repository that contains Oracle information
    sudo add-apt-repository ppa:webupd8team/java
    Press "Enter" to continue.
  3. Update the list of available packages
    sudo apt-get update
  4. Install Oracle 7
    sudo apt-get install oracle-java7-installer
    Enter "Y" and click on "<OK>" to accept the license
  5. Verify Java is installed by running
    java -version
You can set the JAVA_HOME for all processes by adding a adding a JAVA_HOME entry in /etc/environment.  You have do this with escalated privileges/sudo. I added this:
    JAVA_HOME=/usr/lib/jvm/java-7-oracle

Install Tomcat

Tomcat requires java so you must have some version installed as shown above.
  1. Install the Tomcat app server using apt-get
    sudo apt-get install tomcat7 tomcat7-admin ufw 
    libtcnative-1
  2. Install the Apache web server using apt-get This is optional for just hacking around in tomcat but is recommended for real production systems.
    sudo apt-get install apache2 libapache2-mod-jk 
  3. And now install the MS SQL Server, MySql and Postgresql java drivers if you are going to want them for your app-server installation
    sudo apt-get install libjtds-java libmysql-java libpg-java 
  4. Groovy is cool so install just because
    sudo apt-get install groovy
  5. Edit /etc/default/tomcat7 and set the JAVA_HOME to /usr/lib/jvm/java-7-oracle

Additional Configuration

Reconfigure the Apache mod-jk plugin that talks to Tomcat if you installed Apache2 in the previous section.
  1. Edit /etc/libapache2-mod-jk/workers.properties
  2. Change workers.tomcat_home to be point to tomcat7 instead of tomcat6
    workers.tomcat_home=/usr/share/tomcat7
  3. Change the default java version to point to the oracle JDK
    workers.java_home=/usr/lib/jvm/java-7-oracle
Restart the servers to pick up the changes

Verify 

  • Restart tomcat7 to pickup the changes
    sudo service tomcat7 restart
  • Verify the services running by hitting the tomcat server on port 8080 with your browser http://<ip>:8080/
  • Restart apache2 if installed
    sudo service apache2 restart
  • Verify the web server is running through the browser by requesting on port 80 http://<ip>
  • Verify they web server and tomcat are wired together using the link below

Credits

I was able to glean useful information on this topic from the following web sites.
  • http://www.ubuntugeek.com/how-to-install-oracle-java-7-in-ubuntu-12-04.html
  • http://magnus-k-karlsson.blogspot.com/2012/04/how-to-set-javahome-environment.html
  • https://wiki.umn.edu/Main/Tomcat7ServerOnUbuntuServer12_04LTS
  • http://thetechnocratnotebook.blogspot.com/2012/05/installing-tomcat-7-and-apache2-with.html

Sunday, November 18, 2012

Virtual Machine Networking: NAT or Bridged?

VMWare and other desktop or embedded hypervisors create virtual networks that the virtual machines connect to when running on the hosting software.  These virtual networks provide network connectivity from the guest machines through the host machine out to the general LAN/WAN.  There are two main modes.

  1. NAT:  The host puts the virtual machines on their own private network and acts as a network gateway for that virtual network.  Guest machines can communicate over the shared network and communicate to the WAN/Internet through the host machine.  Guest machines consume none of the physical network's address space because they are connected on the hosts "virtual network".

    Guest network interfaces can be configured using DHCP, provided by the host, or with fixed IP addresses to simulate a stable server environment.  Fixed IP addresses are handy when the virtual machines need to communicate with each other. VMWare typically uses the host network's VMnet8 VLAN which doesn't change network range across reboots.

    This mode works well where the guest machines only need outbound access to the global network.  It is also useful where the host machine is mobile and plugs into multiple other physical networks.  No guest reconfiguration is necessary when the host is connected to a new network.
  2. Bridged: The host essentially creates a new IP adapter for each virtual machine and bridges the virtual network adapters to the physical network.  Guest machines consume address space on the physical network. A host running 2 virtual machines with Bridged networking has 3 IP addresses on the main network. It has one for the host and one for each virtual machine.

    Guest interfaces can be configured using DHCP provided by the network or with fixed IP addresses. Fixed IP addresses make it easier to route requests to the VMs.

    This method is the best method to use if the guest / virtual machines accept inbound traffic from the greater LAN/WAN or Internet. This mode works well if the capacity to allocate to the additional IP addresses.

Oracle has a good explanation of NAT and Bridged in VirtualBox at https://blogs.oracle.com/fatbloke/entry/networking_in_virtualbox1
VMWare has a good NAT explanation in their virtual networking environment http://www.vmware.com/support/ws45/doc/network_nat_details_ws.html

Setting the Size of the Terminal / Console Window in a Linux Server VM

We did a lot of Linux virtual machine installs when I worked with VMWare's SpringSource group. It let us run multi-node clusters off our MacBooks for customer demos.  We usually used desktop installes because they have you multi-terminal access to the machines.  I decided I wanted get some of my Linux/Java/OpenSource mojo back after 8 months on a Microsoft only project.  I downloaded Ubuntu 12.10 and installed server version so I could spin up more VMs on my Dell.

The server install is thin and the Linux console defaults to 640x48 as displayed in the VMWare Player/Workstation.  It wasn't exactly obvious how to make that terminal window larger.  The desktop version of Ubuntu is sync'd with the VMWare window size through the VMWare tools.  That isn't true of the server version.  You have to do the following steps. These assume you have created and installed a new machine and are logged into the console window on the Virtual Machine. I did this with Ubuntu 12.10 Server.

  1. First determine the resolution you wish to have the console run at. Most cards support the standard VESA resolutions and their associated mode numbers.  Pick the resolution you want and remember it's mode number.  I wanted 1024x768x16 so my mode number is 791. Additional information on how this works in ubuntu is available at the Ubuntu help site.
  2. Type sudo su at the terminal prompt and enter your password. This leaves you with a root capable shell.  You're more secure if you just run the su command for each command prompt because you won't do anything dangerous by accident but I'm kind of lazy and elevate my privileges.
  3. Edit /etc/default/grub with something like
        vi /etc/default/grub
  4. Find the line with GRUB_CMDLINE_LINUX_DEFAULT and add vga=xxx where xxx is the mode number.  It should look something like:
        GRUB_CMDLINE_LINUX_DEFAUT="quiet vga=791"
    1. You will see other documentation that talks about modeset or nomodeset. I think those are more applicable when you are not running on a virtual machine and where there are custom settings for a particular video card.
    2. Other document says you change the resolution using the line that includes GRUB_GFXMODE by removing the '#' at the beginning and setting the resolution to the same as the VESA mode. That didn't work for me and was unnecessary when I made the change just discussed.
  5. Save the changes and exit the editor
  6. Update grub with the new settings.  The settings will not take affect with out this.
        update-grub
  7. Restart your machine with the
        reboot command if you are still elevated or
        su reboot
    if not.
Ubuntu server takes up 1.2GB on the disk with additional services or tools installed.  It makes a nice lightweight virtualization platform.

Changing the Login Prompt in Linux

I try and standardize my user ids and passwords on my development virtual machines but sometimes a piece of software has tighter restrictions.  In that case I change the Linux login message to tell me the default user ids and passwords.  The contents of /etc/issue are displayed before the login prompt. Edit /etc/issue and add your user id and password information.  DO NOT do this for any system that is accessible to the Internet or that contains real or sensitive data.

Tuesday, October 30, 2012

Demonstrating Gemfire Components Configured using spring-data

Gemfire, and the applications that use it are made up of serveral components. There is the data caching tier itself that can exist as one or more data nodes. The data tier can be extended to provide database write-behind, write-through and on-demand read-through. The data tier can also be replicated to remote data clusters through the WAN gateway. WAN replication is not currently part of this example. There is also a client tier that can be pure client, client with local caching and notification client or a combination of the three. Client applications can consume data as they would a no-SQL store, database or they can register for data change notifications with the appropriate callback handlers. Gemfire relies on and makes use of other components including the Gemfire locator and JMXAgent. Gemfire clusters will almost always be be coordinated and linked through the use of Gemfire locator processes. JMXAgents add JMX bean access to a gemfire cluster for management and monitoring through either GFMon or some other JMX monitoring tool.


Demonstration program.

You can access this Gemfire demonstration source on github https://github.com/freemansoft/fire-samples Gemfire is a set of Java programs or modules. Gemfire components can be run in as standalone programs using the scripts included with the Gemfire product download or they can be run as part of some other executable. This demonstration provides support for running all of the PoC/demonstration components as part of custom java main() executable or as via a WAR file. The java main() programs provide a simple testing framework for running inside Eclipse. The WAR files provide a method of deploying Gemfire components in environments where all Java programs must run from inside a Java Servlet or J2EE container.



Gemfire Components and Functionality

This is a very basic demo of some of VMWare's Gemfire product. It is demonstrable as both standalone components and as deployable war files. The following is an incomplete list of demonstrated components.
Locator A locator process that shows how the ports are selected
Server Cache nodes A standalone data server or clustered server that includes listeners and continuous queries
Cache Client A standalone client program that has local cache
Continuous Query Clients A standalone client program that receives call backs when data changes in certain regions
WAN Gatway A cluster node that could be used to replicate data to another site. It currently logs batch data inserts and updates as they become available from the cluster nodes.
Cache Listener A component that runs in the server that logs listener events
Cache Loader Two examples, one that just logs when there is a cache miss and the other that is configured to load cache misses form the demo H2 database.
Cache Writer (write-through) Two examples, one that just logs when data changes in the cache and can be written some backend system. The other is configured to write to a DB when data is saved.
Replicated regions Gemfire regions where the data is replicated across all nodes
JmxAgent A JMXAgent wrapper that lets it be run in the same way as the other components.
Disk write-through persistence(?) One of the regions is configured for disk based persistence
Disk write-behind hooks demonstrated via WAN gateway and a wan gateway listener
The @cacheable annotation One of the commands and regions is dedicated the @cacheable annotation. See the "power" command
Spring Batch Used to load data files and to auto-map flat entities to the backing H2 database.

Some things that are not yet in the demonstration
  • WAN Gateway Basic Spring-Data-Gemfire configuration for a WAN gateway that is used, in this case, to log activity.
  • Database Write-behind This is built on top of WAN gateway. The demonstration includes logging mostly because I couldn't use spring batch to do no code writes they way I could do reads
  • Database Write-through The demonstration includes logging write-through. It will eventually have fully functional write-through
  • Partitioned regions This requires that users run at least two nodes to be useful.
  • Co-located data in queries These make use of data whose node location is picked based on its relationship to other data. All regions are currently fully replicated without any partioning.
  • Security hooks TBA
  • Data morphing based on permissions This is dependent on identity management.
  • Index based queries Queries on indicies with logging to show that Gemfire can search without deserializing the data on the server nodes.
  • Properties/meta-data region An example of how gemfire can use gemfire itself to store cluster behavior information for java proceedures or listeners.

Demonstration Components


Shell

The demo has a lightweight menu system available on stdin/stdout/log4j when running as standalone applications. The commands are available in both client or server processes. This provides simple way to muck around with Gemfire features without building your own application. See the menu in the individual programs for the current list of available commands. That command line lets you do the following functions:

Save a note to the notes table. Saves a command line entered note to the NOTES-REGION-WITH-GATEWAY table which is hooked to an asynchronous listener via the WAN gateway. Usage: <cmd> <double quoted message>
Generate 2^n as a demo of @Cacheable. Calls a method that calculates 2^n and caches or uses a previously calculated value. That method is marked with the @Cacheable annotation. Uses the cache version if you repeat the same number more than once. Cache entries expire 15 seconds after creation. Usage: <cmd> <n>
Return Cache sizes for all regions. Clients with local caches return the local cache size Usage: <cmd>
Create random data and stuff into Gemfire Cluster. This creates 400,000 rows in all tables Usage: <cmd>
Load a region from a CSV file (absolute or relative to project root). <cmd> <region_name> <csv_file_name>
Load data from DB pre-defined automapping assumes 1-1 mapping. Uses the Spring batch auto-binding layer <cmd> <region_name>
Load all possible regions from data from DB pre-defined automapping assumes 1-1 mapping. Runs one thread per region Usage: <cmd>
Modify a string attribute. Retrieves the object from the cache and changes its value. <cmd> <region_name> <pk> <property_name> <attribute_name>
Retrieve entire cacheName and show the first entries (requires template). Calls toString() <cmd> <cacheName> <showRowCount>
Retrieve all data in all Regions. Does not display. May run parallel fetch, one per region (uses templates) <cmd>
Retrieve from cacheName the object specified by key key <cmd> <cache_name> <key>
Rebalance cache. Only works on data/server node <cmd>
Set a property in the property region. Ex 'cacheWriterEnable|cacheLoaderEnable' 'true|false'. Useful for testing properties. <cmd> <property-name> <new-value>
Purge all data from cache: WARNING! <cmd>
Exit this program without hesitation. Terminates command line programs <cmd>

Executables

The demonstration is made up of several executables that exist as main() programs that can be run from inside Eclipse with RunAs-->JavaApplication. Some of these can also be run as standalone java programs with their own bootstrap. They were wrapped in the demonstraton to provide a common run interface. Java application class files containing the main() entry point exist in demo.vmware.app to make it easier to explore. These programs are all in the src/main/java/demo/vmware/app folder.
DB.java This is a demonstration database that can be used to test bulk loading and cache-loader code when there is a cache miss. Run this first if you have any of the loaders and listeners enabled.
Locator.java Run this to start your Gemfire locator that is required before any client or server can startup. This code supports an arbitrary number of locators per machine but is currently configured for 2 through spring wiring.
Server.java Run this to start a Gemfire server process. You can run multiple copies of this on the same machine. The caches are configured through spring so you can convert the regions from simple replicated to partitioned though those files. The demo command menu is available in this process type.
JmxAgent.java GFMon, the Gemfire monitoring tool, communicates with a cluster via JMX.This program starts the gemfire/jmx bridge. Run this agent before running gfmon.
Client.java Clients can be standalone or caching clients. This is a caching client that runs the same command menu as the server. Clients should be launched after all the server and services components have been started.
ClientCq.java This is a standalone non-caching client that registers a continuous query for data changes of a certain type. Clients should be launched after all the server and services components have been started.
WanGatewayWritebehind.java This runs a WAN gateway process that records all data that would either be written to a database as database write-behind or data that would be replicated across the WAN gateway to another data center. Only the NOTES-REGION-WITH-GATEWAY region is currently configured for this so you must use the "add a note" command in a server or client to insert data that is sent to this process.
Server, Client, Locator and JmxAgent can also be run inside container using the separate war files defined in src/main/resources. The WAR files are built as part of the standard maven build.

Spring Data Gemfire

The entire demonstration is configured using spring-data-gemfire and spring configuration files. They are located in src/main/resources. No external or custom Gemfire XML configuration files are needed.
spring-cache-client-core.xml Configures the locator and pdx serialization for Gemfire clients. It is shared by both cq and region-only clients.
spring-cache-client-cq-only.xml Configures the continuous query and callback for the Gemfire CQ client.
spring-cache-client-region-only.xml Configures the client cache regions for the Gemfire client application. Adds support for multiple clients on same machine.
spring-cache-gateway-writebehind.xml Configures the WAN gateway for a single replicated region, in this case to simulate write-behind.
spring-cache-jmxagent.xml Wires the JMX agent into the wrapper so we can run it as a java application without using the Gemfire supplied scripts
spring-cache-locator.xml Configures a Gemfire locator service including support for running multiple locators on the same machine.
spring-cache-server.xml Configures the Gemfire locator service including multiple nodes on the same machine. Defines all the server regions
spring-cache-templates.xml Defines Spring-data-gemfire cache templates. These act as cache (brand) agnostic proxies and support data insert and retrieval.
spring-cache-command-processor.xml Configures the stdin/stdout command processor that is available in demonstration clients and servers.
spring-cache-datasync.xml Creates the mappings for Gemfire Region and database table synchronizatoin. Used by the spring batch framework that the demonstration leverages for some I/O
spring-cache-db.xml Configures the H2 database connection
spring-cache-pdx-serializer Enables pdx serialization as the object serialization model.

Quick-Start Build & Demo

Logging levels are set in log4j.properties. Logging level changes require a restart to be picked up. Note: The demonstration programs can be run from inside Eclipse. Server and Client executables have command interpreters embedded in them. You can view and enter commands for them in the Eclipse console panes.

Demonstrate basic cache server functionality

  1. Download the code from github https://github.com/freemansoft/fire-samples
  2. Run mvn clean install to pull down all the dependencies and to build the class files and war packages. You will not be deploying the war packages as part of this quick-start.
  3. Start the H2 database using DB.java. This acts as the read/write-through database for one of the Gemfire regions. The database optional if you disable or remove the database connected cache loaders.
  4. Connect to H2 on http://localhost:8082 with your browser. Use the default H2 username (sa) and password (sa). This can be done from inside Eclipse
  5. Paste the contents of src/test/resources/sql/create-tables.sql into the H2 command window and execute it. This creates the schema and loads a couple rows of test data. You could add 100s or 1000s of rows in the sql file to increase the DB size
  6. Start a Gemfire locator instance using Locator.java. This can be done from inside Eclipse using runAs Java Application. You can run more than one Gemfire Locator on the same machine. The startup code will automatically bind to an unused port for each instance
  7. Start a Gemfire server using Server.java. This can be done from inside Eclipse runAs Java Application. Run the Server.java with larger memory settings if you're going to want to load big data in one of the steps down below. You can run more than one Gemfire server on the same machine. The startup code will automatically bind to an unused port for each Server instance. Load will be distributed across them via the locators.
  8. Verify the size of each cache using the command console of one of the Server instances. One of the commands prints the region sizes. They should all have zero elements in them because no cache operations have been processed yet.

Demonstrate databse read/write-through

  1. Each table in the database backs a region in the cache cluster. Query the database for a key. Then Request an object in one of the regions by key using a commond in the server console. The cache loader will fetch one row in before returning it to you.
  2. Query the region sizes again to see that the region used in the previous operation now has an element in it.
  3. Copy the contents all of the tables from the database into the cache using the server/command console.
  4. Verify the size of each cache using the server command console. Several of the caches should now have more than one object in them.
  5. Look at the server logs to see what happened. The logs may be intermixed with your command output. If you have more than one server running then you may see the log output on the other server(s) also.

Loading more Data

  1. Load some data from a csv using the command console. The command asks you for the file name and the table to be loaded. The path is relative to the eclipse project root so it wold be src/test/resources/datafiles. The filenames are not exactly the same as the region names so pay attention to that when entering the command. The companies file loads the ATTIRBUTE table and the relationships file is for the RELATIONSHIP table.
  2. You can create a 2 million record cache through auto-generate command in the command console. You will receive an "out of memory" error if you ran the servers with the default memory settings.
  3. Start a client running Client.java using the Eclipse runAs Java Application. You will have to use the larger heap settings described in Client.java if you want to pull down big data in the next step.
  4. You can pull down the 2 million rows down to the client via the command menu. It runs one thread per region about 13 seconds on a 2011 macbook.
  5. Pull down the content of one of the regions using the command console and have it print a couple objects using the command options.

Continuous Query

  1. Start a continuous query (cq) client ClientCq.java using the Eclipse Eclipse RunAs Java Application.
  2. Modify one of the objects that is being monitored by the cq client and you should see it log the fact that it got a callback.

Wan Gateway

  1. Start the WAN gateway write-behind process WanGatewayWritebehind.java using the Eclipse RunAs Java Application The WAN gateway is only bound to the "NOTES-REGION-WITH-GATEWAY". Operations on that region will be logged to the WAN gateway console.
  2. Create a note using the note command using one of the server (or client) command consoles. Double quote the notes string to include spaces.
  3. Watch the logs on the WAN gateway to see the gateway pick up the created note. The WAN gateway is "write-behind" so it will receive the data after some delay.

Using the Spring @Cacheable

  1. Find the power 2^n command in the command menu. The command calculates powers of 2 and stores them in the cache where they are purged after 15 seconds
  2. Run the power of two command with some number, say 3 on one of the servers. Note that the output says "calculating"
  3. Run the same command within 15 seconds. Note the same results are given without the "calculating" message
  4. Run the same command after 15 seconds. Not that the same results are given and the output says "calculating."
  5. Try the same with other numbers.
The cache listener and partitioning behavior can be changed via spring config files in the src/main/resources directory. You can leave the locator and datbase running across Server and Client restarts.

Data Organization

Gemfire Regions

The gemfire regions exist to demonstrate a couple different ideas and capabilities.
Data Model Regions
These were created for a specific demonstration and have self and cross reference joins that make them less than ideal candidates for partitioning. Most data models have good breakpoints for partitioning. These regions are replicated across all nodes. All of the data loading and creation happens in these regions. These are the regions we created GemfireTemplates for.
  • ATTRIBUTE
  • RELATIONSHIP
  • ACTIVITYLEGAL
  • CONTACT
  • TRANSFORM.
Properties regions.
These regions act as meta-data for some of cluster-side custom code. This lets us create state that can turn on and off features in our custom code. They can also be used for applicaton wide properties that might otherwise be pushed to mulitple file systems or an RDB.
  • PROPERTIES
"Cache-able" regions.
This codebase demonstrates how to use the @Cacheable annotation and how it can be used with gemfire to manage objects. This is a very easy way to push objects into and out of a cache without knowing the cache exists. All ther uses of the cache in this codebase "know" about the cache. The example is very simple only using a single region. Applications could share types of objects in a single region or they could have one region per object type for these caches. These tables have a 15 second TTL so entries expire 15 seconds after they are created
  • POWERTABLE
Regions supporting write-behind activities
This region is hooked configured in the WAN gateway so that any notes saved to it are asynchronously logged. This shows where database persistense code would go if you wanted to do database write-behind. The timeouts and batch sizes for the WAN gateway are configured in their XML file.
  • NOTES-REGION-WITH-GATEWAY
Regions for future demonstrations.
Other regions exist in the config to show alternate cache configurations including disk-persistent or partitioned regions.
  • NOTES-ON-DISK

Data Model

Data model objects exist for the Data Model Regions. The demonstration data model is very simple wit one entity per cache element. I twas done this way to simplify database write-through and read-through. Demonstration data entities all have single attribute keys. This isn't a requirement but simplifies the demonstration. Arbitrary, not necessarily uniform, or primitive data can be stored in the other regions. The properties region, for instance, could hold serialized objects or complex graphs. The Properties region only stores/retrives strings in the demo but they could be arbitrary property model objects.

Demonstration data

The example contains three sets of demonstration data
  1. Sql data that can be loaded into the H2 database. It is located in src/test/resources/sql
  2. CSV formatted data that can be loaded into tables via the command console in the server or client. It is located in src/test/resources/datafiles. Data exists for the ATTRIBUTE and RELATIONSHIP regions
  3. Program generated data is available through the command console. This can generate millions of rows in some of the tables in the server. The server must be launched with larger than default memory settings if you intend to use the data generator command.

Auto-mapping data and file interaction

To be written













Sunday, October 28, 2012

Spring.net configuration in App.config

I recently (10/2012) converted my TFS build monitor program to be Dependency Injected using the Spring.Net container. I did this because I wanted a no-code way of setting certain parameters and of specifying a build device management class without having to convert a configuration parameter into a class name. I instead injected the spring object with the same name as the configuration parameter using Spring.Net's built in facilities.

Spring.Net is moving towards more annotation based configuration but the XML configuration is still widely used and powerful.  You can easily configure Spring.Net in XML with markup inserted into App.config.  The first thing to do is to tell the system that there are going to be some spring.Net sections added to app.config.  This defines two sections, contexts and objects.  context is the section that describes where to find other pieces.  objects is the section that contains the actual object wiring.  Additional sections might be present if you are using the AOP runtime injection facilities in Spring.Net


    
    
      

The spring/context section describes the object configuration location, in this case in the config file (App.config) in the spring/objects section. The first object defined in objects is the VariablePlaceholderConfigurer. This class implements variable replacement from properties for spring.Net object parameters.


  
    
      
    
    
      
        
          
            
              
            
          
        
      
The following configuration creates a TfsBuildConnection instance under the name myBuildServerConnection. the "${Tfs.Url}" notation is the spring expression language replacement where the value will be the application property TFS.Url.
      
        
        
        
        
      


Consumers of a spring wired object can either have it directly injected through auto-wiring or they can retrieve the object from the spring context in a Service Locator pattern. This following cod retrieves the build server connection created above. The IApplicationContext should only be created once and can support multiple hierarchical context files. This context creation syntax uses the spring configuration from App.config (or web.config) when find the location for spring defined objects.
     IApplicationContext ctx = ContextRegistry.GetContext();
    TfsBuildConnection myBuildConnect =
        ctx.GetObject("myBuildServerConnection") as TfsBuildConnection;

More complete example

The following listing is extracted from a sample App.config that contains the spring.net.   The full file is available on github https://github.com/freemansoft/build-monitors/tree/master/build-lights-net That file contains additional log4net and other configuration.

  
  
    
    
      
An example that demonstrates simple IoC features. " lazy-init="true" >
Thanks for reading...