Saturday, August 10, 2013

Monitoring Azure from a Raspberry Pi?

We have an Azure cloud network that we would like to monitor with a standalone status board.  One option is to do the "all Microsoft" Powershell thing with a windows device. Another option is to use the Linux and MAC Azure management tools released in 2012. There is also a Node.js based set of tools. An older MSDN blog entry is also useful for understanding this. Another option is to directly consume the Azure REST services used by the Node.js library.  I decided to try the Node.js tools.   REST service information is available on MSDN.

Hardware

Sometimes when you have stuff lying around you just have to come up with some way to use it. Sometimes it works out fine and some times not so much.  I have a 700MHz 512GB  Raspberry Pi computer sitting on the shelf with nothing to do.  Its that nifty Linux ARM based computer: with built in Ethernet, USB, video and an hardware extension bus. That hosts our monitoring scripts.

I also have the Adafruit LCD Keypad Kit for the Raspberry Pi.  It gives you 2 line LCD display with RGB backlight.

Raspberry Pi setup

Raspberry Pi setup i pretty straightforward.  Follow the instructions in the quick start guide http://www.raspberrypi.org/quick-start-guide to install and boot the operating system.  I had some additional config to do:

  1. Use raspi-config to set the timezone
  2. Use raspi-config to set the locale and keyboard type
  3. Install the chromium browser so that we don't have to download Azure publishing configuration onto a different machine. The default browser can't download the settings.
    • sudo apt-get install chromium-browser

Azure Command Line Interface (CLI) setup

The library is written on Node.js
  1. Install Node JS. I found this article useful. Later pre-compiled ARM versions of Node.js are available on the nodejs.or web site. I used v0.10.13. Install it into /usr/local
  2. Install the Azure Node.js based command line tools using the npm command as documented on windowsazure web site.  
    1. Download the Azure publishing settings per the windowsazure.com instructions.  I found I had to use a browser for this because of the way credentials are handled. The default Raspberry Pi browser is not powerful enough. Use something like chromium or use a different machine and transfer over the network or via thumb drive.
    2. Load those Azure publishing settings into the Azure Node.js configuration so that it can use those credentials to talk with the Azure cloud.
  3. At this point you can run Azure commandline commands from a terminal windows.  Every command starts with Azure.  Run something like Azure VM list to get a list of all virtual machines under that Azure subscription.

Performance Problems with Azure CLI on Raspberry Pi

The main problem is that it takes about 9.5 seconds to run even the Azure Help  command.  This is a pretty simple command with no obvious network or Azure dependencies. Here are some more timing numbers. 

azure help                 9.7 secondsazure vm list               21 secondsazure vm show <machineName> 21 seconds.

Azure CLI Limitations

The Node.js based tools can control and check status VM status.  They cannot retrieve any metrics or watch the load or behavior in the Azure network. This is a showstopper four our monitoring project. 

Adding an LCD panel to the Raspberry Pi

The Adafruit folks have a great tutorial on how to buy, build and write software for their 2x20 LCD display.  It describes all of the things you have to do to enable and configure the I2C based shield on a stock Raspberry Pi.

Conclusion

This project failed for two reasons:
  1. The CLI API does not provide the information needed for monitoring.
  2. The Azure Node.js library has too much overhead to perform on a Raspberry Pi.

Sunday, August 4, 2013

Azure Point to Site VPN - private access to your cloud environment

You don't really have to worry about connectivity when you have a single in-house data center.  All your proprietary data is on "your" network that you manage. You firewall protects your sensitive information from internet intruders.  The internal network provides routing and name lookup services.

You don't really worry about connectivity when your are consuming publicly available resources on the internet.  Your internal network allows outbound connections to the internet.  Your gateway knows which DNS servers provide name support.

Note: IPV4 network numbers in the diagrams are just examples. They happen to be how my internal and Azure networks are configured.

Azure a Cloud Provider

Cloud providers give you the ability to spin up off-site data centers that are visible and reachable from the internet.  The actual remote data center organization and configuration is somewhat opaque to you since it is managed and controlled by the cloud provider.   The cloud provides internal network connectivity between your virtual hosts.  PaaS services handle internal cloud specific configuration handles in-cloud machine-to-machine connectivity usually through proprietary API.  Public endpoints are usually web services or web sites. 

There is limited management connectivity from the outside. Allowing management access via the public internet increases the number of attack vectors and odds of system intrusion.


Azure users get around the remote nature of the environment by enabling Remote Desktop Protocol access to their windows machines. This lets any machine on the internet RDP into the Azure machines as long as they can provide the right credentials.  This is an obvious security risk.  Azure users add remote Powershell, remote profiling, remote management and non-web accesses by enabling those additional public endpoints.  This increases the number of attack vectors on against the Azure machines.  We want to minimize the number of ports/services that are visible to the internet while providing as much corporate/owner and operational access as possible.

Corporate Azure users get around the remote nature by extending a site-to-site VPN tunnel that joins the cloud and the internal corporate network.  Some companies will not allow this type of network configuration because they are worried about the bi-directional open nature of the connection. Site-to-site has additional issues like the fact that does not help off-site developers and operational folks because only one site-to-site connection is supported. 

Note: that this picture shows an Azure DNS service.  This is used for internal name service for processes that run inside your Azure account.  They could always connect by IP address if azure DNS has not been set up.  Azure DNS is not required for external access. That all happens on cloudnet.app and is supported by the cloudapp.net DNS servers. 

See this document for a description of how to set up Azure point to site VPN networks and connections.

Network Organization as a First Step, VLANs

Azure users must first organize their networks in a way that makes it possible to provide both public access to web sites and services while making it also possible to provide secure non-public access to the management functions.  The default Azure configuration throws all VMs and Cloud Services into a single large network.  
The first step is to create a Virtual LAN (VLAN). VLANs provide a way to organize a network into different segments and is the base level network construct for creating subnets including remote connection tunnels (VPNs).  Pretty much everything you previously created will have to be deleted and recreated sitting on a VLAN instead of your default Azure network. You will want to to subdivide the VLAN namespace into subnets. Normally they are just syntactic sugar because Azure doesn't support firewall or filtering between subnets.  They are important to the point-to-site VPN because Azure VPN will use one of the subnets as the landing zone for VPN connections.

Initial Azure Configuration, Preparing for VPN

Azure VPN requires a VLAN subnet of its own that acts as a type of DMZ network between your on-prem network and the Azure VLAN that you configured.  Systems Administrators use the Azure Management Portal to create the external connectivity subnet. It allows you to pick the number of addresses on the VPN subnet. Each connected VPN client consumes an address on this VPN subnet.  The VPN public gateway takes two addresses also. You should size appropriately for the number point-to-site connections that you expect.


The second piece of the VPN connection is the creation of the actual public IP gateway for the VPN connection subnet.  This is another option on the Azure Management Portal.  Microsoft creates a public IP address and creates internal routing from that address to the VPN subnet. It also creates VPN virtual appliances that sit behind that VPN public IP address.  

The VPN is protected by certificates.  I'm not going to go into details here. You can use Microsoft MSDN  point-to-site vpn configuration page for this information

Azure Point-to-Site Remote Back Channel Connectivity

A VPN connection makes the local machine part of a remote connection using a secure tunnel. Microsoft provides a VPN client side configuration program that is customized to a specific Azure VPN public address and network.  This program is dynamically created in the Azure Management Portal and makes use of built-in Windows 7 / 8 VPN capabilities. 

A point-to-site VPN connection creates an additional network on the local machine that is part of the same network defined in the VPN connection subnet.  Essentially it puts a machine "on the network" in the VPN subnet portion of your VLAN. This mans the local machine has access to all private resources on the VLAN.  The local user can to non-internet-public ports that are unavailable to others outside the Azure cloud.  This is because the local machine is part of the Azure network when connected to the VPN tunnel.


The default Windows VPN configuration does not isolate the local machine to the VPN network. It leaves all the other network connections active.  This means a point-to-site connected machine has access to the internal corporate network, the Azure VLAN and the internet.  Users must keep anti-virus, rootkit, and malware protection software up to date to stop attackers from attacking the Azure or corporate network through the local machine. It is possible to disable all other network connections while attached to a VLAN.  You see this a lot with VPN connections that are restricted due to corporate policies.

Conclusion

<to be written>

Thursday, August 1, 2013

The simple but awesome NeoPixel Shield with an Arduino

The folks at Adafruit have put out a nice "NeoPixel shield" which is essentially an 8x5 addressable RGB LED strip built into an Arduino shield.  They have created a nice library available on github. You can see the project on their product web page.  Here is a picture of the board mounted on an Arduino Uno. The LED in the bottom right corner is LED 0.  The LED in the the bottom left corner is node 7.  The second row up is node 8-15 and so on.  The LED in the upper left corner is node 39.



This picture shows the LED panel on my desktop. It totally overwhelmed the camera to the point that the rest of the room looks dark.


Firmware

I've created simple Arduino firmware that lets you send LED blinky commands over the Serial Port via USB.  You can set each pixel color individually along with one of 10 blink patterns.  Pattern 0 is off and pattern 1 is solid on so there are 8 actual blink patterns.  The firmware is located on github.

The LEDs are daisy chained into a single giant shift register.  The far end of the strand is updated by serially shifting the values through all the other LEDs.   The entire strand is updated on each push operation.   The WS2811/2812 controllers have have fairly strict timing requirements when shifting data through and latching data into strand.  Adafruit has implemented a nice library that meets all the timing requirements through hand coding and interrupt control. This means that interrupts are disabled for long enough periods of time that I had to turn down the serial port rate to 9600 to avoid character loss.

TFS Build Watcher Application Driver

I've also added a driver to the TFS Build Watcher software on github.  Individual build statuses are mapped to individual LEDs so you can get an overall feel of your various builds.

video



/*  
  Written by Freemansoft Inc.  
  Exercise Neopixel (WS2811 or WS2812) shield using the adafruit NeoPixel library  
  You need to download the Adafruit NeoPixel library from github,   
  unzip it and put it in your arduino libraries directory  
  commands include  
  rgb  <led 0..39> <red 0..255> <green 0..255> <blue 0..255> <pattern 0..9>: set RGB pattern to pattern <0:off, 1:continuous>  
  rgb  <all -1>  <red 0..255> <green 0..255> <blue 0..255> <pattern 0..9>: set RGB pattern to pattern <0:off, 1:continuous>  
  debug <true|false> log all input to serial  
  blank clears all  
  demo random colors   
  */  
 #include <Adafruit_NeoPixel.h>  
 #include <MsTimer2.h>  
 boolean logDebug = false;  
 // pin used to talk to NeoPixel  
 #define PIN 6  
 // Parameter 1 = number of pixels in strip  
 // Parameter 2 = pin number (most are valid)  
 // Parameter 3 = pixel type flags, add together as needed:  
 //  NEO_RGB   Pixels are wired for RGB bitstream  
 //  NEO_GRB   Pixels are wired for GRB bitstream  
 //  NEO_KHZ400 400 KHz bitstream (e.g. FLORA pixels)  
 //  NEO_KHZ800 800 KHz bitstream (e.g. High Density LED strip)  
 // using the arduino shield which is 5x8  
 const int NUM_PIXELS = 40;  
 Adafruit_NeoPixel strip = Adafruit_NeoPixel(NUM_PIXELS, PIN, NEO_GRB + NEO_KHZ800);  
 typedef struct  
 {  
  uint32_t activeValues;     // packed 32 bit created by Strip.Color  
  uint32_t lastWrittenValues;  //for fade in the future  
  byte currentState;       // used for blink  
  byte pattern;  
  unsigned long lastChangeTime; // where are we timewise in the pattern  
 } pixelDefinition;  
 // should these be 8x5 intead of linear 40 ?  
 volatile pixelDefinition lightStatus[NUM_PIXELS];  
 volatile boolean stripShowRequired = false;  
 ///////////////////////////////////////////////////////////////////////////  
 // time between steps  
 const int STATE_STEP_INTERVAL = 10; // in milliseconds - all our table times are even multiples of this  
 const int MAX_PWM_CHANGE_SIZE = 32; // used for fading at some later date  
 /*================================================================================  
  *  
  * bell pattern buffer programming pattern lifted from http://arduino.cc/playground/Code/MultiBlink  
  *  
  *================================================================================*/  
 typedef struct  
 {  
  boolean isActive;     // digital value for this state to be active (on off)  
  unsigned long activeTime;  // time to stay active in this state stay in milliseconds   
 } stateDefinition;  
 // the number of pattern steps in every blink pattern   
 const int MAX_STATES = 4;  
 typedef struct  
 {  
  stateDefinition state[MAX_STATES];  // can pick other numbers of slots  
 } ringerTemplate;  
 const int NUM_PATTERNS = 10;  
 const ringerTemplate ringPatterns[] =  
 {  
   // state0 state1 state2 state3   
   // the length of these times also limit how quickly changes will occure. color changes are only picked up when a true transition happens  
  { /* no variable before stateDefinition*/ {{false, 1000}, {false, 1000}, {false, 1000}, {false, 1000}} /* no variable after stateDefinition*/ },  
  { /* no variable before stateDefinition*/ {{true, 1000}, {true, 1000}, {true, 1000}, {true, 1000}}  /* no variable after stateDefinition*/ },  
  { /* no variable before stateDefinition*/ {{true , 300}, {false, 300}, {false, 300}, {false, 300}} /* no variable after stateDefinition*/ },  
  { /* no variable before stateDefinition*/ {{false, 300}, {true , 300}, {true , 300}, {true , 300}} /* no variable after stateDefinition*/ },  
  { /* no variable before stateDefinition*/ {{true , 200}, {false, 100}, {true , 200}, {false, 800}} /* no variable after stateDefinition*/ },  
  { /* no variable before stateDefinition*/ {{false, 200}, {true , 100}, {false, 200}, {true , 800}} /* no variable after stateDefinition*/ },  
  { /* no variable before stateDefinition*/ {{true , 300}, {false, 400}, {true , 150}, {false, 400}} /* no variable after stateDefinition*/ },  
  { /* no variable before stateDefinition*/ {{false, 300}, {true , 400}, {false, 150}, {true , 400}} /* no variable after stateDefinition*/ },  
  { /* no variable before stateDefinition*/ {{true , 100}, {false, 100}, {true , 100}, {false, 800}} /* no variable after stateDefinition*/ },  
  { /* no variable before stateDefinition*/ {{false, 100}, {true , 100}, {false, 100}, {true , 800}} /* no variable after stateDefinition*/ },  
 };  
 ///////////////////////////////////////////////////////////////////////////  
 void setup() {  
  // 50usec for 40pix @ 1.25usec/pixel : 19200 is .5usec/bit or 5usec/character  
  // there is a 50usec quiet period between updates   
  //Serial.begin(19200);  
  // don't want to lose characters if interrupt handler too long  
  // serial interrupt handler can't run so arduino input buffer length is no help  
  Serial.begin(9600);  
  strip.begin();  
  strip.show(); // Initialize all pixels to 'off'  
  stripShowRequired = false;  
  // initialize our buffer as all LEDS off  
  go_dark();  
  //show a quickcolor pattern  
  configureForDemo();  
  delay(3000);  
  go_dark();  
  MsTimer2::set(STATE_STEP_INTERVAL, process_blink);  
  MsTimer2::start();  
 }  
 void loop(){  
  const int READ_BUFFER_SIZE = 4*6; // rgb_lmp_red_grn_blu_rng where ringer is only 1 digit but need place for \0  
  char readBuffer[READ_BUFFER_SIZE];  
  int readCount = 0;  
  char newCharacter = '\0';  
  while((readCount < READ_BUFFER_SIZE) && newCharacter !='\r'){  
   // did the timer interrupt handler make changes that require a strip.show()?  
   // note: strip.show() attempts to unmask interrupts as much as possible  
   // must be inside character read while loop  
   if (stripShowRequired) {  
    stripShowRequired = false;  
    strip.show();  
   }  
   if (Serial.available()){  
    newCharacter = Serial.read();  
    if (newCharacter != '\r'){  
     readBuffer[readCount] = newCharacter;  
     readCount++;  
    }  
   }  
  }  
  if (newCharacter == '\r'){  
   readBuffer[readCount] = '\0';  
   // this has to be before process_Command because buffer is wiped  
   if (logDebug){  
     Serial.print("received ");  
     Serial.print(readCount);  
     Serial.print(" characters, command: ");  
     Serial.println(readBuffer);  
   }  
   // got a command so parse it  
   process_command(readBuffer,readCount);  
  }   
  else {  
   // while look exited because too many characters so start over  
  }  
 }  
 /*  
  * blank out the LEDs and buffer  
  */  
 void go_dark(){  
  unsigned long ledLastChangeTime = millis();  
  for ( int index = 0 ; index < NUM_PIXELS; index++){  
   lightStatus[index].currentState = 0;  
   lightStatus[index].pattern = 0;  
   lightStatus[index].activeValues = strip.Color(0,0,0);  
   lightStatus[index].lastChangeTime = ledLastChangeTime;  
   strip.setPixelColor(index, lightStatus[index].activeValues);  
  }  
  // force them all dark immediatly so they go out at the same time  
  // could have waited for timer but different blink rates would go dark at slighly different times  
  strip.show();  
 }  
 //////////////////////////// handler //////////////////////////////  
 //  
 /*  
  Interrupt handler that handles all blink operations  
  */  
 void process_blink(){  
  boolean didChangeSomething = false;  
  unsigned long now = millis();  
  for ( int index = 0 ; index < NUM_PIXELS; index++){  
   byte currentPattern = lightStatus[index].pattern;   
   if (currentPattern >= 0){ // quick range check for valid pattern?  
    if (now >= lightStatus[index].lastChangeTime   
      + ringPatterns[currentPattern].state[lightStatus[index].currentState].activeTime){  
     // calculate next state with rollover/repeat  
     int currentState = (lightStatus[index].currentState+1) % MAX_STATES;  
     lightStatus[index].currentState = currentState;  
     lightStatus[index].lastChangeTime = now;  
     // will this cause slight flicker if already showing led?  
     if (ringPatterns[currentPattern].state[currentState].isActive){  
      strip.setPixelColor(index, lightStatus[index].activeValues);  
     } else {  
      strip.setPixelColor(index,strip.Color(0,0,0));  
     }  
     didChangeSomething = true;  
    }  
   }  
  }  
  // don't show in the interrupt handler because interrupts would be masked  
  // for a long time.   
  if (didChangeSomething){  
   stripShowRequired = true;  
  }  
 }  
 // first look for commands without parameters then with parametes   
 boolean process_command(char *readBuffer, int readCount){  
  int indexValue;  
  byte redValue;  
  byte greenValue;  
  byte blueValue;  
  byte patternValue;  
  // use string tokenizer to simplify parameters -- could be faster by only running if needed  
  char *command;  
  char *parsePointer;  
  // First strtok iteration  
  command = strtok_r(readBuffer," ",&parsePointer);  
  boolean processedCommand = false;  
  if (strcmp(command,"h") == 0 || strcmp(command,"?") == 0){  
   help();  
   processedCommand = true;  
  } else if (strcmp(command,"rgb") == 0){  
   char * index  = strtok_r(NULL," ",&parsePointer);  
   char * red   = strtok_r(NULL," ",&parsePointer);  
   char * green  = strtok_r(NULL," ",&parsePointer);  
   char * blue  = strtok_r(NULL," ",&parsePointer);  
   char * pattern = strtok_r(NULL," ",&parsePointer);  
   if (index == NULL || red == NULL || green == NULL || blue == NULL || pattern == NULL){  
    help();  
   } else {  
    // this code shows how lazy I am.  
    int numericValue;  
    numericValue = atoi(index);  
    if (numericValue < 0) { numericValue = -1; }  
    else if (numericValue >= NUM_PIXELS) { numericValue = NUM_PIXELS-1; };  
    indexValue = numericValue;  
    numericValue = atoi(red);  
    if (numericValue < 0) { numericValue = 0; }  
    else if (numericValue > 255) { numericValue = 255; };  
    redValue = numericValue;  
    numericValue = atoi(green);  
    if (numericValue < 0) { numericValue = 0; }  
    else if (numericValue > 255) { numericValue = 255; };  
    greenValue = numericValue;  
    numericValue = atoi(blue);  
    if (numericValue < 0) { numericValue = 0; }  
    else if (numericValue > 255) { numericValue = 255; };  
    blueValue = numericValue;  
    numericValue = atoi(pattern);  
    if (numericValue < 0) { numericValue = 0; }  
    else if (numericValue > NUM_PATTERNS) { numericValue = NUM_PATTERNS-1; };  
    patternValue = numericValue;  
    /*  
    Serial.println(indexValue);  
    Serial.println(redValue);  
    Serial.println(greenValue);  
    Serial.println(blueValue);  
    Serial.println(patternValue);  
    */  
    if (indexValue >= 0){  
     lightStatus[indexValue].activeValues = strip.Color(redValue,greenValue,blueValue);  
     lightStatus[indexValue].pattern = patternValue;  
    } else {  
     for (int i = 0; i < NUM_PIXELS; i++){  
      lightStatus[i].activeValues = strip.Color(redValue,greenValue,blueValue);  
      lightStatus[i].pattern = patternValue;  
     }  
    }  
    processedCommand = true;    
   }  
  } else if (strcmp(command,"blank") == 0){  
   go_dark();  
   processedCommand = true;  
  } else if (strcmp(command,"debug") == 0){  
   char * shouldLog  = strtok_r(NULL," ",&parsePointer);  
   if (strcmp(shouldLog,"true") == 0){  
    logDebug = true;  
   } else {  
    logDebug = false;  
   }  
   processedCommand = true;  
  } else if (strcmp(command,"demo") == 0){  
   configureForDemo();  
   processedCommand = true;  
  } else {  
   // completely unrecognized  
  }  
  if (!processedCommand){  
   Serial.print(command);  
   Serial.println(" not recognized ");  
  }  
  return processedCommand;  
 }  
 /*  
  * Simple method that displays the help  
  */  
 void help(){  
  Serial.println("h: help");  
  Serial.println("?: help");  
  Serial.println("rgb  <led 0..39> <red 0..255> <green 0..255> <blue 0..255> <pattern 0..9>: set RGB pattern to pattern <0:off, 1:continuous>");  
  Serial.println("rgb  <all -1>  <red 0..255> <green 0..255> <blue 0..255> <pattern 0..9>: set RGB pattern to pattern <0:off, 1:continuous>");  
  Serial.println("debug <true|false> log all input to serial");  
  Serial.println("blank clears all");  
  Serial.println("demo color and blank wheel");  
  Serial.flush();  
 }  
 //////////////////////////// demo //////////////////////////////  
 /*  
  * show the various blink patterns  
  */  
 void configureForDemo(){  
  unsigned long ledLastChangeTime = millis();  
  for ( int index = 0 ; index < NUM_PIXELS; index++){  
   lightStatus[index].currentState = 0;  
   lightStatus[index].pattern = (index%8)+1; // the shield is 8x5 so we do 8 patterns and we know pattern 0 is off  
   lightStatus[index].activeValues = Wheel(index*index & 255);  
   lightStatus[index].lastChangeTime = ledLastChangeTime;  
  }  
  uint16_t i;   
  for(i=0; i<strip.numPixels(); i++) {  
   strip.setPixelColor(i,lightStatus[i].activeValues);  
  }     
  strip.show();  
 }  
 //////////////////////////// stuff from the Adafruit NeoPixel sample //////////////////////////////  
 // Input a value 0 to 255 to get a color value.  
 // The colours are a transition r - g - b - back to r.  
 uint32_t Wheel(byte WheelPos) {  
  if(WheelPos < 85) {  
   return strip.Color(WheelPos * 3, 255 - WheelPos * 3, 0);  
  } else if(WheelPos < 170) {  
   WheelPos -= 85;  
   return strip.Color(255 - WheelPos * 3, 0, WheelPos * 3);  
  } else {  
   WheelPos -= 170;  
   return strip.Color(0, WheelPos * 3, 255 - WheelPos * 3);  
  }  
 }