UiFlow Python turns an M5Stack into an Azure App Insights Dashboard

I wanted to build an IoT dashboard that auto-populates from query Azure Application Insights and displays the query results. UIFlow, Circuit Python, and Micro Python all let you create low-code programs on IoT devices. I used M5Stack's UiFlow as a low-code Python generator.  The program runs an Azure query via REST API and displays the results on the built-in M5 Core display. The query can be manually triggered or run automatically on a timer without me writing any code.


https://flow.m5stack.com/
This flow is available on This .m5f example on GitHub.

Video Walkthrough

Device Topology

Our IoT device, the M5Stack Core 2, runs an API query against Application Insights.  The search results are returned as JSON. 

The Device Display

UiFlow lets us lay different fields and labels onto the LCD panel via drag and drop.  We can set the name of the component for later access.

UiFlow/Python has pre-built blocks that let me replace the text in any of those labels/fields or draw lines and take other activities.  

This screen shows the ping statistics for two different hosts.  There are three numbers, min, average, and max.  The oval at the bottom is a touch-enabled button that refreshes the data.



M5 UIFlow Python Layout

You can find UiFlow on the M5Stack website at https://flow.m5stack.com/ .

The left-hand side of UiFlow will show an image of the screen of the specific device you are using.  Think of it as the mobile phone designer version of the layout tools that let you see your UI on different screen sizes. 

Each of the fields and buttons is a separate, named, component.  UiFlow Python contains libraries that let you change the text for any of these labels.  All of these fields were dragged, drag-and-drop, into place and then edited.

Device Setup

The setup configures the wifi and builds the URL containing the query that will be sent to Azure Application Insights.  The query looks back a specific time span from "now" so the query does not need to be changed for each run.  Note that the URL includes 
  1. The Application Insights endpoint
  2. The Application Insights instance ID for our App Insights
  3. An escaped query string as a query parameter in the Microsoft query language 
Setup creates a periodic timer that triggers a refresh every 15 minutes and then makes a remote call to get data to initialize the display.

The Instance ID should be considered a secret and not checked into source code repositories.

Re-Use with Functions 

A display refresh can be initiated in 4 different ways.  We put all the refresh code in a function.  The refresh code queries the remote site and populates the display.

Our code refers to the function in the 4 different places a request can be initiated.  This is demo code so no parameters were passed and no values are returned.
  1. On Startup
  2. Every 15 minutes on the timer
  3. When the touch-sensitive area that looks like a big button is pressed.
  4. When the middle button just below the screen is pressed.

Run Query, Parse JSON results and Update Display

This function is the meat of the program.   It makes the request, parses the results, and then updates the display.  Note that this function changes the color of the LEDs to match the current step and the result state.
  1. Make a GET call to the Azure App Insights web endpoint using the URL created in Setup and add an Authentication token header with the name X-Api-Key
  2. If the call is successful then process it.  The returned data is a nested map and list structure.
  3. Update the run time on the display
  4. Update the HTTP result code field.
  5. Convert the response body into a JSON structure
  6. Extract the result rows from the JSON.  
  7. Display the first row's results in the first host's area if there is a first-row containing data.
  8. Display the second row's results in the 2nd host's area on the display if there is a 2nd row of data.
The definition also includes a rudimentary failure handler in the case an HTTPS call to Azure fails for whatever reason.

Pay special attention to the list and map operations where we extract the data from the JSON structures. They exactly match the nested structure.  Note that this is a low-code programmer design tool so there are some surprises like the way lists start with the "1st" element.

Application Insights Metrics Query Results.

All Application Insights are returned in JSON as a list of tables.  Our query only returns a single set of data so there is a single table in the results.  Our application 
  1. Extracts the first table.  This will be a Python Map
  2. We don't use the columns section because we have already built the screen layout in UiFlow.
  3. Extract the rows from the map.  This will be a list of rows.
  4. Populate the display with the values in the first row of the rows array: host, min, avg, max.
  5. Populate the display with the values in the second row of the rows array: host, min, avg, max.

Sample Application Insights Query Result.

{
"tables": [
{
"columns": [
{
"name": "cloud_RoleInstance",
"type": "string"
},
{
"name": "min_value",
"type": "real"
},
{
"name": "avg_value",
"type": "real"
},
{
"name": "max_value",
"type": "real"
}
],
"rows": [
[
"pi-18d8aec7",
41.922,
559.1242,
601146.3
],
[
"pi-153a3987b",
5.557,
6.763047,
8.683
]
],
"name": "PrimaryResult"
}
]
}

Look at the screens above to see where the data is placed.

Updating all Host Fields with  result row data

I was lazy and hard-coded to functions that each populate a section of the screen layout above.  One function populates the top host block and the other the 2nd host block in the middle of the page.  Note that these return silently if there is no first or second host data.


Serial Port Output

This is the output generated by the print statements

https://api.applicationinsights.io/v1/apps/SCRUBBED-INSTANCE-ID/query?
query=customMetrics+%7C+where+name+%3D%3D+"ST+Ping+Time"+%7C+where+timestamp+>+ago(4h)+%7C+summarize+min(value)%2C+avg(value)%2C+max(value)+by+cloud_RoleInstance
Query: customMetrics+%7C+where+name+%3D%3D+"ST+Ping+Time"+%7C+where+timestamp+>+ago(4h)+%7C+summarize+min(value)%2C+avg(value)%2C+max(value)+by+cloud_RoleInstance
RTC time: 2022-11-30 20:25:00 Wed
Response: {'tables': [
{
'columns': [
{'name': 'cloud_RoleInstance', 'type': 'string'},
{'name': 'min_value', 'type': 'real'},
{'name': 'avg_value', 'type': 'real'},
{'name': 'max_value', 'type': 'real'}],
'rows': [
['pi-18d8aec7', 62.668, 453.9218, 1852.822],
['pi-153a3987b', 5.831, 13.82269, 22.972]],
'name': 'PrimaryResult'
}]}

Response Cols: [
{'name': 'cloud_RoleInstance', 'type': 'string'},
{'name': 'min_value', 'type': 'real'},
{'name': 'avg_value', 'type': 'real'},
{'name': 'max_value', 'type': 'real'}]
Response Rows: [['pi-18d8aec7', 62.668, 453.9218, 1852.822], ['pi-153a3987b', 5.831, 13.82269, 22.972]]

Host 1 Results: ['pi-18d8aec7', 62.668, 453.9218, 1852.822]
Host 2 Results: ['pi-153a3987b', 5.831, 13.82269, 22.972]

Source Code

The source code shown here is not yet available on GitHub

Revision History

Created 2022 11

Comments

Post a Comment

Popular posts from this blog

Installing the RNDIS driver on Windows 11 to use USB Raspberry Pi as network attached

Image
I do a lot of my development and configuration via ssh into my Raspberry Pi Zero over the RNDIS connection. Some models of the Raspberry PIs can be configured with  gadget drivers  that let the Raspberry pi emulate different devices when plugged into computers via USB. My favorite gadget  is the network profile that makes a Raspberry Pi look like an RNDIS-attached network device.  All types of network services travel over an RNDIS device without knowing it is a USB hardware connection. A Raspberry Pi shows up as a Remote NDIS (RNDIS) device when you plug the Pi into a PC or Mac via a USB cable. The gadget  in the Windows Device Manager picture shows this RNDIS Gadget connectivity between a Windows machine and a Raspberry Pi. The Problem Windows 11 and Windows 10 no longer auto-installs the RNDIS driver that makes magic happen. Windows recognizes that the Raspberry Pi is some type of generic USB COM device. Manually running W indows Update  or  Update Driver  does  not  install the RNDI
Read more

Understanding your WSL2 RAM and swap - Changing the default 50%-25%

Image
The Windows Subsystem for Linux operates as a virtual machine that can dynamically grow the amount of RAM to a maximum set at startup time.  Microsoft sets a default maximum RAM available to 50% of the physical memory and a swap-space that is 1/4 of the maximum WSL RAM.  You can scale those numbers up or down to allocate more or less RAM to the Linux instance. The first drawing shows the default WSL memory and swap space sizing. The images below show a developer machine that is running a dev environment in WSL2 and Docker Desktop.  Docker Desktop has two of its own WSL modules that need to be accounted for.  You can see that the memory would actually be oversubscribed, 3 x 50% if every VM used its maximum memory.  The actual amount of memory used is significantly smaller allowing every piece to fit. Click to Enlarge The second drawing shows the memory allocation on my 64GB laptop. WSL Linux defaults to a maximum RAM size of 5
Read more

Almost PaaS Document Parsing with Tika and AWS Elastic Beanstalk

Image
The Apache Tika project provides a  library capable of parsing and extracting data and meta data from over 1000 file types.  Tika is available as a single jar file that can be included inside applications or as a deployable jar file that runs Tika as a standalone service. This blog describes deploying the Tika jar as an auto-scale service in Amazon AWS Elastic Beanstalk.  I selected Elastic Beanstalk because it supports jar based deployments without any real Infrastructure configuration. Elastic Beanstalk auto-scale should take care of scaling up and down for for the number of requests you get. Tika parses documents and extracts their text completely in memory. Tika was deployed for this blog using EC2 t2.micro instances available in the AWS free tier. t2.micro VMs are 1GB which means that you are restricted in document complexity and size. You would size your instances appropriately for your largest documents.   Preconditions An AWS account. AWS access id and secret key.
Read more