Monitor Internet Broadband service with a Raspberry Pi 4 and some Python

You can easily automate capturing broadband connection statistics with some Python code running on a Raspberry Pi, a Mac or, a PC.  I used a Raspberry Pi 4 as my test appliance because it is cheap and can support 1GB/s ethernet connections. That means it is fast enough to service most residential or low-end commercial connections.

I'm lazy and wanted the data to end up in a secure public cloud that could be populated and viewed from anywhere. We can send our broadband statistics from 1 or more locations and graph the different locations against each other. Any tool could be used.

Monitoring One or Compare Two 

We wanted to compare two different internet provider's service levels.  One provider is a FIOS 1GB down / 1GB up.  The other is a cable service with 1GB down / 50MB up. The providers and the technology were different.  We wanted to know if the complaints about one of the providers were valid.

Relies on Speedtest.net infrastructure

We're going to leverage the popular SpeedTest.net infrastructure and services. They provide measurements for
  1. Ping time
  2. Download speed
  3. Upload speed
It turns out that the amount of time it takes to set up the test also varies by the quality of the internet connection and those of nearby servers.   We will capture
  1. Test server selection time

Two different sites running connection and speed tests. Test results are fed to Azure application Insights

Video Talk

The Code

The code runs anywhere you can run Python 3. It has been tested on Raspberry Pi models 400 and 3.  You can find the configuration scripts and code at

Application Flow

This diagram describes the calling sequence and the roles of each tier.
  1. Get the active server.
  2. Search a list of servers for the fasted/nearest one
  3. Run ping tests
  4. Run download and upload tests if they have been requested


Test results and certain log messages can be sent to Microsoft Azure Application Insights. This gives you access to visuals and graphs without having to install or run a visualization server.

The Dashboard!

Speedtest Ping Times

This Azure Application Insights chart shows two things.
  1. FIOS ping time is about 1/8th of the ping time of the cable company.
  2. FIOS ping time was pretty consistent.
  3. Cable ISP ping times are inconsistent.  They can have significant outliers.

Speedtest Upload Times

We see the expected upload time differences between cable and FIOS.   Cable service is asymmetrical with slower upload speeds.  FIOS bandwidth is the same in both directions.

Speedtest Download Times

Both services claim to provide the same 1Gb/sec download speed.  You see here that the RPi can see almost 1GB for FIOS.  Mediacom cable never reaches 1/3 of their rated download speed in this 4-hour test.  750Mb/s -vs- 275Mb/s

Monitoring Speedtest Setup Performance

I noticed that test startup time can vary widely by the provider and by general internet contention.  The Python code captures that setup time. You can see here that FIOS can find the best test server in about 300Msec.  The cable provider setup time varies wildly between 1000msec and 8000msec in the 12-hour window shown below.

Future Additions

DNS lookup times can have a big performance impact. I'd like to add DNS lookup timing as a metric. I'm not sure how to avoid the local DNS cache.

References

Python Metrics and logs in Azure Applications Insights
  • https://joe.blog.freemansoft.com/2022/11/querying-python-transactions-and-spans.html
  • https://joe.blog.freemansoft.com/2021/03/querying-python-logs-azure-application.html
  • https://joe.blog.freemansoft.com/2021/03/querying-python-metrics-custom-tags-as.html
  • https://joe.blog.freemansoft.com/2021/03/displaying-python-metrics-in-azure.html
  • https://joe.blog.freemansoft.com/2021/03/monitor-internet-broadband-service-with.html
Microsoft Guides
  • https://docs.microsoft.com/en-us/azure/azure-monitor/app/opencensus-python
  • https://docs.microsoft.com/en-us/azure/azure-monitor/app/create-new-resource
  • https://github.com/census-instrumentation/opencensus-python/tree/master/contrib/opencensus-ext-azure
Python Data generator 
  • https://github.com/freemansoft/speedtest-app-insights
Created 3/2021

Comments

Post a Comment

Popular posts from this blog

Installing the RNDIS driver on Windows 11 to use USB Raspberry Pi as network attached

Image
I do a lot of my development and configuration via ssh into my Raspberry Pi Zero over the RNDIS connection. Some models of the Raspberry PIs can be configured with  gadget drivers  that let the Raspberry pi emulate different devices when plugged into computers via USB. My favorite gadget  is the network profile that makes a Raspberry Pi look like an RNDIS-attached network device.  All types of network services travel over an RNDIS device without knowing it is a USB hardware connection. A Raspberry Pi shows up as a Remote NDIS (RNDIS) device when you plug the Pi into a PC or Mac via a USB cable. The gadget  in the Windows Device Manager picture shows this RNDIS Gadget connectivity between a Windows machine and a Raspberry Pi. The Problem Windows 11 and Windows 10 no longer auto-installs the RNDIS driver that makes magic happen. Windows recognizes that the Raspberry Pi is some type of generic USB COM device. Manually running W indows Update  or  Upd...
Read more

Understanding your WSL2 RAM and swap - Changing the default 50%-25%

Image
The Windows Subsystem for Linux operates as a virtual machine that can dynamically grow the amount of RAM to a maximum set at startup time.  Microsoft sets a default maximum RAM available to 50% of the physical memory and a swap-space that is 1/4 of the maximum WSL RAM.  You can scale those numbers up or down to allocate more or less RAM to the Linux instance. The first drawing shows the default WSL memory and swap space sizing. The images below show a developer machine that is running a dev environment in WSL2 and Docker Desktop.  Docker Desktop has two of its own WSL modules that need to be accounted for.  You can see that the memory would actually be oversubscribed, 3 x 50% if every VM used its maximum memory.  The actual amount of memory used is significantly smaller allowing every piece to fit. Click to Enlarge The second drawing shows the memory allocation on my 64GB laptop. WSL Linux defaul...
Read more

Almost PaaS Document Parsing with Tika and AWS Elastic Beanstalk

Image
The Apache Tika project provides a  library capable of parsing and extracting data and meta data from over 1000 file types.  Tika is available as a single jar file that can be included inside applications or as a deployable jar file that runs Tika as a standalone service. This blog describes deploying the Tika jar as an auto-scale service in Amazon AWS Elastic Beanstalk.  I selected Elastic Beanstalk because it supports jar based deployments without any real Infrastructure configuration. Elastic Beanstalk auto-scale should take care of scaling up and down for for the number of requests you get. Tika parses documents and extracts their text completely in memory. Tika was deployed for this blog using EC2 t2.micro instances available in the AWS free tier. t2.micro VMs are 1GB which means that you are restricted in document complexity and size. You would size your instances appropriately for your largest documents.   Preconditions An AWS account. AWS ac...
Read more