Microcontroller - service connections and acquisition patterns

February 05, 2023

I ended up with some crosscutting content when I wrote up some patterns for getting data onto and off of microcontrollers. The consumption and generation patterns are cross-matched with implementation details. You may know where you want to send data or in how many directions you want to send data. Now you have other concerns. Connecting the devices, triggering transfers, and balancing latency with overhead.

The next step is to understand how you are going to handle the connectivity and connection initiation. That depends on the connection types, network topologies, and tools you have.
You then need to figure out how you are going to know when to capture data or receive commands. Is it time-based or event-based?
Then you need to understand what your payloads look like and balance that against CPU, latency, and other constraints.

This was just a subset but should give you the idea.

Microcontroller or remote-initiated connections

The data transfer process has three phases no matter which direction of the traffic or who starts it.
The first phase is connection initiation.
The second phase is the actual data transfer.
The connection is torn down.
Connection initiation and data transfer setup may be directionally aligned so that they both start in the host/process. Ex: Both start on the MCU and terminated in a cloud MQTT service.

They may be out of alignment with connection setup happening in one direction and data transfer in the other.

Ex: An MCU-based status display needs data that is available from a cloud endpoint. Firewall rules limit the connection setup blocking the cloud endpoint from seeing the IOT devices. This forces an approach where the MCU initiates the call the cloud endpoint to set up a persistent connection

Triggering: Interval or Event

Data is typically shared either at some regular interval or based on some trigger. The selected approach is based on data latency requirements and technical issues like networking or system capabilities.

Regular periodic updates:

Samples are taken at regular intervals and transferred or tagged with the interval

Are we OK with missing activities between updates?
Do we oversample and then only send changes almost simulating event-driven?
If we miss a periodic update, does that mean the controller is down or do we have another problem?

Event-driven transfers:

Data captured is triggered by events.

This could send data as soon as the data is available or delay it by some amount of time.
Bandwidth requirements are variable with activity.
Zero events mean no activity. Tough to tell difference between "no activity" and "down".

Hybrid Approach:

A hybrid approach that is both periodic and event-based.

Periodic

Periodic heartbeat transfers signal system health.
Periodic updates for certain data types.

Event

Event-driven data transfers based on MCU hardware or external triggers.
Transfers data only when necessary

Batching Considerations

We may wish to send data as soon as it is available or collect the data until some event occurs on an interval ends. We need to decide if MCU data transfer operations are synced immediately or captured and staged waiting for pickup. This decision has to be made for every data interchange in either direction.

This diagram shows samples being collected at regular intervals (in Purple). These could be collected on the MCU or in some remote service. The direction matters in implementation but not for the purposes of discussing individual data point transfer vs batching transfer.

The red circles convey the idea that the data is transferred soon after it is captured or created. There is a stream of notifications that mirrors the sample stream.
The blue circles capture the idea that data can be acquired and loaded into a buffer instead of sending them out. The entire batch is sent in a single transfer after some period of time or a number of samples.

Selecting an approach should take into account network connectivity, CPU power, or other things. Some protocols have heavy setup time. Some connection types are cheap-ish. I ran a benchmark that showed my IOT controller could only process 4 inbound HTTP requests per second. In this case, we needed to use some batching technique or retain some persistent connection to reduce the per-message overhead.

Connectivity

Connection initiation may depend on connectivity issues.

Are devices reachable from the services?
Is the data hub reachable from the devices?

Video

Security and credentials.

We need to understand how credentials are managed in either approach for security and maintainability.

Can we easily manage authentication credentials if they exist?

Revision History

Created 2023 02

Blog de Joe Freeman