Blog de Joe Freeman

Azure Event Hubs implement a Kafka compatible API. This means we can migrate an application from Kafka to Azure EventHubs with minor changes. Retargeting basic Kafka traffic from Kafka to Event Hubs can be just a couple of configuration changes. The sample program sent its logs to Kafka via the Log4J Kafka Appender. It takes just 4 lines of changes in the log4j2.xml configuration file to send the logs to Azure EventHubs instead. You can find the sample program on GitHub in the Event=Hubs-Sink branch https://github.com/freemansoft/spring-boot-log-to-kafka-example/tree/feature/Event-Hubs-Sink Video See the video for a longer explanation. Create Azure Resources We created an Azure Event Hubs Namespace and two individual hubs called logs and audit env.properties We can isolate our endpoint configurations in environment-specific properties files.  The def...

JSON and AVRO are both great serialization models.  JSON is all text, human readable, and very verbose.  AVRO is an efficient binary format.  They can serialize the same data but they can also handle schema evolution or field changes differently JSON supports field order changes because all of its fields come with their own label  in every single message.  Avro messages do not always handle field order changes. Field Order Avro serializer/deserializers operate on fields in the order they are declared. Producers and Consumers must be on a compatible schema including the field order .  Do not change the order of AVRO fields. All Producers and Consumers are must be updated at the same time if you change the field order. The AVRO 1.8 documentation says  Records   A record is encoded by encoding the values of its fields in the order that they are declared. In other words, a record is encoded as just the concatenation of the encodings of its fields. Fi...

We often want to stream business events or raw logs from our applications to an analytical or operational data stores.  Kafka is currently one of the streaming APIs/platforms of choice for this.   Java applications can use their standard logging APIs and send the logs to Kafka via the Kafka Appender that can be attached to the log4j2 subsystem. We can make just a few tweaks to this and use the same logging mechanisms and Kafka platform to capture custom Business Events. Example code is availabe  https://github.com/freemansoft/spring-boot-log-to-kafka-example , It builds on previous work.   The sample code demonstrates using the logging subsystem to route logs and Audit to different destinations based on their severity or associated tags. The example send logs to the console and to a Kafka logs topics.  It sends tagged log messages to the Kafka Audit . topic.  All of this happens with the application code only ...

Kafka provides all kinds of metrics that can be used for operational support and tuning work.  We can use Telegraf/Jolokia to capture metrics from the various Confluent broker nodes and put those metrics into an Influx DB. We can then create Grafana dashboards for those metrics using Grafana Example Topology Deployed Components We run a custom Kafka Broker Docker Image built on top of the confluent/cp-server image. That image just adds the Jolokia JVM Agent jar file.   Run a docker-compose.yml to start Kafka that enables Jolokia as a Java Agent The agent URL for demontration purposes The Telegraf standalone agent conf enables the Jolokia2 input adapter.  The configuration file can be built into a new image or the configuration file can be mounted inside the standard DockerHub Telegraph image. Telegraf runs with this new configuration file. Telegraf retrieves the data from the Jolokia REST endpoints and sends it to InfluxDB Grafana provides visualization.  It u...