Routing Java Logs and Business Events to Azure Event Hubs using the Kafka logging appender

Azure Event Hubs implement a Kafka compatible API. This means we can migrate an application from Kafka to Azure EventHubs with minor changes. Retargeting basic Kafka traffic from Kafka to Event Hubs can be just a couple of configuration changes.

The sample program sent its logs to Kafka via the Log4J Kafka Appender. It takes just 4 lines of changes in the log4j2.xml configuration file to send the logs to Azure EventHubs instead.
You can find the sample program on GitHub in the Event=Hubs-Sink branch


See the video for a longer explanation.

Create Azure Resources

We created an Azure Event Hubs Namespace and two individual hubs called logs and audit

We can isolate our endpoint configurations in environment-specific properties files.  The default profile file would be  Retrieve the SAS connection string you need from the Azure Portal.
sasl.mechanism=PLAIN \
required username="$ConnectionString" \



Inject the properties from the env-<profile>.properties file into log4j2.xml using the Properties section. Bind the properties to the logging appender in the log4j2.xml file's Properties section.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="info" name="spring-boot-log-to-kafka-example" packages="io.woolford">
        <Property name="bootstrap.servers">${bundle:env:bootstrap.servers}</Property>
        <Property name="audit.logmarker">${bundle:env:audit.logmarker}</Property>
        <Property name="kafka.topic.audit">${bundle:env:kafka.topic.audit}</Property>
        <Property name="kafka.topic.logs">${bundle:env:kafka.topic.logs}</Property>

        <Property name="sasl.mechanism">${bundle:env:sasl.mechanism}</Property>
        <Property name="sasl.jaas.config">${bundle:env:sasl.jaas.config}</Property>
        <Property name="security.protocol">${bundle:env:security.protocol}</Property>

        <!-- Kafka Audit Appender only forwards messages with the "AuditRecord" marker -->
        <!-- We put only the message here assuming it will be formatted json-->
        <Kafka name="kafkaAuditAppender" topic="${kafka.topic.audit}">
            <PatternLayout pattern="%message%n" />
            <Property name="bootstrap.servers">${bootstrap.servers}</Property>
            <Property name="sasl.mechanism">   ${sasl.mechanism}</Property>
            <Property name="sasl.jaas.config"> ${sasl.jaas.config}</Property>
            <Property name="security.protocol"> ${security.protocol}</Property>
            <MarkerFilter marker="${audit.logmarker}" onMatch="ACCEPT" onMismatch="DENY" />



Run the program as any other spring boot application. It will log to the hubs in the files.

Related Articles

  • Microsoft article on using EventHubs as a Kafka API compatible stream
Created 2022/01


Popular posts from this blog

Accelerate Storage Spaces with SSDs in Windows 10 Storage Pool tiers

Understanding your WSL2 RAM and swap - Changing the default 50%-25%

Java 8 development on Linux/WSL with Visual Studio Code on Windows 10