Routing Java Logs and Business Events to Azure Event Hubs using the Kafka logging appender

Azure Event Hubs implement a Kafka compatible API. This means we can migrate an application from Kafka to Azure EventHubs with minor changes. Retargeting basic Kafka traffic from Kafka to Event Hubs can be just a couple of configuration changes.

The sample program sent its logs to Kafka via the Log4J Kafka Appender. It takes just 4 lines of changes in the log4j2.xml configuration file to send the logs to Azure EventHubs instead.
You can find the sample program on GitHub in the Event=Hubs-Sink branch
https://github.com/freemansoft/spring-boot-log-to-kafka-example/tree/feature/Event-Hubs-Sink

Video

See the video for a longer explanation.


Create Azure Resources

We created an Azure Event Hubs Namespace and two individual hubs called logs and audit

env.properties

We can isolate our endpoint configurations in environment-specific properties files.  The default profile file would be env.properties.  Retrieve the SAS connection string you need from the Azure Portal.
bootstrap.servers=<namespace>.servicebus.windows.net:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule \
required username="$ConnectionString" \
password="Endpoint=sb://<namespace>.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<key-from-portal>";

audit.logmarker=AuditRecord
kafka.topic.audit=audit
kafka.topic.logs=logs

log4j2.xml

Inject the properties from the env-<profile>.properties file into log4j2.xml using the Properties section. Bind the properties to the logging appender in the log4j2.xml file's Properties section.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="info" name="spring-boot-log-to-kafka-example" packages="io.woolford">
    <Properties>
        <Property name="bootstrap.servers">${bundle:env:bootstrap.servers}</Property>
        <Property name="audit.logmarker">${bundle:env:audit.logmarker}</Property>
        <Property name="kafka.topic.audit">${bundle:env:kafka.topic.audit}</Property>
        <Property name="kafka.topic.logs">${bundle:env:kafka.topic.logs}</Property>

        <Property name="sasl.mechanism">${bundle:env:sasl.mechanism}</Property>
        <Property name="sasl.jaas.config">${bundle:env:sasl.jaas.config}</Property>
        <Property name="security.protocol">${bundle:env:security.protocol}</Property>
    </Properties>

    <Appenders>
        <!-- Kafka Audit Appender only forwards messages with the "AuditRecord" marker -->
        <!-- We put only the message here assuming it will be formatted json-->
        <Kafka name="kafkaAuditAppender" topic="${kafka.topic.audit}">
            <PatternLayout pattern="%message%n" />
            <Property name="bootstrap.servers">${bootstrap.servers}</Property>
            <Property name="sasl.mechanism">   ${sasl.mechanism}</Property>
            <Property name="sasl.jaas.config"> ${sasl.jaas.config}</Property>
            <Property name="security.protocol"> ${security.protocol}</Property>
            <MarkerFilter marker="${audit.logmarker}" onMatch="ACCEPT" onMismatch="DENY" />
        </Kafka>


...

Demonstration

Run the program as any other spring boot application. It will log to the hubs in the env.properties files.

Related Articles

  • http://joe.blog.freemansoft.com/2020/10/routing-java-logs-and-business-events.html
  • Microsoft article on using EventHubs as a Kafka API compatible stream https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview
Created 2022/01

Comments

Popular posts from this blog

Installing the RNDIS driver on Windows 11 to use USB Raspberry Pi as network attached

Understanding your WSL2 RAM and swap - Changing the default 50%-25%

Almost PaaS Document Parsing with Tika and AWS Elastic Beanstalk