Routing Java Logs and Business Events to Kafka - via logging

We often want to stream business events or raw logs from our applications to an analytical or operational data stores.  Kafka is currently one of the streaming APIs/platforms of choice for this.  

Java applications can use their standard logging APIs and send the logs to Kafka via the Kafka Appender that can be attached to the log4j2 subsystem. We can make just a few tweaks to this and use the same logging mechanisms and Kafka platform to capture custom Business Events.

Example code is availabe https://github.com/freemansoft/spring-boot-log-to-kafka-example, It builds on previous work.  

The sample code demonstrates using the logging subsystem to route logs and Audit to different destinations based on their severity or associated tags. The example send logs to the console and to a Kafka logs topics.  It sends tagged log messages to the Kafka Audit. topic.  All of this happens with the application code only having to write through the usual logging APIs.

Limitations

1. The Kafka Appender supports JSON and does not support AVRO.  
2. It isn't clear if the Kafka Appender supports all security models.
3. Systems must have a fallback plan for when Kafka was down if they could not lose the _events_ or _logs_

Java Logs and Audit to Kafka via Logging

Visualizing Logs and Audit streaming through Kafka

Created 2020/10