Capture Confluent Kafka Metrics easily with Jolokia , Telegraf, InfluxDB and Grafana (JTIG)
Kafka provides all kinds of metrics that can be used for operational support and tuning work. We can use Telegraf/Jolokia to capture metrics from the various Confluent broker nodes and put those metrics into an Influx DB. We can then create Grafana dashboards for those metrics using Grafana Example Topology Deployed Components We run a custom Kafka Broker Docker Image built on top of the confluent/cp-server image. That image just adds the Jolokia JVM Agent jar file. Run a docker-compose.yml to start Kafka that enables Jolokia as a Java Agent The agent URL for demontration purposes The Telegraf standalone agent conf enables the Jolokia2 input adapter. The configuration file can be built into a new image or the configuration file can be mounted inside the standard DockerHub Telegraph image. Telegraf runs with this new configuration file. Telegraf retrieves the data from the Jolokia REST endpoints and sends it to InfluxDB Grafana provides visualization. It u...