The highest (maximum) lag in time for a given consumer group. Kafka_consumergroup_group_max_lag_seconds

The highest (maximum) lag in offsets for a given consumer group. Labels: cluster_name, group, is_simple_consumer For more information on how this is calculated read the Estimate consumer group lag in time section below. This metric correlates with lag in offsets. The difference between the last produced offset and the last consumed offset for this partition in this topic partition for this group. The offset of the last consumed offset for this partition in this topic partition for this group. Labels: cluster_name, group, topic, partition, member_host, consumer_id, client_id

Helm and when enabling the Kubernetes pod self-discovery features within Prometheus server, Prometheus server willĪutomatically detect the HTTP endpoint and scrape its data. Kafka LagĮxporter exposes several metrics as an HTTP endpoint that can be readily scraped by Prometheus. Prometheus is a standard way to represent metrics in a modern cross-platform manner. Filtering Metrics without Prometheus Server.Monitor Kafka Consumer Group Latency with Kafka Lag Exporter. Kafka Lag Exporter interpolates latency based on observed latest committed offset measurements of consumer groups.įor more information about Kafka Lag Exporter's features see Lightbend's blog post: If you like using this project and would like to support its development, please consider a donation using Patreon. Kafka Lag Exporter is maintained by Sean Glover ( and a community of contributors. Kafka Lag Exporter is an Akka Typed application written in Scala. It can run anywhere, but it provides features to run easily on Kubernetes clusters against Strimzi Kafka clusters using the Prometheus and Grafana monitoring stack. Kafka Lag Exporter makes it easy to view the offset lag and calculate an estimate of latency (residence time) of your Apache Kafka consumer groups. Need help monitoring Kafka, Java apps, or other Dockerized applications? Contact us.Monitor Kafka Consumer Group Latency with Kafka Lag Exporter Overview The ports and host name of each component may be configured inside of docker-compose.yml for a non localhost setup. It's currently set to simply scrape the running JMX exporter agent. If you wish to edit the Prometheus configuration file, you can find it under mount/prometheus/prometheus.yml. You should now be able to head over to Prometheus and query some Kafka, JMX, and JVM metrics. This blogpost assumes that you already have Docker and Docker Compose installed on your machine.īegin by grabbing the example code which contains a Docker setup that will spin up Zookeeper (a Kafka dependency), a Kafka instance, the JMX exporter agent, and a Prometheus instance to monitor it all.ĭo this by running docker-compose up inside of the prometheus_kafka/ directory. This time however, Kafka and the JMX exporter Java agent will be inside of a Docker container. Like the last post on this topic, we'll be using the JMX exporter to expose Kafka's metrics for our Prometheus to scrape. Previously we've looked at monitoring Kafka using Prometheus.

Kafka exporter