Back in November 2021, Grafana Labs released version 2.4 of Grafana Loki. One of the new features it included was a Promtail Kafka Consumer that can easily ingest messages out of Kafka and into Loki for storing, querying, and visualization. Kafka has always been an important technology for distributed streaming data architectures, so I wanted to share a working example of how to use it to help you get started.

When you send telemetry into Honeycomb, our infrastructure needs to buffer your data before processing it in our “retriever” columnar storage database. For the entirety of Honeycomb’s existence, we have used Apache Kafka to perform this buffering function in our observability pipeline.