The blog will take you through best practices to observe Kafka-based solutions implemented on Confluent Cloud with Elastic Observability. (To monitor Kafka brokers that are not in Confluent Cloud, I recommend checking out this blog.) We will instrument Kafka applications with Elastic APM, use the Confluent Cloud metrics endpoint to get data about brokers, and pull it all together with a unified Kafka and Confluent Cloud monitoring dashboard in Elastic Observability.
Apache Kafka has come a long way since its initial development at LinkedIn in 2010 and its release as an open-source project the following year. Over the past decade, it has grown from a humble messaging bus used to power internal applications into the world's most popular streaming data platform. Its evolution is remarkable, and it has taken the industry by storm, quickly becoming a go-to solution for data streaming and processing.