As the first part of a three-part series on Apache Kafka monitoring, this article explores which Kafka metrics are important to monitor and why. When monitoring Kafka, it’s important to also monitor ZooKeeper as Kafka depends on it. The second part will cover Kafka open source monitoring tools, and identify the tools and techniques you need to further help monitor and administer Kafka in production.
Open source software adoption continues to grow within enterprises (even for legacy applications), beyond just startups and born-in-the-cloud software. In this second part of our Kafka monitoring series (see the first part discussing Kafka metrics to monitor), we’ll take a look at some open source tools available to monitor Kafka clusters. We’ll explore what it takes to install, configure, and actually use each tool in a meaningful way.
Monitoring Kafka is a tricky task. As you can see in the first chapter, Kafka Key Metrics to Monitor, the setup, tuning, and operations of Kafka require deep insights into performance metrics such as consumer lag, I/O utilization, garbage collection and many more. Sematext provides an excellent alternative to other Kafka monitoring tools because it’s quick and simple to use.
SaltStack is an open source configuration management tool that lets you manage your infrastructure as code. Using SaltStack, you can manage tens of thousands of servers remotely using either a declarative language or imperative commands. It’s similar to configuration management tools such as Puppet, Chef, and Ansible.
The Linux Audit framework is a kernel feature (paired with userspace tools) that can log system calls. For example, opening a file, killing a process or creating a network connection. These audit logs can be used to monitor systems for suspicious activity.
Scenario Linux has a number of built-in tools, commands and files which can track and store information about every user activity. These tools are common in most Linux distributions and can be used to investigate suspicious logins or failed login attempts into the system. In this article, we will talk about some of the initial methods to identify possible security breaches. We will use an Amazon EC2 instance to show these commands.
AWS offers a plethora of log and metric data but in order to extract meaningful insights and react to production issues on a dime, a centralized logging solution is critical. Today we’re excited to announce the availability of Coralogix on the AWS Marketplace for the first time. You can now get our advanced machine learning log insights directly in your AWS cloud account for full-stack log aggregation, convenient billing, tighter security with your AWS infrastructure, and faster deployment.
If you look around the Serverless Application Repository console, you will find a number of applications that can help you ship logs from CloudWatch Logs to external services. One such example is the LogzioCloudWatchShipper application below.
All businesses today are built on layers of platforms. The app running your business is built on top of the Kubernetes application deployment platform, running on the AWS cloud platform. AWS is built on top of platforms such as the Linux operating system and the Intel X86 processor architecture. Smartly managed, a good product evolves into a platform for users to extract value and for developers to create new products and platforms. We all stand on the shoulders of giants.
If you liked Sematext Docker Agent you’ll love our new agent for Docker monitoring that provides you with even more insight into your Docker, Kubernetes, and Swarm clusters. Because of its power, small footprint, and ease of installation the old Sematext Docker Agent enjoyed high adoption by the Docker DevOps community.
