In Part 1 of this series, we saw how Amazon MQ routes messages between services in a distributed application, and we looked at some of the key metrics that describe the performance of the message broker and its destinations. Now that we’ve introduced the metrics and their meaning, we’ll look at some tools you can use to collect and query metrics from Amazon MQ:
In Part 2 of this series, we showed you how to use CloudWatch to monitor metrics and logs from Amazon MQ. With CloudWatch, you can easily create ad-hoc graphs to visualize the performance of your messaging infrastructure and other AWS services you use (such as EC2, Lambda, and S3). But to monitor your Amazon MQ brokers, destinations, and clients alongside the rest of your applications and infrastructure, you need a monitoring platform that easily integrates with your whole technology stack.
To centralize logging from your entire stack—from traditional infrastructure to serverless components—Datadog is announcing native support for the launch of FireLens for Amazon ECS. FireLens streamlines logging by enabling you to configure a log collection and forwarding tool such as Fluent Bit directly in your Fargate tasks. We’ve partnered with AWS to provide built-in Fluent Bit support for Datadog so that you can now seamlessly route container logs from AWS Fargate.
Since Kubernetes was open sourced by Google in 2014, it has steadily grown in popularity to become nearly synonymous with Docker orchestration. Kubernetes is being widely adopted by forward-thinking organizations such as Box and GitHub for a number of reasons: its active community, rapid development, and of course its ability to schedule, automate, and manage distributed applications on dynamic container infrastructure.
Fluent Bit is a lightweight, multi-platform tool that can collect, parse, and forward log data from several different sources. Because Fluent Bit has a small memory footprint (~450 KB), it is an ideal solution for collecting logs in environments with limited resources, such as containerized services and embedded Linux systems (e.g., IoT devices).
Vertica is a platform that uses machine learning capabilities to help you analyze large amounts of data. Vertica provides high availability and parallel processing by replicating data onto multiple nodes in a cluster, and uses a column-based data store for efficient querying. You can deploy Vertica in the cloud, on premise, or as a hybrid of the two.
The Java Virtual Machine (JVM) dynamically manages memory for your applications, ensuring that you don’t need to manually allocate and release memory in your code. But anyone who’s ever encountered a java.lang.OutOfMemoryError exception knows that this process can be imperfect—your application could require more memory than the JVM is able to allocate.
Real-time collaboration helps teams resolve issues quickly, which is crucial during outages when you don’t have a minute to lose. If your organization is using DingTalk, Alibaba’s platform for cross-team communication and collaboration, Datadog’s new integration lets you share and discuss annotated graphs on the fly and route alerts to your teams’ group chats so you can start troubleshooting issues without skipping a beat.
Application performance monitoring (APM) dovetails nicely with infrastructure monitoring, allowing you to monitor app performance and end-user satisfaction in context with the rest of your infrastructure. That’s why we unveiled Datadog APM to complement our infrastructure monitoring platform and provide full-stack observability.
Logs provide invaluable information about issues you need to troubleshoot. In some circumstances, that may mean that you have to look back at old logs. For example, you may be running a security audit and need to analyze months-old HTTP request logs for a list of specific IP addresses over a period of time. Or you might need to investigate why a scheduled service never occurred, or run an exhaustive postmortem on incidents that happened over a couple months but that you suspect are related.
