Monitoring an Azure environment can be a challenging task for even the most experienced and skilled team. Applications deployed on Azure are built on top of an architecture that is distributed and extremely dynamic. But all is not doom and gloom. Azure users have a variety of tools they can use to overcome the different challenges involved in monitoring their stack, helping them gain insight into the different components of their apps and troubleshoot issues when they occur.

AWS CloudWatch Logs gives you full visibility into your AWS infrastructure, from individual workloads to the services that bind them. Monitoring these logs helps ensure their smooth and continued operation, ongoing stability, and performance. Integrating CloudWatch Logs with LogDNA makes it easier to parse, search, and analyze AWS logs in order to detect anomalies and troubleshoot problems faster.

Logs Monitoring plays an important role in analyzing, troubleshooting and alerting on problems. Organizations have lots of data in logs that should be mined to get valuable insights on users, applications and systems behavior. Real time alerting from logs monitoring does help in identifying the problems early. It also provides security related insights when someone is trying to break into your organization.

As Stephen Marsland once said, “if data had mass, the earth would be a black hole.” A vast part of the immense amount of structured and unstructured data that we call “Big Data” is nothing but machine-originated log data. Logs are generated for a lot of different purposes – from security to debugging and troubleshooting. They constitute a gold mine of useful information and actionable insights if properly stored, managed, and analyzed.

Last week I was at Monitorama and it was an amazing time! Not only where the talks fantastic, but I also had the opportunity to join Liz Fong-Jones from Honeycomb.io for a podcast with The New Stack to dig a little deeper into some of the topics in monitoring and site reliability engineering.

What are the essential things to monitor in your infrastructure? Sure, CPU utilization, memory usage, and IO throughput. However, once you notice a significant load somewhere in your infrastructure you want to know what is causing it, and that typically boils down to needing to find the process that’s using too much CPU or memory or that’s doing disk or network IO like there’s no tomorrow.

Pulling in data exposed via API is not one of the most common use cases for ELK Stack users but it is definitely one I’ve come across in the past. Developers wrapping their database services with REST API, for example, might be interested in analyzing this data for business intelligence purposes. Whatever the reason, the ELK Stack offers some easy ways to integrate with this API. One of these methods is the Logstash HTTP poller input plugin.

Amazon S3 is an object storage service widely used for storing many different types of data in the cloud. While it’s inexpensive and easy to set up and maintain S3 buckets, it’s also very easy to overlook S3 security aspects and expose them to malicious attacks. A typical example is accidentally allowing public access to S3 files. Several recent high-profile data breaches were caused by lax S3 security.

One question that customers often ask is “why does the LogDNA agent need to run as root?” With IT departments and DevSecOps teams pushing to secure systems against cyberattacks, running a cloud-based logging agent as root sounds like a huge risk. While it’s true that you should avoid running applications as root, there are several reasons why our agent runs as root out of the box and several ways that we reduce your risk of attack.

Now that you have your brand new Graylog instance up and collecting your organization’s logs, all the data is quickly searchable and available for troubleshooting any issues as they arise. Just as easy as it is for you to use, an attacker with access to the logs now has a much simpler job of understanding your environment and seeing all of your data. You need to make sure you are doing all the due diligence you can to protect the data.