This article is the third of a four-part series of articles about Elasticsearch monitoring. In the first article, we put together an Elasticsearch guide, covering how Elasticsearch works and why the setup and tuning of Elasticsearch requires a good knowledge of configuration options and performance metrics.

Observability has become one of the most important areas of your application and infrastructure landscape, and the market has an abundance of tools available that seem to do what you need. In reality, however, most products - especially leading open source tools - were created to solve a single problem extremely well, and have added additional supporting functionality to become a more robust solution; but the non-core functionality is rarely best of breed. Examples of these are Prometheus and Grafana.

Let’s say you have a script that works when run in an interactive session, but does not produce expected results when run from cron. What could be the problem? Some potential culprits include: Or it could be something else. How to troubleshoot this then, and where to start? Instead of trying fixes at random, I prefer to start by looking at logs.

Incidents happen. What matters is how they’re handled. Most organizations have a strategy in place that starts with log searches—and logs/log searching are great, but log searching is also incredibly time consuming. Today, the goal is to get safer software out the door faster, and that means issues need to be discovered and resolved in the most efficient way possible.

Observability data provides the insights engineers need to make sense of increasingly complex cloud environments so they can improve the health, performance, and user experience of their systems. These insights can quickly answer business-critical questions like, “what is causing this latency in my front end?” Or, “why is my checkout service returning errors?” Observability is about accessing the right information at the right time to quickly answer these kinds of questions.

Does your organization’s data include sensitive information, like intellectual property or personally identifiable information (PII)? Do you want to protect your data from being stolen and sent (i.e., exfiltrated) to external web services? If the answer to these questions is yes, then Elastic’s Data Exfiltration Detection package can help you identify when critical enterprise data is being stolen and exfiltrated.