In a previous blog post, we introduced a new method of monitoring the Elastic Stack with Metricbeat. Using Metricbeat to externally collect monitoring information about Elastic Stack products improves the reliability of monitoring those products. It also provides flexibility with how the monitoring data may be routed to the Elasticsearch monitoring cluster.

SaaS adoption is continuously on the rise and so is the number of companies migrating their email services to Microsoft Office365. It’s the most popular SaaS service and while over 90% of enterprises use it, only less than a quarter of them have already migrated to the cloud-based suite. Nonetheless, this number is growing steadily, as cloud adoption rates are increasing.

What makes data structured or unstructured and how does that affect your logging efforts and information gain? Below we've provided a comparison of structured, semi-structured, and unstructured data. Also below, we discuss ways to turn unstructured data into structured data.

So, you’ve installed Coralogix’s STA and you would like to start analyzing your traffic and getting valuable insights but you’re not sure that you’re mirroring enough traffic or wondering if you might be mirroring too much data and could be getting more for less. The harsh truth is that in order to be able to detect everything, you have to capture everything and in order to be able to investigate security issues thoroughly, you need to capture every network packet.

I have a confession to make. I was going to use this blog to wax poetically about the flaw inherent in the well-established notion of the Three Pillars of Observability. I was going to argue that a fourth pillar was badly needed: topological metadata. In other words, without annotating each data stream across the three common data types with a pointer to where that data in the deployment topology was coming from, all you have is a lot of data with no chance of making sense of it.

In the first part of this blog series, we presented a use case on how machine learning can help to improve police operations. The use case demonstrates how operational planning can be optimized by means of machine learning techniques using a crime dataset of Chicago. However, this isn’t the only way to predict and prevent crime. Our next example takes us to London to have a look at what NCCGroup’s Paul McDonough and Shashank Raina have worked on.

Hidden among your normal, everyday logs are any number of unexpected events and anomalies. These events can provide key insights when troubleshooting problems, tracing transactions, or monitoring performance and behavior. However, finding them can be challenging, especially if you’re unsure of what, or how, to look for them. Searching for unexpected events involves more than just searching for keywords like “error” or “critical”.

This release unifies views, dashboards, and search for a more flexible and comprehensive approach to threat hunting. The expanded search introduces greater efficiency by making it easier to reuse searches you need to run on a regular basis with saved search and search workflows. Other enhancements such as full screen dashboards, and updates to alerting round out v3.2.

A sequel to our first post, Automating the installation of Elastic Cloud Enterprise with Ansible, this blog shows how to extend automation to cloud provisioning with Terraform. In the first post, we detailed how to deploy and configure Elastic Cloud Enterprise (ECE) across three availability zones in AWS using Ansible. However, the provisioning of the underlying EC2 instances and configuration of the security groups was all manual.

The Elastic Common Schema (ECS) defines a common set of fields for ingesting data into Elasticsearch. A common schema helps you correlate data from sources like logs and metrics or IT operations analytics and security analytics. Further information on ECS can be found in the official Elastic documentation, GitHub repository, or the Introducing Elastic Common Schema article.