In our Sentry for Data series, we explain precisely why Sentry is the perfect tool for your data team. The present post focuses on how we optimized Airflow for deeper insights into what goes wrong when our data pipelines break. Data enables Sentry’s go-to-market teams by generating high-quality leads and tailored marketing campaigns. Of course, data is also used to steer the business by influencing how we think about Sentry pricing, future opportunities, and feature roadmap.

One of the benefits of using Logstash in your data pipelines is the ability to transform the data into the desired format according to the needs of your system and organization. There are many ways of transforming data in Logstash, one of them is using the mutate filter plugin. This Logstash filter plugin allows you to force fields into specific data types and add, copy, and update specific fields to make them compatible across the environment.

The use cases and requirements of a logging platform in an organization varies between teams and job functions. The problem isn’t in collecting log data (we are a logging company after all), but in deciding how to manage these logs for each team. For example, our backend developers need detailed, short-lived logs in order to build and test new features; while our infrastructure team needs lengthy retention periods for auditing and compliance.

Looking to learn about Logstash as quickly as possible? This Logstash Tutorial is for you: we’ll install Logstash and push some Apache logs to Elasticsearch in less than 5 minutes. Logstash is a good (if not the) swiss-army knife for logs. It works by reading data from many sources, processing it in various ways, then sending it to one or more destinations, the most popular one being Elasticsearch.

Over the past few years, and coupled with the growing adoption of microservices, distributed tracing has emerged as one of the most commonly used monitoring and troubleshooting methodologies. New tracing tools and frameworks are increasingly being introduced, driving adoption even further. One of these tools is Jaeger, a popular open source tracing tool. This article explores the integration of Jaeger with the ELK Stack for analysis and visualization of traces.

Software has eaten the world and every company today is a software company. This is because every company today is more and more serving its customers digitally. That service can be a spectrum, such as offering traditional physical products and services through digital channels on one end to offering entirely new digital products on the other end. Regardless of where on the spectrum a company is, it does not change the fact that its primary interface with its customers has become its software.

Every developer’s worst nightmare is having to dig through a huge log file trying to pinpoint problems. The troubleshooting most likely won’t stop there. They’ll either have to follow the trail to multiple other log files and possibly on other servers. The log files may even be in different formats. This may go on until one loses themselves completely. Log aggregation is what you need to stop this seemingly never-ending cycle.