The traditional approach for searching observability data is a tried-and-true: Once all the search staging is accomplished, we can perform high-speed, high-performance, deep-dive analysis of the data. But is this the best way or even the only way to search all that observability data? The answer to the first question is maybe, as it depends on what you are trying to accomplish. The answer to the second question must be a resounding no.

Over the course of two decades, Adform grew from a dream between friends huddled in a basement to a leading advertising tech platform powering more than 25,000 clients worldwide. Success brought external accolades, but it also created the need for internal innovation to support the company’s continued growth. In 2018, Adform was still operating in startup mode, which meant developers and teams cherry-picked the tools that worked best for them.

With the introduction of Environments & Services, we’ve seen a dramatic increase in the creation of new datasets. These new datasets are smaller than ones created with Honeycomb Classic, where customers would typically place all of their services under a single, large dataset. This change has presented some interesting scaling challenges, which I’ll detail in this post, along with the solution we used, and how we leveraged Honeycomb’s own telemetry to scale Honeycomb.

Intercom’s mission is to build better communication between businesses and their customers. With that in mind, they began their journey away from metrics alone and towards complete observability. The first step was tooling, and they learned quickly that trying to work with multiple solutions was not the answer.

Our recent blog post about observability pipelines highlighted how they centralize and enable data actionability. A key benefit of observability pipelines is users don't have to compare data sets manually or rely on batch processing to derive insights, which can be done directly while the data is in motion. As a result, teams get access to the data they need to make decisions faster.

If you’re writing software today, then you likely use a CI/CD pipeline to build and test your code before deploying it to production. Having a fast and efficient build pipeline saves you development time, shortens feedback loops, and helps you ship features faster. Conversely, slow and unreliable build pipelines are full of lost productivity and sadness.

An observability pipeline is a tool or process that centralizes data ingestion, transformation, correlation, and routing across a business. Production engineers across ITOps, Development, and Security teams use them to more efficiently and cost-effectively transform their telemetry data to drive critical decisions. Businesses of all sizes can enjoy several benefits and gain a significant competitive advantage by implementing an observability pipeline.