As our Field CTO Liz Fong-Jones says, production excellence is important for cloud-native software organizations because it ensures a safe, reliable, and sustainable system for an organization’s customers and employees. A CoPE helps organizations cultivate the practices and tools necessary to achieve that consistently. In part one of our CoPE series, we analogized the CoPE with safety departments.

Our storage engine, affectionately known as Retriever, has served us faithfully since the earliest days of Honeycomb. It’s a tool that writes data to disk and reads it back in a way that’s optimized for the time series-based queries our UI and API makes. Its architecture has remained mostly stable through some major shifts in the surrounding system it supports, notably including our 2021 implementation of a new data model for environments and services.

Software is in a crisis. This is nothing new. Complex distributed systems are perpetually in a state far from equilibrium, operating in what Richard Cook has called a “degraded mode.” It’s through a combination of technical artifacts, organizational practices and policies, and pure gumption that they manage to maintain themselves through time. However, there are some organizations that seem to have an easier time of it than others.

Amperity required an observability partner to facilitate their transition into the modern engineering era as their previous tooling struggled to support their growth strategy. When customer data is scattered everywhere, how do you put the pieces together to get an accurate customer 360° view? That’s the power of Amperity’s customer data platform (CDP), and the company has been driving customer data innovation for nearly a decade.

In my February 2024 piece The Cost Crisis in Observability Tooling, I explained why the cost of tools built atop the three pillars of metrics, logs, and traces—observability 1.0 tooling—is not only soaring at a rate many times higher than your traffic increases, but has also become radically disconnected from the value those tools can deliver. Too often, as costs go up, the value you derive from these tools declines.

Over the past five years, software and systems have become increasingly complex and challenging for teams to understand. A challenging macroeconomic environment, the rise of generative AI, and further advancements in cloud computing compound the problems faced by many organizations. Simply understanding what’s broken is difficult enough, but trying to do so while balancing the need to constantly innovate and ship makes the problem worse.

We recently introduced relational fields, a new feature that allows you to query spans based on their relationship to each other within a trace. You can now query for spans where its root span, direct parent span, or any other single span in the trace has certain attributes. We currently support the following three prefixes: root. - Identifies the root span within a trace. To find a match, any additional root. filters in your query will search through fields only in the specified root. span.