Have you ever deep dived into the sea of your tracing data, but wanted additional context around your underlying system? For instance, it may be easy to see when/where certain users are experiencing latency, but what if you needed to know what garbage collection is mucking up the place or which allocated memory is taking a beating? Imagine having a complete visual on how an application is performing when you need it, without having to manually dig through logs and multiple UI screens.

Hey Perplexed, We’re huge advocates for testing in production, and that includes performing chaos engineering/continuous verification in production. But you’re right to want to be cautious about exactly how you are performing these tests. After all, it’s chaos engineering, not just pure unbridled chaos.

Today, I found a bug before I noticed it. Like, it was subtle, and so I wasn’t quite sure I saw it—maybe I hadn’t hit refresh yet? Later, I looked at the trace of my function and, boom, there was a clear bug. Here’s the function with the bug. It responds to a request to /win by saving a record of the win and returning the total of my winnings so far. Can you spot the problem in the TypeScript? It’s subtle. Now here’s a trace in Honeycomb: Now do you see the bug?

I have a good sense of how to use traces to understand my system’s behavior within request/response cycles. What about multi-request processes? What about async tasks spawned within a request? Is there a higher-level or more holistic approach?

When you send telemetry into Honeycomb, our infrastructure needs to buffer your data before processing it in our “retriever” columnar storage database. For the entirety of Honeycomb’s existence, we have used Apache Kafka to perform this buffering function in our observability pipeline.

One of the most common questions we get at Honeycomb is “What insights can you get in the browser?” Browser-based code has become orders of magnitude more complex than it used to be. There are many different patterns, and, with the rise of Single Page App frameworks, a lot of the code that is traditionally done in a backend or middle layer is now being pushed up to the browser. Instead, the questions should be: What insights do frontend engineers want?

Dear Trapped, Thanks for asking the question! Approaching observability as an all-or-nothing problem often leads to the project feeling daunting. But that’s not specific to observability—any project can be overwhelming if you think it needs to be done all at once, perfectly. Such as, erm, writing an entire book on observability! *looks around worriedly*

First, I’d like to say that pager duty isn’t something we should treat like chronic pain or diabetes, where you just constantly manage symptoms and tend to flare-ups day and night. Being paged out of hours is as serious as a fucking heart attack. It should be RARE and taken SERIOUSLY. Resources should be mustered, product cycles should be reassigned, until the problem is fixed.

From September to early October, Honeycomb declared five public incidents. Internally, the whole month was part of a broader operational burden, where over 20 different issues interrupted normal work. A fraction of them had noticeable public impact, but most of the operational work was invisible. Because we’re all about helping everyone learn from our experiences, we decided to share the behind-the-scenes look of what happened.