Dear Miss O11y, I’ve been told I need to use the OpenTelemetry Collector, but I have no idea what it is, or why I need it? Should I be using it? What value does it add? What the hell am I missing?
More often than not, as developers, when we get a report that a large customer is hitting 502 errors, there's a flurry of activity. What's wrong? Is something deeply broken? So you start digging through AWS logs to see what you can find, but it's hard to reproduce. Sometimes, there's no clear answer, and you move on without any resolution. What if I told you it doesn't have to be this way?
Dear Miss O11y, How is modern observability using tracing, such as Honeycomb, different from the previous distributed tracing software I'm familiar with, like Dapper, at my company? I haven't really been able to wrap my head around Dapper. Does "advanced" observability mean that it's even more complicated than Dapper is? Auntie Alphabet.
The book is done. The book is FINISHED. There will be no more writing of the book! Holy macaroni. If you think you might be sensing just a SMIDGE of relief, you wouldn’t be wrong. We started writing this book, Observability Engineering: Achieving Production Excellence, three whole years ago, in the summer of 2019. Remember that?
When do you delete instrumentation? You delete instrumentation when you delete code. Other than that, if you’re doing things right: almost never. One of the best things about honeycomb is that it completely transforms the incentives around preserving instrumentation. With metrics-based tools, the most valuable metrics are always custom metrics. You need to define a custom metric for literally any question you might ever want to ask about the app and its utilization or performance.
Any significant shift in an organization’s software engineering culture has the potential to feel tectonic, and observability (o11y for short)—or more specifically, Observability Driven Development—is no different. Leaning into observability, which calls for tool-enhanced investigation, hypothesis testing, and data richness can be cumbersome even for the most veteran of teams.
Miss O11y is delighted to welcome our newest band member: Martin Thwaites! Martin has been a member of the Honeycomb user community practically since its inception. He is a UK-based consultant who specializes in helping teams scale up and tackle challenging business problems, and a long-time contributor to the Azure and.NET communities. We think he looks ✨amazing✨ in a tiara.
It’s time we had a real conversation about why UX designers everywhere are still unhappy, why that elusive “seat at the table” feels so impossibly out of reach to so many (even at companies that embrace design), and how this impacts your business Design is facing down an epidemic of designers who feel burned out, taken for granted, marginalized, and disrespected. Yes, there is something different about our experience compared to other disciplines.
Ah, good question! TL;DR: Trace instead of log. Traces show connection, performance, concurrency, and causality. Logs are the original observability, right? Back in the day, I did all my debugging with `printf.` Sometimes I still write `console.log(“JESS WAS HERE”)` to see that my code ran. That’s instrumentation, technically. What if I emitted a “JESS WAS HERE” span instead? What’s so great about a span in a trace? Yeah, and so do logs in any decent framework.
I always tell people "Observability is not logs, metrics, and traces! Observability is empowering your team to ask questions." That's very aspirational and sounds good, but it's not at all clear. I now have a self-contained story that perfectly explains it!
