The overall theme is high-performing engineering teams are generally the ones that humanize the process. Whether you’re trying to increase productivity or release better-quality code, the biggest piece of advice is to lead with empathy.
It’s harder to understand and operate production systems in 2021 than it was in 2001. Why is that? Shouldn’t we have gotten better at this in the past two decades? There are valid reasons why it’s harder: The architecture of our systems has gotten a lot more sophisticated and complex over the past 20 years. We’re not running monoliths on a few beefy servers these days.
Ooh, good question! My favorite thing about this part of the year is that work slows down, everybody is on vacation, and those of us not traveling get to work on little projects that we’re too busy to touch most of the year. As Martin Thwaites put it: “The Product Owners are away, the devs will play.” For Martin, this year, “play” means adding tracing to more of their services.
At Honeycomb Developer Week, I got an opportunity to walk through a couple of fun new features we’ve shipped since August and ways that we’ve been able to improve Honeycomb for you. Hearing feedback from our users and customers— through support requests, in the Pollinators community, from Twitter, etc.—helps us make Honeycomb better for you.
Getting started with observability can be time consuming. It takes time to configure your apps and practice to change the way you approach troubleshooting. So it can be hard to prioritize investing time, especially if you can’t clearly see how that investment will pay off. That’s why we put together Honeycomb Developer Week: short, snackable, time-efficient learning sessions to jumpstart your observability journey.
Have you ever deep dived into the sea of your tracing data, but wanted additional context around your underlying system? For instance, it may be easy to see when/where certain users are experiencing latency, but what if you needed to know what garbage collection is mucking up the place or which allocated memory is taking a beating? Imagine having a complete visual on how an application is performing when you need it, without having to manually dig through logs and multiple UI screens.
Hey Perplexed, We’re huge advocates for testing in production, and that includes performing chaos engineering/continuous verification in production. But you’re right to want to be cautious about exactly how you are performing these tests. After all, it’s chaos engineering, not just pure unbridled chaos.
Today, I found a bug before I noticed it. Like, it was subtle, and so I wasn’t quite sure I saw it—maybe I hadn’t hit refresh yet? Later, I looked at the trace of my function and, boom, there was a clear bug. Here’s the function with the bug. It responds to a request to /win by saving a record of the win and returning the total of my winnings so far. Can you spot the problem in the TypeScript? It’s subtle. Now here’s a trace in Honeycomb: Now do you see the bug?
I have a good sense of how to use traces to understand my system’s behavior within request/response cycles. What about multi-request processes? What about async tasks spawned within a request? Is there a higher-level or more holistic approach?
When you send telemetry into Honeycomb, our infrastructure needs to buffer your data before processing it in our “retriever” columnar storage database. For the entirety of Honeycomb’s existence, we have used Apache Kafka to perform this buffering function in our observability pipeline.
