Today, users and customers are driven by response rates to their online requests. It’s no longer good enough to just have a request run to completion, it also has to fit within the perceived limits of “fast enough”. Yet, as we continue to build cloud-native applications with microservice architectures, driven by container orchestration like Kubernetes in public clouds, we need to understand the behavior of our system across all aspects, not just one.

This is the story of how I used Honeycomb to troubleshoot redis/redis-rb#924 and discovered a surprising workaround.

Dark is a programming language and platform that enables building serverless backends. There’s no infra, framework or deployment nightmares. It’s a new paradigm in software delivery. As a startup, the Dark team is constantly making decisions about where to invest in improvements to support customer needs. With Honeycomb, they can observe user behavior and make business decisions based on meaningful data.

Monitoring — by its very nature — requires privileged access to internal and external services. In order to safely maintain visibility into critical systems, it’s vital to have some form of secrets management to manage authentication credentials (AKA, "secrets"), including passwords, keys, APIs, tokens, and any other sensitive pieces of information in your IT infrastructure.

On May 27, the first OpenObservability Conference was held to bring together leaders, practitioners, and users of leading open source observability tools for sessions on the experiences, strategies, and future of the industry. For the Logz.io team, as long-time proponents of open source, it was rewarding to see everyone come together to explore the challenges and opportunities of open source observability.

The most successful software development movement of my lifetime is probably test-driven development or TDD. With TDD, requirements are turned into very specific test cases, then the code is improved so the tests pass. You know it, you probably use it; and this practice has helped our entire industry level up at code quality. But it’s time to take a step beyond TDD in order to write better software that actually runs well in production. That step is observability driven development.

Are you already applying IT observability to keep up with rapid changes in your IT operations landscape? Currently, the fast adoption of new infrastructures, including hybrid clouds, containers, and microservices, challenges the market. As organizations move towards these highly dynamic architectures, the requirements for traditional IT monitoring change dramatically. More data keeps on coming, and having the time and skill to keep up with this ongoing stream seems to get harder and harder.

Practicing observability isn’t just about tools. It also means improving how you work together and how you share lessons across the team. Learning from each other helps everyone on your team become better engineers that can create amazing experiences with code, or that make code work at incredible scale (or both!). Writing software and operating it in production is—and must be—a team sport.

During World War II, a mathematician named Abraham Wald worked on a problem – identifying where to add armor to planes based on the aircraft that returned from missions and their bullet puncture patterns. The obvious and accepted thought was that the bullets represented the problem areas for the planes. Wald pointed out that the problem areas weren’t actually these areas, because these planes survived.