If there’s one thing folks working in internet services love saying, it’s: "Yeah, sure, but that won’t scale." It’s an easy complaint to make, but in this post, we’ll walk through building a service using an approach that doesn’t scale in order to learn more about the problem. (And in the process, discovering that it actually did scale much longer than one would expect.)

Rich Anakor, chief solutions architect at Vanguard, is on a small team with a big goal: Give Vanguard customers a better experience by enabling internal engineering teams to better understand their massively complex production environment—and to do that quickly across the entire organization, in the notoriously slow-moving financial services industry. They also had a big problem: The production environment itself.

This is the second post in our series about Lattice, Honeycomb’s new design system and how we’re applying a user-centric design philosophy to our product. Lattice begin! At Honeycomb, we understand that our users are often under a great deal of pressure when troubleshooting complicated issues in their applications.

Slack experienced meteoric growth between 2017 and 2020—but that level of growth came with growing pains. In his talk at the 2021 o11ycon+hnycon, Frank Chen (LinkedIn), a Slack Senior Staff Engineer, detailed one of Slack’s biggest pain points in that period: flaky tests. A flaky test returns both a passing and failing result despite no changes in the code. At one point, between 2017 and 2020, Slack’s flaky test rate reached as high as 50%.

Modern software services are expected to be highly available, and running a service with minimal interruptions requires a certain amount of reliability-focused engineering work. At the same time, teams also need to build new features and improve existing ones, so that users are delighted and don’t churn.