Who's ultimately responsible for reliability? "You need an executive champion that cares about this. And to me, it's the CTO. The CTO is responsible for the quality of the code that you're writing, the quality of the customer experience, the quality of the product. And so, you know, your software doesn't work. The quality is zero. Not half points here. If you can't use it, it doesn't work.

Check out how Nagarro used Gremlin to help a client prevent a cascading failure before it caused an outage. "Once we had tested a critical software that was doing millions of online transactions on a daily basis. The design was fail safe, providing redundancy on critical services by having multiple instances deployed on different VMs. What we did was we ran a virtual machine terminate test to bring down an instance of that service with the hypothesis that it will recover automatically. Well, the service did recover automatically, but the system saw a cascading failure.

Are you making really reliability a priority? Or are you just giving it lip service? "At Amazon, I was part of the retail website. Outages were lost money, lost money was bad. So Amazon cared deeply about this. That was part of it. The other part was it was part of the engineering culture. When I arrived, one of the things I was told was, we expect you to write high quality, performant, efficient, available code. It's just everybody.

Part of the Gremlin Office Hours series: A monthly deep dive with Gremlin experts. Ad-hoc Chaos Engineering experiments are great for learning more about how your systems work, but they don’t tell you how your systems behave over time. As new features get deployed, environments change, and regressions get introduced, even the most resilient systems can gain reliability risks. QA and performance testing are already built into CI/CD - why not reliability?