An incident postmortem is an excellent framework for learning from incidents and turning problems into progress. It also builds trust with customers, colleagues, and end users (basically the folks affected by the incident) and lets them know your team is working to minimize future incidents and impact.
Nothing puts a drag on IT service teams and customer support teams like answering the same question across multiple tickets. Outages and incidents have a way of sending these teams an avalanche of duplicate tickets. During a service interruption, tickets start flying in, the team can’t keep up, and end users have a bad experience made even worse.
We’ve talked about the increasingly-interconnected nature of cloud tools and the domino-goes-crashing-down effect thatcan happen when just one critical service has downtime. Web uptime is more important than ever, and it’s critical that these services we all rely on are up and running as often as possible.
The unfortunate reality about running a web service is that every now and again, you’re going to have downtime. Even the best web companies have the occasional blip in service. If downtime is inevitable, then it’s best to plan ahead so that you can be ready. After all, prior preparation prevents poor performance.
The goal of a planned maintenance announcement should be to get the right people informed and confident about the upcoming maintenance. Here’s our guide to announcing scheduled maintenance, along with some examples from our friends who are doing it well.
On October 21, 2016 at approximately 4am PST, the internet broke. OK, we know the internet doesn’t “break.” But hundreds of important services powering our modern web infrastructure had outages – all stemming from a DDoS targeting Dyn, one of the largest DNS providers on the internet.
Often I am asked how I always seem calm and poised during incidents. This persona is not entirely accurate as I am more like a duck in these scenarios, calm on top of the water, but paddling frantically under the water. I have learned some tricks that have helped me stay calm and drive incidents to resolution.
The Incident and Component status workflows are now unified in Statuspage, meaning easier incident communication and fewer steps to get status updates to the right people.
Nobody likes an unavailable website. But if you must take your site offline, a clear and well-designed maintenance page can turn around a bad experience.
It takes a village to respond to and resolve incidents. But the teams involved in incident response often work in silos: SREs and devs are heads down fixing the problem, support is flooded with emails/tickets, and marketing/PR may be putting out fires on Twitter. Even if there’s some communication happening over chat or across desks, there’s typically room for improvement with getting these teams to work together when it matters most.
