The on-call experience is sometimes a dreaded one for software engineers. Those late-night alerts and frantic Slack messages, after all, don’t exactly sound pleasant. But what’s an on-call shift really like? Is that perception of constant fire-fighting and 3 AM wake-up calls actually realistic? Michael Mandrus and Owen Smallwood, both senior software engineers here at Grafana Labs, wanted to set the record straight.

‘Manual deployment is (should be) a sin.’ Well, calling manual deployment a sin may sound strong, but consider this: building the ultimate development pipeline demands a focus on automation. Although the selection of a deployment method depends on the specific needs and requirements of a project or environment, can you really deny the power of automated deployment? There's a better way.

ITOps and DevOps teams face many challenges. Their responsibilities are extensive, from navigating complex IT environments at scale to quickly addressing performance issues and minimizing downtime and outages. Enhancing your organization’s IT service assurance requires you to ensure the reliability, performance, and availability of IT services.

Everyone experiences stress at work—thankfully, it’s a topic folks aren’t shying away from anymore. But for on-call engineers, alert fatigue is a phenomenon closer to home. Unfortunately, like stress, it can be just as insidious and drastically impact those it affects. First discussed in the context of hospital settings, this phrase later entered engineering circles.

Dealing with a flood of incidents, each with varying degrees of urgency, can be a daily struggle for Incident Response teams. Suppose a low-priority alert pings while you're tackling a critical incident. This pulls your focus away from the urgent issue. This constant alert bombardment can: How do engineers ensure that high-severity issues take precedence? Don't they want to avoid being bothered or bombarded with notifications while addressing critical matters? They sure do.

Incidents are a stressful time for your team: your service isn't working the way you expect and your customers/stakeholders want to know what's going on. The last thing you want to do is let your team improvise everything when it comes to responding to incidents. Google's own SRE book has great overall tips for incident management, part of which involves "develop(ing) and document(ing) your incident management procedures in advance", which this article dives into.

On this weeks' episode of The Debrief, we chatted with Jeff Forde, an Architect on the Platform Engineering team at Collectors. With a background spanning finance, healthcare, and various product-led startups, Forde has honed his expertise in DevOps, site reliability, and platform engineering. Beyond his professional life, he's also a dedicated volunteer first responder and certified fire instructor in Connecticut, offering him a unique perspective on managing incidents of all typesz.

Say hello to smarter incident management with smart summaries, mitigation message suggestions, and our new conversational assistant! 🚀✨

Today, incident management is a core part of organizations both big and small. But what if you don't have a program in place...where do you start? Or what if incident management is already a key part of your org, but you're looking to optimize it—where do you kick things off in that case? Consider another situation: What if you're an established organization with years of incident management experience—what are some things that you can do to take things to the next level?

Embracing digital transformation means increasing your reliance on a variety of IT systems, applications, and networks. Organizations are adopting advanced solutions like IT monitoring software and Artificial Intelligence for IT Operations (AIOps) to manage this complexity. These tools provide real-time insights into IT ecosystem health and performance, using AI and machine learning to support proactive decision-making and automation.