In many cases — as you’re monitoring a particular state of a system — you probably know some steps to triage or in some cases automatically fix the situation. Let’s take a look at how we can automate this using check hooks and handlers.

In my previous post, I talked about the real costs of alert fatigue — the toll it can take on your engineers as well as your business — and some suggestions for rethinking alerting. In part 2 of this series, I’ll share some best practices for fine-tuning Sensu to help reduce alert fatigue.

Alert fatigue occurs when one is exposed to a large number of frequent alarms (alerts) and consequently becomes desensitized to them. This problem is not specific to technology fields: most jobs that require on-call, such as doctors, experience it in slightly different manners, but the problem is the same.

At Monitorama 2018, I shared some of the cool process and knowledge I’ve learned from developing a product for people other than myself to consume. After spending six years on call, I now build software that wakes people up in the night — AKA, infrastructure and tooling for systems monitoring and performance analysis. As someone who’s been there, I’m conscientious about building quality software that people delight in using.

NGINX can be used as a proxy to provide authenticated access to specific endpoints for any RESTful service API — including the Sensu API. Below I provide an NGINX configuration to grant external service provides narrow access to only create check results in the Sensu 1.4 API external service providers. But first, here's some backstory of how I got here.

At Sensu, we define monitoring as "the action of observing and checking the behavior and outputs of a system and its components over time." Essentially, you want to be made aware of a problem before your users are.