Operations | Monitoring | ITSM | DevOps | Cloud

Serving large language models at scale is fundamentally a distributed systems problem. A single GPU, or even a single node, is rarely enough once you need multiple models, multiple replicas, tensor-parallel sharding across GPUs, or high-availability rollouts. Kubernetes solves general container orchestration well, but it has no native concept of a GPU-aware, actor-based compute cluster.

The story of 2026 engineering has a plot twist that on-call teams saw coming. AI coding assistants now write a large share of production code, pull requests per developer are up, and delivery velocity looks great on the dashboard. Then the pager goes off.

AI-related outages just moved from a fringe worry to a mainline reliability problem, and the on-call rotation is where that shift lands first. A new StackGen analysis of nearly 178,000 public status-page records found that incidents disclosed by AI model and AI application companies now account for more than one in ten reported outages, a sixfold jump from 1.7 percent in 2023 to 10.7 percent so far in 2026.

Your monitoring console probably covers the switches, the hosts, the VMs and the application traces. The database tier is the gap. It tends to live in a separate tab. Somebody opens that tab once the incident bridge has already started. That gap got more expensive this year. On 21 April 2026, Oracle moved MySQL 8.0 to Sustaining Support. The version most production estates still run no longer gets new fixes. Good MySQL monitoring tools close the gap.

What do you do when the switch carrying most of your traffic is the one device that cannot tell you what is on it? On high-speed core and data centre links, full flow export pushes device CPU past a comfortable line, so the export gets switched off and the busiest segment quietly becomes the least visible one. sFlow was built for that exact situation.

CloudZero had a full team on the ground at Ai4 in Las Vegas during the first week of August 2026. The team included CTO Erik Peterson, who spoke on a panel about AI cost economics. The same problem surfaced everywhere we went: teams can see what they’re spending, but not whether it’s working. DIY cost tooling that fails time and time again, agent sprawl, and a widening gap between finance and engineering kept coming up throughout the week.

AI tokens are the small chunks of text, roughly four characters or three quarters of a word each, that language models read and generate. Every prompt and every response is measured in tokens, and AI providers bill per million of them. That makes the token the base unit of AI spend: 1,000 tokens is about 750 words, and every AI feature you ship is a token meter running.

In 2025, 85% of organizations increased AI investment, and 91% plan to do the same this year, according to Deloitte. Despite continued spending, however, ROI lags behind, with just 6% seeing payback within one year. While AI use cases tend to have a longer payback period, often in the 2-4 year range, companies can’t afford to keep spending money without some measure of its practical impact both immediately and over time.

You know exactly where you want to go in CloudZero. Getting there sometimes takes a moment as you click into the nav, open a menu, scroll a dropdown, find the thing, click again. Every trip back to a familiar spot can take a few steps. Shortcuts remove that friction. Press command+K on Mac or ctrl-K on Windows anywhere in CloudZero, type where you want to go, and hit Enter. That means there’s no clicking through the nav and no scrolling to find what you already know the name of.

AI coding assistants have changed how fast application code gets written. Copilot, Cursor, Claude Code, and agentic tools built on top of them can generate a working feature in minutes. But none of them know your database's history. They don't know that a migration already renamed that column last sprint, that a policy forbids unqualified DELETE statements, or that the target environment has drifted from what your migrations say it should look like.