Operations | Monitoring | ITSM | DevOps | Cloud

Table of Contents GPU sharing with NVIDIA hardware is becoming easier to adopt in Kubernetes but it hasn’t been easier to observe. Time-slicing lets multiple workloads share the same GPU. MPS allows CUDA workloads to execute concurrently. Schedulers like KAI make it easier to manage these shared environments. But sharing a GPU introduces a problem that is easy to underestimate.

Table of Contents A few months ago, we introduced Kubex support for the KAI Scheduler to improve GPU sharing for production inference workloads. The basic model is simple: The KAI Scheduler handles placement and GPU sharing. Kubex continuously observes usage and adjusts those allocations as demand changes. KAI provides the scheduling foundation. It lets multiple workloads share a GPU while accounting for the amount of GPU each workload requests. Kubex then closes the loop.

Table of Contents For most of Kubernetes’ history, memory management has been a blunt instrument. Cross your limit, and the kernel kills your container. There has been no equivalent to CPU throttling, no graceful backpressure, just a hard stop. With Kubernetes 1.37, that changes: Memory QoS, built on cgroups v2, graduates to Beta and is enabled by default.