The cloud native revolution brought by Kubernetes has transformed the way we build and deliver software, but the world of big data has for too long been left on the side of this transformation. Thanks to many contributions from the open source community, Apache Spark integration on top of Kubernetes is now officially generally available with the recent releases this year.

In the relatively short window of time that Kubernetes has been around, it’s rapidly matured as a critical technology foundation for the cloud, and now even applications that were previously thought to be unviable for containers are running with Kubernetes. As companies expand their usage of it, the need to operationalize Kubernetes with automation and optimization is critical to maintaining speed, agility and control in the long-term.

The dynamic nature of cloud native applications is both a blessing and a curse. The ability to use compute, storage, and network resources without managing physical hardware is a real blessing. Your applications can take advantage of the seemingly limitless resources available in the public cloud. Unfortunately, the curse becomes clear when the bill arrives! It is a significant CloudOps challenge to find the balance between providing optimal application performance and minimizing cost.

Spot by NetApp’s Ocean continually optimizes a Kubernetes cluster’s data plane. When Ocean identifies nodes that aren’t being fully utilized, it simulates various bin packing scenarios. Once a more efficient layout is identified, Ocean reschedules pods, and then gracefully terminates the underutilized nodes. Ocean’s continuous optimization of cluster infrastructure saves your organization money and is better for planet Earth!