As the size and frequency of data pipelines scale, organizations are constantly looking for ways to reduce costs and more efficiently utilize their data infrastructure. As the leading managed platform for Spark on Kubernetes, we’ve helped several customers make the decision to move their Spark workloads to Ocean for Apache Spark, taking advantage of strategic spot instance selection, flexible pod configurations, and resource utilization tools that can correct overprovisioning.

When organizations deploy clusters — in, for example, Dev or Production environments — they may have different objectives around cost and availability. Dev clusters usually can tolerate more interruptions, as they are not running live services, but the expectation is saving money at the expense of availability.

Very early in my career, a mentor taught me an extremely important lesson: “There is no substitute for talking to your customers.” No market research, focus groups, or surveys can take the place of getting face to face with your customers and having an honest conversation.

Elastigroup, Spot by NetApp’s first product and core technology, can now be easily provisioned and managed as an AWS Quickstart Guide, created in collaboration with the AWS QuickStart team. Elastigroup for AWS is an IaaS Optimization platform in which the user can provision, manage, and scale EC2 instances to support any elastic application or load balanced workload while leveraging multiple instance–purchasing options.

With frequent software changes for hundreds of services over multiple environments, even a small change made by a developer can generate an unexpected blast radius in production. The standard Kubernetes deployment strategy ignores this modern complexity layer. DevOps organizations that maintain complex solutions have already understood that it’s not scalable and costly to manage.