As almost 90% of organizations are executing on a multi-cloud strategy for migrating their data and analytics workloads to the cloud, the term “modern data stack” continues to gain more traction. A modern data stack is a suite of technologies and apps built specifically to funnel data into an organization, transform it into actionable data, build a plan for acting on that data, and then implement that plan.

In part one of this two-part blog post, we began our deep dive into Apache Spark tuning to optimize resources. We looked at what is involved in executor and partition sizing, particularly choosing the number of partitions and choosing an executor size. After establishing some principles of optimization here, we ended by asking an important question: Is it really practical for all applications to be optimized? As our recent State of the Market report helped reveal, the answer is two-sided. The good news?

As our recent survey showed, Apache Spark is poised to continue as big data’s most dominant large-scale big data processing platform. Thus it is imperative that Spark users learn and master Spark tuning if they want to get the most out of their Spark environments. But what is tuning in Spark? How is it done? Read on to know more about Spark tuning.

Magnite, the biggest independent sell-side advertising solution, empowers publishers with their innovative monetization technology. Trusted by the world’s foremost agencies and brands, Magnite’s advertising technology provides publishers access to brand-safe, high-quality ad inventory, powering billions of advertising transactions every month. In 2021, Magnite’s revenue exceeded $416.5 million.

Here at Pepperdata, we put a premium on individuals whose skills and experience can help us reach our goals. We believe our team is composed of the best people in their respective fields. Our “Pepperdata Profiles” series gives a spotlight on our gifted individuals and allows us to highlight our employees’ experiences.

Site reliability engineering (SRE) is fast becoming an essential aspect of modern IT operations, particularly in highly scaled, big data environments. As businesses and industries shift to the digital and embrace new IT infrastructures and technologies to remain operational and competitive, the need for a new approach for IT teams to find and manage the balance between launching new systems and features and ensuring these are intuitive, reliable, and friendly for end users has intensified as well.

Self service analytics are becoming increasingly popular and essential in this data-driven world. For many businesses, there is a growing need for their internal departments to access their data and business intelligence and harness its power themselves. Traditionally, business intelligence processes are the purview of IT teams and data specialists.

One of our clients is a software developer that specializes in design and manufacturing software solutions. This software firm caters to some of the biggest organizations operating in primary sectors like engineering, architecture, manufacturing, entertainment, and more. Every new software and update our client rolls out must meet stringent SLA requirements.

A change in leadership is an exciting phase for any organization. Here at Pepperdata, we are excited to announce that we have a new CEO at the helm: Maneesh Dhir. Our former CEO, Ash Munshi, will assume the role of Executive Chairman of the company’s board of directors, following five years of continued growth. We are thrilled to have Maneesh as our new chief executive and look forward to his extensive experience and expertise in the tech space helping us thrive in the coming months and years.

Today, we have some exciting news to share: Pepperdata Capacity Optimizer now supports Azure Kubernetes Service (AKS). This means users of AKS are able to automatically optimize their workloads running on the Azure Kubernetes Service. AKS users can now rely on Pepperdata to help them monitor and optimize Spark applications on Kubernetes AKS. This evolution of our platform has been in the works for some time, and we know our users will be delighted by this development.