Data Teams Need Help Controlling Cloud Costs
A new approach to taming cloud data costs
Cloud costs are choking innovation
Data FinOps for modern data stacks
First-generation FinOps tools were built to optimize web applications. Now modern data stacks enable faster analytics and AI, but costs are also on the rise. Unoptimized Databricks workspaces and clusters, BigQuery slots and I/O, and over-provisioned Snowflake clusters have cropped up all over the organization and cloud data spending has skyrocketed, making data management the biggest and fastest-growing cloud cost—about 40% of the typical cloud bill—leading to cloud cost overruns. DataFinOps brings FinOps to the modern data stack.
What causes cloud data cost overruns?
It’s like stumbling in the dark
Most organizations can’t even see why their costs are growing. Before you can improve the efficiency of your cloud data spend, you first need full visibility into where spending is going and how teams are tracking towards their budgets.
Enterprise data estates frequently have millions of rows of cloud billing data that must be aggregated and analyzed before you can clearly see current spending patterns. Aggregating and analyzing millions of rows of cloud billing data can be a nightmare.
Every element of your cloud data estate should be categorized, tagged, and brought together into a single view. Cloud services don’t automatically do this work, yet it has to be done in order to see spending by team, pipeline, data application, product, or business division.
Without visibility into cloud data resource consumption and spending, you have no way to measure improvements.
Forecasting is a guessing game
The variable consumption model of the cloud enables greater agility and flexibility. What makes forecasting difficult is that resources are dynamically provisioned and consumed through code.
Across the organization, there is no single source of truth. Data engineering teams use one set of tools. DataOps teams use a separate set of tools. And Finance runs yet another set of tools.
Each team operates in silos, using different tools that lead to an information mismatch, confusion, and frustration. Forecasts end up 20-70% off base.
Without open and consistent communication between engineering, operations, finance teams, and executives, projecting cloud data spend is impossible.
Inability to identify waste and inefficiency at scale
Cloud services are fundamentally different from on-premises systems and require much more efficient utilization in order to achieve cost savings.
Turning off idle clusters is a great place to start, but most enterprise data pipelines and applications require manual tuning. This is a resource-intensive, manual process that requires more time and resources than most organizations have to spare.
Cloud resource optimization is so difficult that enterprise executives estimate 32% of cloud spending is wasted each year (Flexera). Modern data stacks are resource-intensive and figuring out exactly which resources the data app/pipeline actually needs to run is complicated.
Who’s in charge?
The same flexibility that enables faster business results also leads to cloud data cost overruns. When the cloud bill arrives, it’s too late to take corrective action.
Defining guardrails for data workloads while maintaining trust between teams is difficult. You don’t want finance teams to arbitrarily set limits that impact performance or quality of critical data pipelines.
Without well-defined limits in place, it’s far too easy for inexperienced or junior data team members to have a job go off the rails, leading to massive cost overruns.
Data FinOps
FinOps integrates financial accountability into cloud operations. DataOps brings lean and agile approaches to data management. DataFinOps combines the best of DataOps and FinOps to help you maximize the return on your modern data stack investment.
Unravel Data is a data observability platform that simplifies and accelerates each phase of the DataFinOps lifecycle–from observability to optimization and governance. Innovative companies such as Mastercard, Equifax, and DBS Bank use Unravel for cloud cost management and FinOps. Unravel provides data teams a unified view of data pipeline and application performance, cost, and quality.
Data FinOps customer success
Equifax uses Unravel to enable “self-service” efficiency/cost optimizations across its massive Google Cloud data ecosystem.
Global logistics giant Maersk tackled cloud data costs at speed and scale up-front in dev–realizing a 70% savings in 6 months.
DBS Bank uses Unravel to validate cloud data resource usage before deploying code into production.
Mastercard data application owners use Unravel to self-tune their queries, alert on “toxic” workloads, and prevent risky workloads from running at all.
Unravel is a member of the FinOps Foundation
The practice of FinOps, with its collaborative and iterative approach to observability, optimization, and governance can help data teams increase the return on their modern data stack investments by proactively spotting opportunities for efficiency, adding alerts, and automating guardrails.
See how to use Unravel for DataFinOps
Jumpstart your DataFinOps journey by attending an upcoming event with Unravel. Creating a culture of communication between executives, finance, business, and engineering teams is a key FinOps capability. Begin by collaborating on a single source of truth. Improve performance, cost, and quality with DataFinOps using Unravel.
