Beyond Visibility: How Unravel Transforms Databricks System Tables from Dashboards into Action

Why excellent observability data requires intelligent automation to drive real optimization impact

If you’re managing a Databricks environment, you’ve likely discovered the power of System Tables. These comprehensive data sources—spanning schemas like system.billing.usage, system.query.history, and system.compute.clusters—provide unprecedented visibility into your data platform’s operations. Many organizations initially believe these tables, combined with internal dashboards, can deliver all the data observability capabilities they need.

They’re both right and wrong.

System Tables are a remarkable foundation for observability, offering detailed insights into costs, performance, and usage patterns. But there’s a critical gap between seeing what’s happening and optimizing what needs to change. This gap represents the difference between reactive monitoring and proactive optimization, between dashboards that inform and systems that transform.

The Promise and Reality of System Tables

Databricks System Tables deliver on their core promise: comprehensive visibility. Through schemas like:

• system.billing.usage: Hourly DBU consumption with workspace and user attribution
• system.query.history: Detailed query execution metrics and resource consumption
• system.compute.clusters: Cluster lifecycle and configuration data
• system.access.audit: Service-level event tracking and command duration analysis

Organizations can build sophisticated dashboards showing spend by workspace, user, cluster type, and time period. They can track resource utilization, monitor job performance, and create detailed cost allocation reports.

But here’s where the limitations become apparent.

Consider a large food & beverage company’s recent evaluation of its Databricks observability needs. Despite having Overwatch configured across 200 workspaces, feeding data into comprehensive dashboards, they identified critical gaps in their $24 million annual Databricks investment:

• No optimization recommendations despite clear visibility into inefficiencies
• No cost forecasting to prevent budget overruns before they occur
• No proactive alerting for anomalous usage patterns
• No automated remediation to implement identified optimizations

As the company noted in their assessment: “Databricks Dashboards are easy to implement [but] do not provide any kind of recommendations or forecasting of cost.”

The Actionability Gap: Where Expertise Becomes a Bottleneck

The fundamental challenge isn’t data availability—it’s data actionability. System Tables provide the raw materials for optimization, but transforming that data into concrete improvements requires two scarce resources:

1. Deep Technical Expertise

Identifying the root causes of performance issues or cost inefficiencies requires specialized knowledge of:

• Spark optimization principles and execution plans
• Databricks pricing models across different instance types
• Cluster configuration best practices for different workload patterns
• SQL and code optimization techniques

2. Manual Analysis Time

Even with excellent dashboards, optimization requires hours of manual investigation:

• Correlating performance metrics with cost data
• Analyzing query patterns to identify optimization opportunities
• Determining optimal cluster configurations for specific workloads
• Implementing and testing optimization changes

For large-scale deployments, this manual approach becomes unsustainable. Organizations managing 3,000+ clusters simply cannot dedicate the necessary expert hours to optimize every workload continuously.

How Unravel Transforms System Tables into Optimization Engines

Rather than replacing Databricks System Tables, Unravel builds upon them—using System Tables as the foundational data source while adding the intelligence and automation layer that drives real optimization impact.

Built on Databricks, Powered by System Tables

• Databricks-Native Foundation: Unravel runs on Databricks infrastructure, ensuring security and compliance alignment
• System Tables Integration: Uses system.billing.usage, system.query.history, and other schemas as primary data sources
• Secure by Design: Data never leaves your Databricks environment, maintaining your security posture
• API Integration: Leverages Databricks APIs for automated optimization implementation

Adding AI-Powered Intelligence

Where System Tables provide data, Unravel adds intelligence:

Predictive Cost Management: ML models trained on billions of job executions forecast budget overruns days ahead. Instead of discovering cost escalations after they occur, teams receive proactive alerts with recommended mitigation strategies.

Automated Root Cause Analysis: When performance issues arise, Unravel automatically correlates metrics across System Tables to identify specific causes—whether it’s inefficient joins, suboptimal cluster configurations, or data skew issues.

Intelligent Optimization Recommendations: Rather than requiring manual analysis to identify inefficiencies, Unravel automatically generates specific, actionable recommendations with estimated impact and implementation steps.

Driving Automated Actionability

The critical differentiator is Unravel’s ability to move beyond recommendations to automated implementation:

Policy-Based Optimization: Automatically implement approved optimizations like cluster rightsizing and auto-scaling adjustments through Databricks policies.

SQL Query Optimization: Provide before/after query rewrites with performance improvements, enabling teams to apply optimizations automatically.

Resource Scheduling: Optimize cluster startup/shutdown timing based on actual usage patterns identified in System Tables.

Real-World Impact: From Reactive to Proactive

The difference between System Tables’ visibility and Unravel’s actionable optimization becomes clear in real-world results:

A global logistics company with a similar scale achieved:

• 70% cost reduction in six months on cloud data spend
• 20× ROI: Every $1 invested saves $20
• 75% automation time savings across 3,000 data engineers

These results weren’t achieved by replacing System Tables, but by transforming their insights into automated actions.

The Partnership Advantage

Unravel and Databricks operate as strategic partners, not competitors:

• Complementary Capabilities: Databricks provides the platform and foundational observability data; Unravel adds the optimization intelligence and automation layer.
• Shared Customer Success: Both companies benefit when customers achieve better performance and cost efficiency on Databricks.
• Continuous Innovation: As Databricks releases new features and System Table schemas, Unravel integrates these capabilities to provide enhanced optimization opportunities.
• Technical Integration: Unravel’s Databricks-native architecture ensures seamless operation within existing Databricks governance and security frameworks.

Making the Most of Your Databricks Investment

If you’re currently using System Tables for Databricks observability, you’re on the right track. The question isn’t whether to continue using System Tables—it’s how to maximize their value.

System Tables Excel At:

• Comprehensive cost visibility and attribution
• Historical performance analysis
• Usage pattern identification
• Compliance and audit reporting

Unravel Adds:

• Predictive analytics and forecasting
• Automated optimization recommendations
• Root cause analysis and correlation
• Implementation automation and guardrails

The Path Forward

The most successful Databricks deployments combine the best of both worlds: Databricks System Tables provide the foundational visibility, while Unravel transforms that visibility into automated optimization that drives real business impact.

For organizations managing significant Databricks investments—whether it’s a large food & beverage producer’s $24 million annual spend or smaller but growing deployments—the actionability gap represents both a challenge and an opportunity. System Tables give you the data to see what’s happening. Unravel gives you the intelligence and automation to optimize what happens next.

The result? Dashboards that don’t just inform—they transform.

Ready to transform your Databricks System Tables from visibility into action? Learn how Unravel’s AI-powered optimization platform can complement your existing observability infrastructure while driving measurable cost savings and performance improvements.