As usual, the latest release of Unravel (4.7.8.0) introduces new features and a slew of improvements and enhancements. As always, these new capabilities reflect the emerging needs of data team practitioners and their executive teams. Three of the new features are summarized below. Complete Release Notes for Unravel v4.7.8.0 can be found here.
The top three things any data-forward enterprise cares about are Performance, Cost, and Quality. That reliable results get delivered on time, every time, in the most cost-effective manner. Unravel’s strength has always been in performance observability, and we’ve taken the lead in cost governance for data workloads (DataFinOps). Now we bring in the third pillar: data quality.
Unravel integrates with Data Quality solutions
Unravel now integrates and correlates external data quality checks within its AI-driven insights. Unravel has taken the path of integrating data quality checks that run outside of Unravel—all the homegrown checks, assertions built in any number of excellent solution vendor tools—rather than building yet another data quality tool inside Unravel.
The first data quality integration is with the open source leader, Great Expectations.
Now data teams have insights and details about performance, cost, and quality in a single pane of glass. No more jumping from tool to tool. And as different personas care about these different dimensions, everybody is working from the same single source of truth. With one click, drill down into quality check results, timelines, lineage, partitions, schema, usage, size, users, and more—right alongside AL recommendations and insights for performance and cost.
AutoActions for Amazon EMR help control cloud costs
AutoActions can now monitor EMR apps and clusters, and you can set the AutoAction policy to generate alerts when EMR exceeds a set dollar threshold, a running time threshold, or when it’s been idle for a long time.
Unravel gets EMR cost data while clusters are running, so you get a notification in real time that the cluster is exceeding a threshold that impacts costs—not after the fact.
You can get a list of all threshold violations across the various clusters, their cost, with drill-down details into each cluster.
Check out this video with a sample use case.
Get proactive budget-tracking alerts and notifications
Unravel can now proactively send notifications—via email or Slack—to specified individuals or groups whenever certain conditions are triggered. Perhaps most usefully, this feature gives both individual users and their managers a heads-up when projected costs are estimated to be at risk of exceeding budget (or have already done so). A common, reusable notification module can be used for multiple purposes, and a single notification may be used by multiple budgets.
Costs may be expressed in DBUs (for Databricks) or dollars (for Amazon EMR). To avoid alert storms and notification fatigue, only one alert is generated per month for budgets at risk, and one per month for exceeded budgets. (But if budgets are changed, alerts can be triggered again for the same month.)
Explore this feature further in this video.