Today, we’re excited to announce the Unravel Winter Release !
This winter release introduces major enhancements and improvements across the platform, including comprehensive cost management for Databricks, support for Delta Lake on Databricks, data observability for Google BigQuery, interactive pre-check before installation and upgrade. Also, major enhancements include deep dive into Spark applications and YARN resource manager, a view of all Databricks applications from a single tab, workload fit and cluster discovery report for cloud data migration, multi-cluster support for dashboard app on the Unravel app store and more.
These innovations will enable businesses to manage and simplify the complexities of modern data operations, optimize their costs, and derive more value from their data.
Here are the innovations we introduced in eight key areas of Unravel:
New features and improvements:
1. Cost 360 for Modern Data Stack: Understand and optimize your cloud data infrastructure costs to get the most out of your data cloud investments. Data operations cost control is a major concern, especially for cloud users. Most of the time the costs get unpredictable and unexpectedly overshoot the budgets.
What is Cost 360 for Modern Data Stack?
Cost 360 for Modern Data Stack refers to a comprehensive, end-to-end, single view of the costs associated with an enterprise’s data operations along with granular details about the user, team, data workload, usage type, data job, data application, compute, and resources consumed to execute the data application. Also, it ensures that every data operations team member in the organization has access to the same version of the truth about the cost of data operations today and projected future costs.
Unravel provides DataOps and data engineering teams with full-stack visibility into their complex data pipelines and the ability to see the costs incurred by different users and data applications. With this release, organizations now have a better view of the cost incurred across data operations in Databricks and can optimize their data workload on Databricks.
Better governance over Databricks cost: Unravel gives Databricks users meaningful insights into Databricks costs.
Trends: Get a better understanding of your existing cost trends and forecasts.
- View your cost trends for DBUs as well as the overall number of clusters used. This allows you to uncover interesting time periods with anomalies such as
unexpected spikes. you can further investigate the chargeback function or optimize the resources.
- View the trends of DBUs cost and number of clusters for a specific user, workspace, or team. This will assist in identifying interesting time periods with anomalies for that entity and swiftly optimize its cost.
Chargeback: With Unravel you can leverage the chargeback feature for Databricks to allocate costs to internal users to improve accountability, cut waste, and empower data teams to spend efficiently.
Budgets and Forecasts: Unravel allows you to create customized budgets that notifies you if you exceed (or if you are expected to exceed) the limits that you have set.
- Set customized budgets that fit your goals to keep your Databricks spending on track. You can define a target budget for individual users, groups, or workloads.
- Set role-based permission to make the budget view visible only to those users with the right permissions, so that not every user can see it. It’s only exposed to admin and read-only admin.
Optimization: Unravel provides you actionable insights to optimize Databricks workloads for efficiency and cost optimization.
- Insights and recommendations to optimize compute [clusters] and jobs. Visibility into estimated cost improvements to prioritize workload optimization
- Insights and recommendations to improve compute [clusters] and jobs performance. Visibility into estimated cost savings to prioritize workload optimization.
2. Databricks Delta Lake: Support for Delta Lake. Now with Unravel, you can configure the application to fetch the metadata of the Databricks Delta tables and monitor them from the Data tab of the Unravel UI. With this, you can view Delta tables related data insights on the tables page, like., Delta table format, size of tables, number of partitions created over time, table metadata etc.
3. Unravel Role-Based Access Control (RBAC): Unravel’s role-based access control allows admins to manage who has access to various views and features of Unravel and what areas they have access to. This empowers users and admins to view more and do more.
Support for custom roles and permissions:
- Define custom roles beyond the default — admin, read-only, and user.
- Define views that a role can see.
- Define data filters to apply. You can choose from user tags, app tags, app data fields, and even write an elasticsearch query filter to meet your requirements.
- Generate user tags using user tagging script.
4. Google Cloud [Dataproc and BigQuery]: Gain complete visibility into your Google Cloud Dataproc and Google Cloud BigQuery projects [private preview].
Google Cloud Dataproc Observability:
- View and understand cluster resources, jobs, and stages across Google Dataproc instances and workspaces.
- Visibility into data usage, including tables accessed by user or by application, and identification of hot/cold/warm tables and data.
- Pinpoint accuracy for runtime tunable parameters gets maximum performance from Google Cloud Dataproc.
- Unravel provides meta-intelligence about cloud server instances to right-size infrastructure for the best performance of Spark and Hadoop apps.
- Faster diagnosis and resolution of application failures, slowdowns in Spark and Hadoop applications and data pipelines
- Reduced mean time to identification and recovery from failures (MTTI/MTTR)
Google Cloud BigQuery Observability [private preview]: Unravel for Google BigQuery is available in private preview, with public preview coming shortly. Now Unravel helps you to gain insights and improve performance of your BigQuery applications, users, and infrastructure.
A private preview of support for Google BigQuery includes:
- Observability: Observe the performance of Google BigQuery and troubleshoot the issues before they impact the application underneath and the users.
- View the jobs running on the cluster.
- View details of errors encountered by the jobs.
- Governance: Google BigQuery can quickly get expensive if you are managing massive amounts of data every day and do not use cost optimization techniques.
- Gain the chargeback view of resources used in BigQuery.
- Associate jobs with business priority tags.
- Optimization: Monitor and detect slow Google BigQuery queries, and optimize inefficient queries and operations.
- Analysis of the job execution based on the resource usage and time spent.
- Visibility into data/tables usage (hot, warm, cold).
5. Cloud Data Migration: Cloud data migration can be difficult. If an enterprise does not have complete insight into its data, data users, data applications, and data dependencies, the process can become time-consuming and costly. With a data-driven assessment plan in place, an organization can minimize the time and cost of cloud data migration.
Unravel enables you to get full visibility and understanding of an organization’s data environment, highlighting the best data applications to migrate, removing migration bottlenecks, maximizing cost savings, and simplifying the process.
Cloud data migration assessment report
- Assess (basic assessment) — Make a well-informed go/no go decision to the cloud.
- Plan (deep assessment) — When an organization has decided to migrate to the cloud, deep assessment will help them to plan for a smoother and faster migration.
6. UI Enhancements: Unravel now has improved its UI to boost usability.
- Budget tab: View a list of different aspects of budgets and quickly compare budgets between different periods.
- Alert tab: Set custom alert notification to email or Slack.
7. Unravel’s Billing Services: The Billing tab in Unravel shows the charges of Unravel for its support for AWS EMR.
The following pricing plans are supported:
- Pay-as-you-go: As per this plan, Unravel tracks the number of instance hours that you have incurred and shows the charges based on the usage.
- Pay-in-advance: As per this plan, you can pay in advance for a specific number of instance hours. Credits will be taken based on the usage, with the remaining credits shown on the Billing tab daily. You can monitor when they run out of credits and plan accordingly.
8. AppStore Page: The App store feature is added from where you can manage all the Unravel applications. From the app store, you can install an application, run the administrative tasks for managing your apps, navigate to different apps, open and run the apps. Now you can get different views and insights on the telemetry data for your cluster that Unravel has collected.