Databricks Serverless Compute Limitations

Databricks serverless compute limitations are the specific capabilities that are not available on serverless compute the way they are on classic clusters. Here is the current list, sourced directly from Databricks' own documentation.

At a glance

Environment variables: Not supported on serverless compute. Use widgets to create job and task parameters instead.
Init scripts and compute policies: Not supported. Dependencies are configured per notebook or per job task through the Environment panel instead.
Scala and R in notebooks: Not supported in notebooks, which are Python and SQL only. Scala still works through JAR tasks in jobs; R is not supported on serverless in any form.
Spark UI and Spark logs: Not available. Use the query profile to inspect query execution instead.
DataFrame and SQL caching: Not supported. Calling .cache(), .persist(), CACHE TABLE, or similar raises an exception.
External data access: Required for connecting to external data sources; direct DBFS access and DBFS mounts with AWS instance profiles are limited.
Streaming triggers: Only Trigger.AvailableNow() and the deprecated Trigger.Once() are supported. Trigger.Continuous() and Trigger.ProcessingTime() are not.
Instance pools: Not supported at the compute level.

Serverless environment and configuration limitations

Environment variables are not supported on serverless compute. Where a classic cluster might rely on an environment variable for configuration, Databricks recommends using widgets to pass job and task parameters instead.

Compute-scoped init scripts, compute policies, and instance pools are also not supported, since there is no persistent cluster to configure. Instead, teams configure their own serverless environments per notebook or job task rather than a shared cluster-level setup, choosing a base environment and adding dependencies through the Environment side panel. Databricks caches the resulting virtual environment, so dependencies do not need to reinstall every time a notebook reconnects. Teams can also build and reuse a custom environment specification (a YAML file), and workspace admins can configure shared base environments or private package repositories across all serverless environments in the workspace.

Serverless notebook and language limitations

Serverless notebooks support Python and SQL only. R is not supported, and while JAR libraries are not supported inside notebooks, JAR tasks in jobs still work. Serverless compute also runs on Spark Connect rather than Spark RDD APIs, which can change the behavior of some existing code, particularly around when analysis and name resolution happen.

A few smaller notebook-specific gaps: there is no Spark UI or Spark log access (the query profile covers query-level detail instead), no autocomplete or Variable Explorer for dataframes, and notebook-scoped libraries are not cached across separate development sessions the way job dependencies are. PySpark itself should never be manually installed as a notebook dependency, since doing so stops the session.

Serverless data and storage limitations

Serverless compute requires Unity Catalog to connect to external data sources; direct cloud storage access goes through Unity Catalog external locations instead of the classic-cluster mount pattern, and DBFS access is limited in favor of Unity Catalog volumes or workspace files. Maven coordinates and global temp views are not supported either.

Supported data sources are broader for reads than for writes. Write operations (create, update, delete) are limited to CSV, JSON, AVRO, DELTA, KAFKA, PARQUET, ORC, TEXT, XML, BINARYFILE, Unity Catalog tables, and Iceberg. Read operations add a longer list of external connectors, including Snowflake, BigQuery, Redshift, Salesforce, MongoDB, and several others. Hive SerDe tables and Hive variable syntax (for example ${env:var} or ${configName}) are not supported on serverless in either direction.

Serverless caching and session limitations

DataFrame and SQL cache APIs are not supported on serverless compute. Calling .cache(), .persist(), .unpersist(), .checkpoint(), or SQL commands like CACHE TABLE and CLEAR CACHE all raise an exception rather than silently no-op. Separately, serverless sessions cache metadata, so switching catalogs mid-session may not fully reset the session context; resetting the compute resource or starting a new session clears it.

Serverless jobs and streaming limitations

Within a single job run, task logs are not isolated per task, so a given task's log can include output from other tasks in that same run. There is also no default query execution timeout unless one is explicitly configured.

Streaming support is narrower than on classic clusters: only Trigger.AvailableNow(), which Databricks recommends, and the deprecated Trigger.Once() are supported. Trigger.Continuous() and Trigger.ProcessingTime() are not, and since Spark's own default trigger is a form of ProcessingTime, a supported trigger has to be set explicitly rather than left to that default.

Frequently asked questions

Does Databricks serverless compute support environment variables?

No. Environment variables are not supported on serverless compute. Databricks recommends using widgets to create job and task parameters instead.

Can I use init scripts with Databricks serverless compute?

No. Compute-scoped init scripts and compute policies are not supported. Custom dependencies are installed per notebook or per job task through the Environment configuration instead.

Does Databricks serverless support Scala or R?

Not in notebooks, which support Python and SQL only. Scala still works through JAR tasks in jobs, and R is not supported on serverless in any form.

Can I cache a DataFrame or table on serverless compute?

No. DataFrame and SQL cache APIs, including .cache(), .persist(), and CACHE TABLE, are not supported and raise an exception if used.

Are these limitations the same on AWS, Azure, and GCP?

Databricks documents largely the same set of limitations across all three clouds, though it is worth checking the version for your specific cloud, since some details can vary.

Weighing a specific workload?

Whether a limitation actually rules out serverless for a given workload depends on what that workload needs. The full decision framework covers what to check before you commit a workload to serverless, including compatibility questions like the ones above:

Take the caching limitation above. A workload that leans on .cache() or .persist(), for example an iterative join, or a machine learning training loop that reads the same DataFrame many times, gets real value from that on a classic cluster: the data stays in memory instead of getting read again from source on every pass. On serverless, that call simply raises an exception. If someone strips the caching calls out just to get the job running, rather than rebuilding the logic around the limitation, the workload now reads and reprocesses the same data on every pass. Runtime goes up, and since serverless bills by usage, so does the cost, for a workload that looked like a good fit right up until it started running.

Streaming jobs carry a similar risk. A pipeline built around Trigger.ProcessingTime(), Spark's own default trigger, or Trigger.Continuous() for near real time processing will not run as designed on serverless, since neither is supported; only Trigger.AvailableNow() and the deprecated Trigger.Once() are. A team that migrates a streaming workload without catching this either hits a hard failure, or rebuilds around AvailableNow() and ends up with a workload that behaves more like batch processing than the low latency pipeline it was built to be, sometimes without anyone flagging that the latency profile changed until a downstream SLA gets missed.

Neither of these is a reason to avoid serverless. They are reasons to check a workload's specific dependencies against this list before committing it, not after.

Before you move it: What to check before you commit a workload to serverless. (/guides/databricks-serverless-before-stage)

For a broader comparison of serverless against classic compute, see the Databricks Serverless vs. Classic Compute guide. (/guides/databricks-serverless-vs-classic-compute, slug not yet confirmed) Or see the complete framework on the Databricks Serverless Compute pillar page. (/guides/databricks-serverless-compute)

How Unravel fits in

Some of these limitations only surface once a workload is already running on serverless, and by then the cost of finding out is higher. Unravel helps teams check compatibility questions like these against a specific workload before committing to serverless, and keeps watching for constraints that start to matter as that workload evolves.

‍