Data skew in Spark and JVM garbage collection are two of the most common reasons Spark jobs run slowly or fail after they reach production scale. This guide covers how to spot each one, what Spark now handles automatically, and the fixes that still need an engineer.
Quick definition
Data skew in Spark happens when data is spread unevenly across partitions, so a few tasks get far more rows than the rest. The stage cannot finish until its slowest task does, so one oversized partition holds up the whole job. Garbage collection (GC) is the JVM reclaiming memory from objects Spark no longer needs. When it runs too often or too long, executors stall, miss heartbeats and get dropped.
Why skew and garbage collection slow Spark down
This is Part 2 of a two-part series. Part 1 covered memory management and the out of memory errors that follow from it. Part 2 covers data skew and garbage collection. Like most Spark performance problems, both get worse as data grows.
In an ideal run, the keys in a join are evenly distributed and every partition gets a similar share of the work. Real business data is rarely that tidy. A handful of customers, products or dates usually account for a large share of the rows, and those keys pile up in a few partitions.
Data skew is not a Spark defect. It is a data problem. Uneven distribution comes from the data itself, the way it is laid out, or the shape of the query.
Data skew in Spark
Skew shows up when Spark shuffles data, as it does for joins and aggregations. A shuffle moves every record with the same key into the same partition so it can be processed together. It is also the most expensive operation in Spark, because data moves across the network between executors.
Spark decides where each record goes by hashing its key and dividing by the number of shuffle partitions. Every record for a given key lands in one partition. If one key has millions of rows and the rest have hundreds, one partition becomes huge, and any query engine struggles with that without special handling.
Common symptoms of data skew in Spark:
- Stages that stall near the end while one or two tasks keep running
- Low CPU utilization across the cluster, because most executors sit idle
- Out of memory errors or heavy spill on the executors holding the large partitions
- Longer runtimes on cloud clusters, where idle capacity still costs money
How to identify data skew in Spark
The Spark UI shows skew clearly once you know where to look.
- Stage summary metrics. Open a slow stage and compare Max against the median and 75th percentile for Duration and Shuffle Read Size. The Databricks Spark UI guide uses a simple rule: if the Max duration is 50% more than the 75th percentile, you may be dealing with skew.
- Event timeline. One bar that runs far longer than the others is the classic picture of a skewed task.
- SQL tab. In the final Adaptive Query Execution plan, a join marked
skew=truemeans Spark detected and split a skewed partition. The shuffle read node reports how many skewed partitions it found.
Figure: One shuffle task takes far longer than the rest of the stage
Sometimes the skew starts in the source table. A table partitioned on a month column, with far more records in some months than others, produces skewed processing in every stage that reads it. Restructuring the table with better partition keys helps, but that is not always practical when other pipelines depend on the same table. On Databricks, liquid clustering replaces partitioning and Z-ordering for new tables and handles skewed data well. Databricks' own partitioning guidance is to skip partitioning entirely for tables under 1TB.
For more detection signals, see Databricks Data Skew: 5 Detection Signals Teams Miss Daily.
[cta-banner]
How to fix data skew in Spark
Start with what Spark does automatically, then move to manual fixes for the cases it does not cover.
Let Adaptive Query Execution handle skewed joins
Adaptive Query Execution (AQE) has been on by default since Spark 3.2. It re-plans a query at runtime using real partition sizes, and it splits skewed partitions in joins automatically. A partition counts as skewed when it is both five times larger than the median partition and larger than 256MB.
| Setting | Default | What it does |
|---|---|---|
spark.sql.adaptive.enabled | true (Spark 3.2+) | Turns on runtime re-planning |
spark.sql.adaptive.skewJoin.enabled | true | Splits skewed partitions in joins |
spark.sql.adaptive.skewJoin.skewedPartitionFactor | 5 | How many times larger than the median a partition must be |
spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes | 256MB | Minimum size before a partition counts as skewed |
spark.sql.adaptive.advisoryPartitionSizeInBytes | 64MB | Target size for the split partitions |
On Databricks, AQE is on by default and handles skew without any hints. Skew join hints are now legacy.
AQE covers a lot, but not everything:
- It splits skew in sort merge joins and shuffled hash joins, not broadcast joins
- It cannot split either side of a full outer join
- Its skew handling targets joins, so skewed
groupByaggregations and window functions still need manual fixes - It backs off when splitting would add an extra shuffle, unless you force it
If AQE misses skew you can see in the Spark UI, lowering the skew factor or threshold is the first thing to try.
Broadcast the smaller table
When one side of a join is small, broadcasting it removes the shuffle entirely, so there is nothing to skew. Spark broadcasts tables under spark.sql.autoBroadcastJoinThreshold (10MB by default) automatically, and AQE can switch to a broadcast join at runtime. Databricks raises the runtime threshold to 30MB.
Raising the threshold helps, but only so far. The broadcast table has to fit in driver and executor memory, and Databricks documents a case where broadcasting a large table caused heavy garbage collection and lost executors. Part 1 covers how broadcast joins use driver memory.
Handle null keys
Null join keys all hash to the same partition, which can create one very large one. For inner joins, Spark already filters out null keys before the join, so no action is needed. For outer and anti joins, the nulls stay in. Split the null-key rows out, process them separately and union them back, so they never land in a single partition.
Salt the skewed keys
Salting spreads a hot key across many partitions by adding a random suffix to it. It remains the right tool when AQE cannot help: skewed aggregations, window functions, full outer joins and very heavy hot keys.
Take two tables, a large fact table and a smaller dimension table, joined on a key where one value (key 1) holds most of the rows:
spark.sql(""" SELECT t1.key, count(*) FROM fact_table t1 JOIN dimension t2 ON t1.key = t2.dim_key GROUP BY t1.key ORDER BY t1.key """).show()
After the shuffle, every row for key 1 lands in partition 1, so that partition takes far longer to process than the others.
Figure: Key 1 fills partition 1 after the shuffle
To salt the join, add a random salt to the key on the large side, and replicate each row on the small side once per salt value so every salted key still finds its match:
from pyspark.sql import functions as F SALTS = 20 # size this so each salted partition is roughly 100 to 300MB fact_salted = fact.withColumn("salt", (F.rand(seed=42) * SALTS).cast("int")) dim_salted = dim.withColumn( "salt", F.explode(F.array([F.lit(i) for i in range(SALTS)])) ) result = ( fact_salted.join( dim_salted, (fact_salted.key == dim_salted.dim_key) & (fact_salted.salt == dim_salted.salt), ) .groupBy("key") .count() )
With the salt in place, the work spreads evenly and the stage finishes in a fraction of the time.
Figure: Task times even out after salting
A few rules keep salting from creating new problems:
- Salt only the hot keys. Replicating the whole small table multiplies its size by the salt count. Filter the few skewed keys out, salt those, and join the rest normally.
- Size the salt count. Pick a number that brings each salted partition down to a reasonable size, not a fixed 20.
- Use two stages for aggregations. Aggregate on the key plus salt first, then aggregate the partial results on the key alone.
- Test with and without the salt. For small data the difference may be negligible, and skewed data often compresses well during the shuffle, which narrows the gap.
For a Databricks-specific walkthrough, see Eliminating Databricks Data Skew Before It Kills Performance.
Garbage collection in Spark
Spark runs on the Java Virtual Machine, and it keeps large amounts of data in memory. That makes it heavily dependent on JVM garbage collection, and it is why GC problems are more common in Spark than in most Java applications.
Common symptoms of excessive garbage collection in Spark:
- Slow tasks with no obvious cause in the data
Executor heartbeat timed out, followed byExecutorLostFailurejava.lang.OutOfMemoryError: GC overhead limit exceededorJava heap space- Shuffle
FetchFailedExceptionerrors when an executor stalls or disappears mid-shuffle
GC problems are easy to spot. The Executors tab in the Spark UI shades an executor's Task Time (GC Time) in red when garbage collection takes more than 10% of task time, and the shade deepens as the share grows.
Figure: The Spark UI flags excessive GC time in red
Long pauses also explain the heartbeat errors. Executors send a heartbeat every 10 seconds (spark.executor.heartbeatInterval), and the driver drops an executor it has not heard from within spark.network.timeout (120 seconds by default). A long enough GC pause looks, from the driver's side, exactly like a dead executor.
The JVM has changed since the original version of this post. Spark 4.0 runs on Java 17 by default, which makes G1 the default garbage collector. The Spark tuning guide notes that large executor heaps may need a larger G1 region size (-XX:G1HeapRegionSize).
How to fix garbage collection issues in Spark
Use built-in functions instead of UDFs
In Spark SQL and DataFrames, built-in functions work directly on Spark's internal binary row format and fit into whole-stage code generation. User-defined functions convert each row into regular objects, which creates garbage on every row. On Databricks, built-in functions also let Photon run the work natively, outside the JVM, with no garbage collection involved.
Keep data structures lean
In RDD-based code and UDFs, use data structures with fewer objects: arrays instead of lists, primitive types instead of boxed ones. For large collections of primitives, specialized libraries such as fastutil cut memory use further. DataFrame and SQL code rarely needs this, because Spark manages those rows itself.
Be stingy about object creation
Spark jobs routinely process billions of rows. A temporary 100-byte object created for every row adds up to 100GB of garbage across a billion rows. Reuse objects where you can, and avoid building intermediate collections inside per-row logic.
Store data off-heap
Spark's execution engine and storage can both keep data off the JVM heap, where the garbage collector does not scan it:
spark.memory.offHeap.enabled=truespark.memory.offHeap.size=<size>
Off-heap memory does not shrink the heap. It is added on top of it, so the container has to be sized for both. Part 1 walks through the container math.
Turn on GC logging
To see what the collector is doing, add GC logging to the executor JVM options. On Java 17 and later, the old -XX:+PrintGCDetails style flags are deprecated or removed, so use unified logging:
spark.executor.extraJavaOptions=-Xlog:gc*:file=/tmp/gc.log
For deeper tuning on Databricks, see Databricks Garbage Collection Tuning Detection Made Easy.
Data skew and garbage collection symptoms at a glance
| Symptom | Likely cause | First fix |
|---|---|---|
| One or two tasks run far longer than the rest | Data skew in a join or aggregation | Confirm AQE is on; check for skew=true in the SQL plan |
Skewed groupBy or window function | Hot keys AQE does not split | Two-stage salted aggregation |
| Skewed full outer join | AQE cannot split either side | Salt the hot keys or split them out |
| Executor Task Time shaded red in the Spark UI | GC over 10% of task time | Replace UDFs with built-ins; reduce object creation |
Executor heartbeat timed out / ExecutorLostFailure | Long GC pauses or memory pressure | Check GC time; reduce data held per task |
GC overhead limit exceeded | Heap nearly full, GC reclaiming almost nothing | Add partitions or memory; look for large broadcasts or caches |
[cta-banner]
Frequently asked questions
What is data skew in Spark?
An uneven spread of data across partitions, usually because a few keys hold most of the rows. The tasks processing those partitions run much longer than the rest and hold up the stage.
Does Adaptive Query Execution fix data skew automatically?
For most skewed joins, yes. AQE splits skewed partitions in sort merge and shuffled hash joins by default since Spark 3.2. It does not split skewed aggregations, window functions or full outer joins.
When should I use salting in Spark?
When AQE cannot help: skewed aggregations, window functions, full outer joins, or hot keys too heavy for automatic splitting. Salt only the hot keys to keep the extra data small.
How much GC time is too much in Spark?
The Spark UI flags executors in red when garbage collection takes more than 10% of task time. That is a good threshold to investigate.
How do UDFs affect garbage collection?
UDFs convert Spark's internal rows into regular objects, which creates garbage for every row processed. Built-in functions avoid that conversion.
Why tuning alone doesn't keep jobs healthy
Skew and GC problems rarely stay fixed. Adaptive Query Execution, liquid clustering and Photon on Databricks now handle much of what teams once tuned by hand, and they handle it well. But data keeps changing. A new customer becomes the hot key, a table doubles, a new UDF lands in a critical pipeline.
At enterprise scale, the challenge is seeing those shifts across thousands of jobs before they turn into missed SLAs.
How Unravel fits in
Unravel gives teams that visibility. It provides automatic root cause analysis for Spark applications, with insight into partitioning, parallelism, garbage collection, caching, misconfiguration and container resource utilization, so engineers can see which job, stage and key is responsible without digging through the Spark UI one application at a time.
On Databricks, Arvix, Unravel's AI engine, goes further: it rewrites inefficient code and rightsizes compute, validates every change against real workload behavior before applying it, and rolls it back automatically if anything degrades. Teams choose how much to automate, from human approval to AutoApply. Unravel's DataOps Agent cuts troubleshooting time by 99%.
See how Unravel makes Spark perform better and more reliably, or request a free health check to find the skew and GC issues in your environment.