Most Spark problems fall into 3 groups: jobs that fail, jobs that run slow, and jobs that cost too much. The usual causes are memory settings, data skew, too many or too few partitions, inefficient joins and oversized clusters. Start with the Spark UI and the driver and executor logs, then fix the job before you resize the cluster.
Below: a symptom table to find your problem fast, where to look first, and the 10 job-level and cluster-level problems we see most.
Why Spark is hard to troubleshoot
The features that make Spark fast also make it hard to debug:
- It runs in memory. That's where the speed comes from. It's also why jobs crash when memory runs short, and why the evidence disappears with the executor.
- It runs in parallel. One slow or failing task hides among hundreds of healthy ones.
- It runs everywhere. Open-source Spark on YARN or Kubernetes, Amazon EMR, Databricks, and other managed services all behave a little differently and come with different tools.
- It has hundreds of settings. Getting one right is hard. Getting several related ones right is guesswork, so teams over-allocate to stay safe.
- Testing is slow. If a job takes 6 hours, each config change costs 6 hours to test. A few rounds and the week is gone.
Find your problem by symptom
| What you see | Likely cause | Where to look | Go deeper |
|---|---|---|---|
OutOfMemoryError on the driver | collect(), toPandas(), large broadcast, too little driver memory | Driver log, the code calling the action | Part 1: Memory Management |
| Executor killed for exceeding memory limits | Memory overhead too low, too many cores per executor | Executor logs, YARN or Kubernetes events | Part 1: Memory Management |
| One task runs far longer than the rest | Data skew | Spark UI stage page, task duration max vs median | Part 2: Data Skew & Garbage Collection |
| Long GC time on executors | Too many objects in memory, heap too large or too small | Spark UI executors tab, GC time column | Part 2: Data Skew & Garbage Collection |
| Query reads far more data than it returns | Missing partition filters, no pruning | Query plan, PartitionFilters, rows and bytes read | Static vs Dynamic Partition Pruning in Spark |
| Slow join | Wrong join strategy, missing statistics | Query plan, join node type | What the Spark Catalyst Optimizer Does for Performance |
| Thousands of tiny tasks, or a few huge ones | Partition count wrong for the data | Spark UI stage page, task count | Job-level problem 3, below |
| Driver busy, executors idle | Driver bottleneck | Spark UI timeline, driver CPU | Resolving Databricks Driver Bottlenecks |
| Job works, but the bill keeps growing | Over-allocation, idle clusters | Cluster utilization, cost by job | Cluster-level problems, below |
Where to look first
- Spark UI. The Jobs, Stages and SQL tabs show where time goes. Sort stages by duration and compare task max to median.
- Driver and executor logs. The stack trace usually names the failure. Search for the first error, not the last.
- Query plans. Run
explain()or check the SQL tab to see joins, scans and shuffles. - Event logs and history. On cloud platforms, logs on the cluster disappear when it terminates. Turn on log delivery to cloud storage so you can debug short-lived job clusters after the fact.
[cta-banner]
Job-level problems
These show up in individual jobs. Data engineers and developers usually own them. Taken together, they decide how much cluster you need.
1. How many executors and cores should a job use?
The number of tasks running at once is executors times cores per executor. Too few and the job crawls. Too many cores per executor and memory per task shrinks, which leads to out-of-memory errors.
A common starting point is about 5 cores per executor. From there, check the Spark UI: high core usage and steady memory mean you're close. Idle cores or frequent spills mean you're not.
2. How much memory should each executor get?
Executor memory has 2 parts: the heap you set with spark.executor.memory, and overhead for everything outside the heap. Overhead defaults to 10% of executor memory, with a 384MB minimum.
Example: an 8GB executor heap gets about 819MB of overhead, so the container needs about 9GB. Leave room on each node for the OS and cluster manager, then divide the rest by executors per node.
That's a starting point. The right number depends on the data and the code. Part 1 walks through each memory region and the errors you hit when one runs out.
3. How do I handle data skew and small files?
Data skew is when a few keys hold most of the data. Their tasks run far longer, use far more memory, and can fail the whole stage. Spark's Adaptive Query Execution splits skewed join partitions automatically, but it can't fix every case. Techniques for the rest are in Part 2.
Small files are the opposite problem. Thousands of tiny files mean thousands of tiny tasks, and scheduling overhead outweighs the work. Compact them, or repartition before writing.
4. How do I troubleshoot at the pipeline level?
Pipelines chain jobs together. Each step can look fine alone while the pipeline misses its deadline. A step can also pass bad output downstream without failing. Track runtime and output size per step, per run, so you can see which step changed.
5. How do I know if a job is optimized?
Spark doesn't tell you. A job can finish on time and still waste half its resources. Compare runs over time: if the same job on the same data starts taking longer or using more, something changed. Check the query plan for full scans and shuffle-heavy joins. The Catalyst optimizer does a lot, but it works from estimates.
[cta-banner]
Cluster-level problems
These show up across many jobs. Platform and operations teams usually own them. Fix the job-level problems first. A cluster full of unoptimized jobs can't be tuned.
6. Are nodes matched to instance types?
Each node has fixed CPU and memory. Executors that don't fit cleanly leave cores or memory stranded on every node. On-premises that waste is invisible. In the cloud it shows up on every bill. Pick instance types that fit your executor shape.
7. How do I see what's going on across the cluster?
The Spark UI shows one application at a time. It doesn't tell you which jobs used the most resources this month, or whether a job is getting worse over time. For that you need history across runs and jobs, from event logs or a monitoring tool.
8. Is the data partitioned for the queries we run?
A query that filters on the partition column reads a fraction of the table. One that doesn't reads all of it. Partition on the columns your queries filter on most, and check plans for empty partition filters. See Static vs Dynamic Partition Pruning in Spark.
9. When should I use autoscaling?
Autoscaling adds workers during peaks and removes them after. It helps bursty jobs. It also has no natural ceiling, so set a maximum.
Optimize and profile the job first. Autoscaling an inefficient job just buys more of the inefficiency. On Databricks, run scheduled production work on job compute or serverless. Keep shared all-purpose clusters for interactive work. For the trade-offs, see When to Use Databricks Serverless vs Classic Compute.
Spot instances cost less than on-demand, but the cloud provider can reclaim them. Use them for workers on jobs that can tolerate losing a node, not for the driver.
10. How do I find and fix problems before users notice?
Most teams troubleshoot after a job fails or an SLA is missed. The better signal is a job that's slowly getting worse: longer runtimes, more spill, more memory. Noisy neighbors make it harder. A job can run fine one day and slow the next because of what else is on the cluster.
The cost of getting it wrong
Most Spark problems come back to allocation:
- Under-allocation: too little memory or too few cores. Jobs run slow or crash.
- Over-allocation: the usual fix for under-allocation. Jobs stop crashing, but the waste moves to the bill. We've seen a job that needed 2GB of memory allocated 85GB.
Over-allocated jobs also slow down everything else on a shared cluster, which leads to more over-allocation.
Right-sizing pays off. Mastercard cut usage of their clusters by roughly half while data volumes kept growing.
How Unravel fits in
Troubleshooting one job by hand works. Doing it for thousands of jobs across teams doesn't. Arvix, Unravel's autonomous AI operator, detects problems across every workload, rewrites inefficient queries and resizes compute based on actual demand. Learn more about Spark optimization with Unravel, or see 5 types of Spark troubleshooting tools compared.
Get a free health check to find the Spark jobs costing you the most.





