A Spark job that used to finish in minutes now takes hours. How do you diagnose and fix it?

TL;DR: Open the Spark UI and find the bottleneck stage: look for a few tasks running far longer than the rest (data skew), heavy shuffle/spill, or too many tiny files. The usual culprits are a skewed join/group key, an exploded shuffle, or spill to disk from undersized partitions. Fixes: handle skew (salting, broadcast the small side, AQE), cut shuffle, and right-size partitions.

How to approach it. Refuse to guess; say you would profile in the Spark UI first to localize the slow stage and see the symptom (skew vs shuffle vs spill vs small files). Then map each symptom to its fix. Mention Adaptive Query Execution, since modern Spark handles several of these automatically.

A strong answer. Diagnose via the Spark UI. Find the stage consuming the time, then read its task distribution:

• A few tasks take far longer than the median: classic data skew. One join/group key (a null, a default, a whale customer) sends most rows to one partition, so one task does most of the work while others idle.
• Large shuffle read/write: a wide transformation (join, groupBy, distinct) is moving huge data across the network.
• Spill to disk (memory then disk metrics): partitions too big for executor memory, so Spark spills, which is slow.
• Huge task count on tiny files: the small-files problem; scheduling overhead dominates.

Fixes by cause:

• Skew: broadcast the small side of the join (broadcast hash join) so the large side need not shuffle at all; if both are large, salt the skewed key (append a random suffix to spread it, then aggregate), or lean on Adaptive Query Execution (AQE), which detects and splits skewed partitions automatically in recent Spark.
• Excess shuffle: filter and project early (predicate/column pruning) so less data moves; pre-partition or bucket tables on the join key to avoid repeated shuffles; avoid unnecessary repartition/distinct.
• Spill: increase partition count (or spark.sql.shuffle.partitions) so each partition fits in memory; AQE coalesces too-small partitions post-shuffle.
• Small files: compact on write (OPTIMIZE/coalesce) and read fewer, larger files.

Then ask what changed. A job that regressed usually points to a data change: volume growth, a newly skewed key, or a schema/partitioning change. Check input sizes and key distributions against the last good run.

The symptom-to-fix mapping is what separates a clean answer from flailing:

Key takeaways

• Localize the slow stage in the Spark UI before touching a single config; the task-duration distribution names the cause.
• Skew does not parallelize, so adding executors burns money without helping the straggler.
• AQE handles skew-join splitting and post-shuffle coalescing at runtime; turn it on before hand-tuning partition counts.
• A sudden regression is almost always a data change (new whale key, volume growth), so diff key distributions against the last good run.

What interviewers probe next.

• "Broadcast join limits?" Only when one side is small enough to fit in each executor's memory (tunable threshold); too large and it OOMs, so fall back to salting/AQE.
• "What is salting exactly?" Append a random key to spread a hot key across partitions, aggregate per salted key, then re-aggregate; it trades extra work for parallelism.
• "Catalyst/AQE role?" Catalyst optimizes the logical/physical plan (predicate pushdown, join selection); AQE re-optimizes at runtime using actual partition stats (skew handling, partition coalescing).
• "Spark vs a warehouse for this?" If it is repeated SQL aggregation, a well-partitioned lakehouse table (or a warehouse) may serve better than a hand-tuned Spark job.

Common mistakes.

• Throwing more executors at a skewed job; the straggler task does not parallelize, so it barely helps.
• Not opening the Spark UI and guessing at the cause.
• repartition everywhere, adding shuffles instead of removing them.
• Ignoring what changed in the data, fixing symptoms while the skewed key keeps growing.