Adaptive Query Execution

Adaptive query execution is the most over-trusted setting in Spark. Teams turn it on, see no change, and conclude it is broken. It is usually working exactly as designed — and declining to act, for reasons it will not tell you.

AQE waits until a shuffle write has finished, reads the map output statistics — the real byte size of every shuffle partition — and then rewrites the rest of the plan using numbers instead of estimates. Three rules matter in practice: coalesce partitions that came out too small, split partitions that came out too large, and switch a sort-merge join to a broadcast join if a side turned out to be tiny.

Every one of those rules is guarded by a threshold. The panel below is the decision itself: set the statistics, set the thresholds, and read why AQE did or did not act on each partition.

Start with One hot key and defaults. The skew rule reports that it fired and then that it could not help, because the partition is one key and a key cannot be divided across reducers — the same wall described in the shuffle lesson. Now switch to Long tail: same settings, and the rule works, because those partitions are many medium keys that can be sliced apart.

The two conditions, and why both matter

A partition is treated as skewed only when both of these are true:

• it is larger than skewedPartitionFactor × the median partition size, and
• it is larger than skewedPartitionThresholdInBytes.

The median condition is the one that surprises people. Choose Long tail and drag skewedPartitionFactor from 5 down to 2: more partitions qualify. Drag it to 10 and the rule goes quiet, even though the data has not changed. Skew here is defined relative to your own job — a job where everything is large has no skew by this definition, and AQE will leave a 900 MB partition alone if the median is 400 MB.

The absolute threshold exists so AQE does not shatter a small job into hundreds of tiny tasks. Its default of 256 MB is also why AQE looks inert on modest datasets: with a 2 GB shuffle across 200 partitions, nothing is close to 256 MB, so no matter how lopsided the job feels, the rule never arms. Set the shape to Even and drag the threshold down to 64 MB to watch a job that was fine start getting split for no benefit.

Coalescing only merges neighbours

Choose Over-partitioned. Twenty-four 9 MB partitions become a handful of tasks, each near advisoryPartitionSizeInBytes. This is the rule that quietly earns AQE its keep: it means spark.sql.shuffle.partitions can be set generously high and trimmed at runtime, instead of guessed per job.

The constraint that catches people is that coalescing merges only contiguous partition ranges. It cannot gather partition 3 and partition 17 into one task, because a task reads a contiguous slice of the reducer id space from each mapper — that contiguity is what keeps the read sequential. So a shuffle that alternates tiny, huge, tiny, huge coalesces poorly no matter how much slack the advisory size gives it. Choose Long tail and watch how few merges happen: the fat partitions are walls between the small ones.

Coalescing also does not reduce the amount of data read. It reduces the number of tasks reading it. If your problem is total shuffle volume, this rule does nothing for you, and you want a plan with no shuffle instead.

Why AQE cannot see the problem before it happens

Everything above happens after the shuffle write. The map tasks have already serialised, sorted, compressed and written every byte to local disk before AQE knows anything. That is the whole design: statistics instead of estimates, at the cost of only being able to act on the read side.

So AQE never prevents a skewed write. If one map task is producing most of the output — a skewed source partition rather than a skewed join key — AQE has nothing to say, and you are back to fixing the input layout. It also cannot undo a shuffle that should not have happened; a missing broadcast hint costs you the whole write, and the runtime switch to a broadcast join only applies at a stage boundary where a side turned out small enough.

The honest summary: AQE converts partition-size problems into non-problems, automatically and well. It does not convert key-distribution problems into non-problems, and the two look identical in the Spark UI until you check whether the fat partition holds one key or many.

Checking it yourself

In the Spark UI, an AQE plan shows AQEShuffleRead nodes with labels like coalesced and skewed, and the number of partitions after the rewrite. If you see AQEShuffleRead with no skewed label on a stage you believe is skewed, the rule declined — and now you know it was one of the two thresholds, or a single key.

Next, the mechanism AQE is compensating for: how the shuffle assigns rows to partitions, and the setting people reach for first, repartition and coalesce.