Inside a Spark Job: Stages, Shuffles, AQE and Memory, and How to Read Them

On this page

Apache Spark makes distributed processing look like ordinary DataFrame code, which is both its strength and the source of most production problems. A job that took ten minutes last month takes two hours today; one task out of two thousand runs for forty minutes; executors die with out-of-memory errors on a cluster that has plenty of memory. None of these is random. Each follows from how Spark turns code into stages, shuffles and tasks, and each shows up in the query plan and the Spark UI before anyone needs to guess.

This article goes inside a Spark job as of November 2025, with Spark 4.0: lazy evaluation and planning, stages and shuffles, join strategies, Adaptive Query Execution (AQE), partitions and skew, and the executor memory model. Where Spark sits among the other tools in a data platform is covered in the earlier article on the modern data stack; this one stays inside the engine. The sample code turns each behaviour into a test that runs on a laptop.

What changed in Spark 4.0

Spark 4.0.0 was released in May 2025, with over 5,100 tickets resolved. Three changes matter for existing jobs. JDK 17 became the default and JDK 8 and 11 were dropped, alongside Scala 2.12. ANSI SQL mode is now on by default, so overflows, invalid casts and division by zero raise errors instead of quietly producing nulls. Spark Connect, which separates the client from the cluster, gained a lightweight Python client and a spark.api.mode setting to switch it on. Support for Apache Mesos was removed, leaving Kubernetes, YARN and the standalone manager.

⚠️
ANSI MODE WILL SURFACE OLD DATA PROBLEMS

Jobs that ran cleanly on Spark 3.x can fail on 4.0 because bad values that used to become nulls now raise errors. Run existing jobs against 4.0 in a test environment with ANSI mode on, fix the data or the casts each error points to, and only set spark.sql.ansi.enabled=false as a temporary measure.

From code to tasks

DataFrame transformations become a logical plan, Catalyst optimises it, the scheduler cuts it into stages at shuffles, tasks run per partition, and AQE re-plans between stages; below, an executor heap split into reserved, execution and storage, and user memory
A stage ends wherever data must move between executors. Count the shuffles and you know the shape of the job.

Transformations such as filter, select, join and groupBy only describe work. Nothing runs until an action such as count, collect or a write. At that point the Catalyst optimiser turns the logical plan into a physical plan: it pushes filters into the data source, reads only the columns needed and chooses join strategies. The scheduler then splits the physical plan into stages wherever rows must be redistributed by key, a shuffle, shown as an Exchange node in the plan. Within a stage, Spark runs one task per partition on the executors’ cores.

Narrow operations (filter, select, map) stay within a partition and cost little. Wide operations (groupBy, join, distinct, window functions, repartition) shuffle: every executor writes its rows to local disk, partitioned by key, and every task in the next stage fetches its share over the network. Shuffles dominate the cost of most slow jobs, so the first question for any job is how many shuffles it has and how much data each one moves.

Joins and Adaptive Query Execution

For joins Spark mainly chooses between a broadcast hash join, which copies the small side to every executor so the large side does not move, and a sort-merge join, which shuffles and sorts both sides by the key. Broadcasting is chosen when the small side is under spark.sql.autoBroadcastJoinThreshold (10 MB by default), estimated from statistics. Missing or stale statistics are the usual reason a large table is shuffled to join with a lookup table.

AQE, on by default since Spark 3.2, re-plans the query between stages using the real sizes of the shuffle output rather than estimates. It does three things: it coalesces many small shuffle partitions into fewer reasonable ones, it switches a sort-merge join to a broadcast join when one side turns out small, and it splits skewed partitions in joins so one hot key does not leave a single task running for an hour.

Read the plan instead of guessing

The sample’s helpers return the executed physical plan of a DataFrame after AQE has finalised it, count its shuffles, name the join strategy and measure partition skew for a key:

Python81 lines
"""Small, testable helpers for looking inside Spark jobs (PySpark 4.0).

- `physical_plan`: the executed physical plan as text, after Adaptive Query
  Execution (AQE) has run, so you can see Exchanges (shuffles) and the join
  strategy Spark actually chose.
- `count_exchanges`, `join_strategy`: read those facts from the plan.
- `partition_skew`: rows per partition and the max/median ratio, to spot
  skew before it becomes one straggling task.
- `executor_memory`: the unified memory model arithmetic for an executor.

Uses the classic (non-Connect) PySpark API, which exposes the JVM query
execution; Spark Connect clients would read the same plan from the UI or
`df.explain(mode="formatted")`.
"""
from __future__ import annotations

import re
import statistics
from dataclasses import dataclass

from pyspark.sql import DataFrame, functions as F

RESERVED_MB = 300            # fixed by Spark for internal objects


def physical_plan(df: DataFrame) -> str:
    """Executed plan after an action, including AQE's final decisions."""
    qe = df._jdf.queryExecution()
    qe.toRdd().count()                                  # run this exact plan so AQE finalises it
    return qe.executedPlan().toString()


def count_exchanges(plan: str) -> int:
    """Shuffle exchanges in a plan (broadcast exchanges are not shuffles)."""
    return len(re.findall(r"(?<![A-Za-z])Exchange\b", plan))


def join_strategy(plan: str) -> str | None:
    for name in ("BroadcastHashJoin", "SortMergeJoin", "ShuffledHashJoin", "BroadcastNestedLoopJoin",
                 "CartesianProduct"):
        if name in plan:
            return name
    return None


@dataclass
class Skew:
    rows_per_partition: list[int]

    @property
    def ratio(self) -> float:
        nonzero = [r for r in self.rows_per_partition if r] or [0]
        median = statistics.median(nonzero)
        return max(nonzero) / median if median else 0.0


def partition_skew(df: DataFrame, key: str, partitions: int) -> Skew:
    """Rows per partition if `df` were hash-partitioned on `key` into `partitions`."""
    counts = (df.repartition(partitions, key)
                .groupBy(F.spark_partition_id().alias("p")).count()
                .collect())
    by_p = {r["p"]: r["count"] for r in counts}
    return Skew([by_p.get(i, 0) for i in range(partitions)])


@dataclass(frozen=True)
class ExecutorMemory:
    heap_mb: int
    unified_mb: float        # execution + storage, shared
    storage_protected_mb: float
    user_mb: float


def executor_memory(heap_mb: int, memory_fraction: float = 0.6, storage_fraction: float = 0.5) -> ExecutorMemory:
    """Unified memory model: (heap - 300 MB) split between Spark's pool and user memory."""
    if heap_mb <= RESERVED_MB * 1.5:
        raise ValueError("executor heap must be at least 450 MB")
    usable = heap_mb - RESERVED_MB
    unified = usable * memory_fraction
    return ExecutorMemory(heap_mb, round(unified, 1), round(unified * storage_fraction, 1),
                          round(usable - unified, 1))

The tests make the behaviours above concrete on a local session: a filter adds no shuffle and a groupBy adds one; a 1,000-row dimension is broadcast unless the threshold is disabled; AQE coalesces the default 200 shuffle partitions and without AQE all 200 remain; a key that holds 80 per cent of the rows gives a skew ratio far above an evenly spread key.

Python88 lines
import pytest
from pyspark.sql import SparkSession, functions as F

import spark_lab as lab


@pytest.fixture(scope="module")
def spark():
    s = (SparkSession.builder.master("local[2]").appName("spark-internals-lab")
         .config("spark.ui.enabled", "false")
         .config("spark.sql.shuffle.partitions", "200")
         .getOrCreate())
    s.sparkContext.setLogLevel("ERROR")
    yield s
    s.stop()


@pytest.fixture(scope="module")
def frames(spark):
    orders = spark.range(0, 200_000).select(
        F.col("id").alias("order_id"),
        (F.col("id") % 1000).alias("customer_id"),
        (F.col("id") % 97).cast("double").alias("amount"))
    customers = spark.range(0, 1000).select(F.col("id").alias("customer_id"),
                                             (F.col("id") % 5).alias("segment"))
    return orders, customers


def test_transformations_are_lazy_until_an_action(spark, frames):
    orders, _ = frames
    tracker = spark.sparkContext.statusTracker()
    before = set(tracker.getJobIdsForGroup())
    plan = orders.filter(F.col("amount") > 50).groupBy("customer_id").agg(F.sum("amount"))
    assert set(tracker.getJobIdsForGroup()) == before          # nothing ran yet
    plan.count()
    assert set(tracker.getJobIdsForGroup()) - before            # the action ran jobs


def test_groupby_introduces_a_shuffle(frames):
    orders, _ = frames
    narrow = orders.filter(F.col("amount") > 50).select("order_id")
    wide = orders.groupBy("customer_id").agg(F.sum("amount").alias("total"))
    assert lab.count_exchanges(lab.physical_plan(narrow)) == 0
    assert lab.count_exchanges(lab.physical_plan(wide)) >= 1


def test_small_dimension_is_broadcast_unless_disabled(spark, frames):
    orders, customers = frames
    joined = orders.join(customers, "customer_id")
    assert lab.join_strategy(lab.physical_plan(joined)) == "BroadcastHashJoin"
    spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
    spark.conf.set("spark.sql.adaptive.enabled", "false")
    try:
        assert lab.join_strategy(lab.physical_plan(orders.join(customers, "customer_id"))) == "SortMergeJoin"
    finally:
        spark.conf.unset("spark.sql.autoBroadcastJoinThreshold")
        spark.conf.unset("spark.sql.adaptive.enabled")


def test_aqe_coalesces_small_shuffle_partitions(spark, frames):
    orders, _ = frames
    agg = orders.groupBy("customer_id").agg(F.sum("amount"))
    spark.conf.set("spark.sql.adaptive.enabled", "true")
    plan_on = lab.physical_plan(agg)
    assert "AQEShuffleRead" in plan_on and "coalesced" in plan_on
    spark.conf.set("spark.sql.adaptive.enabled", "false")
    try:
        plan_off = lab.physical_plan(orders.groupBy("customer_id").agg(F.sum("amount")))
        assert "AQEShuffleRead" not in plan_off and ", 200)" in plan_off   # all 200 partitions kept
    finally:
        spark.conf.unset("spark.sql.adaptive.enabled")


def test_partition_skew_detects_a_hot_key(spark):
    even = spark.range(0, 100_000).select((F.col("id") % 100).alias("k"))
    hot = spark.range(0, 100_000).select(F.when(F.col("id") % 10 < 8, F.lit(0))
                                         .otherwise(F.col("id") % 100).alias("k"))
    assert lab.partition_skew(even, "k", 8).ratio < 1.5
    assert lab.partition_skew(hot, "k", 8).ratio > 4


def test_executor_memory_model():
    m = lab.executor_memory(16 * 1024)                 # 16 GiB heap
    assert m.unified_mb == pytest.approx((16384 - 300) * 0.6, abs=0.1)
    assert m.storage_protected_mb == pytest.approx(m.unified_mb / 2, abs=0.1)
    assert m.user_mb == pytest.approx((16384 - 300) * 0.4, abs=0.1)
    with pytest.raises(ValueError):
        lab.executor_memory(400)
%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#FFF3E0', 'primaryTextColor': '#232F3E', 'primaryBorderColor': '#FF9900', 'lineColor': '#232F3E', 'clusterBkg': '#F8FAFC', 'clusterBorder': '#CBD5E1', 'edgeLabelBackground': '#FFFFFF', 'nodeTextColor': '#232F3E'}}}%%
flowchart TD
    A["Slow stage in the Spark UI"] --> B{"One task much<br/>longer than the median?"}
    B -- "Yes" --> C["Skew: check the key,<br/>AQE skew join, salt it"]
    B -- "No" --> D{"Spill bytes above zero?"}
    D -- "Yes" --> E["Partitions too large:<br/>more partitions or memory"]
    D -- "No" --> F{"Large shuffle for<br/>a small table?"}
    F -- "Yes" --> G["Broadcast the small side,<br/>refresh statistics"]
    F -- "No" --> H{"Thousands of tiny tasks?"}
    H -- "Yes" --> I["Compact input files,<br/>check coalescing"]
    H -- "No" --> J["Profile the code:<br/>Python UDFs, I/O, GC"]
    classDef aws fill:#FFF3E0,stroke:#FF9900,stroke-width:2px,color:#232F3E
    classDef ok fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202
    class A,B,D,F,H aws
    class C,E,G,I,J ok

Partitions, skew and memory

Skewed keys, spill to disk, small files and the wrong join strategy compared by UI symptom, usual cause, first fix and Spark 4 defaults
Most slow jobs are one of these four, and the Spark UI says which.

A task processes one partition, so partition size decides both parallelism and memory pressure. Aim for shuffle partitions of roughly 100 to 200 MB; far smaller and scheduling overhead dominates, far larger and tasks spill. spark.sql.shuffle.partitions defaults to 200, which AQE adjusts downwards but not upwards, so very large shuffles may need it raised. Skew is the exception partitions cannot fix: when one key holds a large share of the rows, its partition is large whatever the count. AQE splits skewed join partitions; for aggregations, adding a random salt to the key and aggregating twice spreads the work.

Executor memory follows the unified model. Spark reserves 300 MB, then gives spark.memory.fraction (0.6) of the rest to a shared pool for execution (shuffles, joins, sorts, aggregations) and storage (cached data, broadcasts), with half of it protected for storage by spark.memory.storageFraction. The remaining 40 per cent is user memory for your own objects. Python workers, network buffers and native libraries live outside the heap in spark.executor.memoryOverhead, which is why PySpark jobs killed by the container manager usually need more overhead, not more heap.

💡
CACHE ON PURPOSE

Caching helps only when the same DataFrame is used by several actions and recomputing it is expensive. Cache, use, then unpersist; caching everything fills storage memory, evicts data that mattered and pushes execution into spills.

⚖️
SHUFFLE FILES AND LOGS HOLD DATA

Shuffle files on executor disks and rows printed to driver logs contain the data being processed. Encrypt local disks and shuffle traffic where the data is sensitive (Spark supports I/O encryption), keep row-level data out of logs, and make sure clean-up of shuffle files and event logs follows your retention rules.

📝
HOW THIS WAS CHECKED

The sample was tested with Python 3.12, pyspark==4.0.1 (September 2025), OpenJDK 21 and pytest==8.4.2 in local[2] mode: six tests cover lazy evaluation, shuffle counts, broadcast versus sort-merge joins, AQE coalescing, skew measurement and the memory arithmetic. Spark 4.0 changes are from the Spark 4.0.0 release notes. The partition size range is a common starting point, not a Spark default.

Key takeaways

  • Spark 4.0 requires JDK 17, turns ANSI mode on and removes Mesos; test existing jobs before upgrading.
  • Transformations are lazy; an action triggers planning, and every shuffle (Exchange) starts a new stage.
  • Broadcast small tables, keep statistics fresh, and let AQE coalesce partitions, switch joins and split skew.
  • Size partitions for memory, treat a single long task as skew, and salt hot keys AQE cannot handle.
  • Know the memory model: 300 MB reserved, 60 per cent shared for execution and storage, overhead outside the heap.

What was the hardest Spark performance problem you traced, and what did the plan or the UI show? Share it in the comments.

Glossary

Stage
A set of tasks that run without moving data between executors; stages are separated by shuffles.
Shuffle
Redistributing rows across executors by key, written to disk and fetched over the network.
Exchange
The physical plan node that represents a shuffle.
Broadcast join
A join that copies the small side to every executor so the large side does not shuffle.
AQE
Adaptive Query Execution: re-planning between stages using real data sizes.
Skew
Uneven distribution of rows across partitions, usually caused by hot keys.
Spill
Writing intermediate data to disk when execution memory runs out.
ANSI mode
Spark SQL behaviour that follows the SQL standard, raising errors on invalid operations.

References & Further Reading


Discover more from C4: Container, Code, Cloud & Context

Subscribe to get the latest posts sent to your email.

Written by

Nithin Mohan TK

Hello Namaste! 👋🏻 I'm an accomplished Enterprise Solution Architect & Engineering Leader with 20+ years of progressive experience architecting, building, and scaling transformative enterprise solutions. ⚜️ Solution Architect / AI & Cloud 📌 Galway, Ireland

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.