Apache Spark Overview (Data Engineering)

Before you start: working Python knowledge is assumed. No prior distributed-systems or big-data experience is needed.

Spark in the Data Engineering Context

Some datasets are too large for one machine to process in reasonable time, no matter how much RAM or CPU that one machine has — a 10TB dataset simply doesn't fit in memory on a single laptop or even most single servers. Apache Spark is a distributed data processing framework: it splits a dataset across many machines (a cluster), processes each piece in parallel, and combines the results — the same transformation logic you'd write for a small dataset works unchanged at far larger scale, because Spark handles the distribution.

Analogy — Imagine sorting a warehouse full of boxes by hand. One person doing it alone might take a month. A hundred people, each sorting a different aisle simultaneously, finish the same job in a fraction of the time — as long as someone coordinates who sorts which aisle and combines everyone's results at the end into one final report. Spark is that coordinator: your code describes what to do (filter these boxes, group those by category), and Spark figures out how to split the work across the cluster's workers and combine their results, without you having to manually manage which worker does what.

1 machine,  1TB dataset  -->  processes sequentially, slow
N machines, 1TB dataset  -->  Spark splits it into partitions,
                               each machine processes its slice
                               in parallel, results combined

The Core Mechanic: Lazy Transformations, Eager Actions

This is the single most important concept in Spark, and it trips up nearly everyone at first. Operations like filter, select, groupBy are transformations — calling them doesn't process any data, it just builds a plan. Only an action (show(), collect(), count(), write()) actually triggers computation — and at that point, Spark's optimizer looks at the entire chain of transformations together and figures out the most efficient way to run all of it at once.

# Nothing has actually run yet after these three lines -- just building a plan
df2 = df.filter(df.amount > 100)
df3 = df2.select("customer_id", "amount")
df4 = df3.groupBy("customer_id").sum("amount")

df4.show()   # THIS line triggers actual execution, of the whole chain at once

Because Spark sees the whole chain before running anything, it can reorder and combine steps for efficiency (e.g. applying the filter before moving data across the network) — this is fundamentally why Spark performs well at scale, and it's also why a print() between transformation lines won't show intermediate data the way ordinary, eager Python code would.

Why Spark Over Alternatives

Try It (2 Minutes)

PySpark runs locally with zero cluster setup — a "local" Spark session simulates a cluster on your own machine:

pip install pyspark
python3

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("try-it").getOrCreate()
df = spark.createDataFrame(
    [("alice", 120), ("bob", 45), ("carol", 300)],
    ["name", "amount"]
)

# Transformations -- nothing runs yet
big_spenders = df.filter(df.amount > 100)

# Action -- THIS triggers execution
big_spenders.show()

Study Resources