Apache Spark — Interview Q&A
Q: Explain the difference between a transformation and an action, and why it matters.
A: Transformations (filter, select, groupBy) are lazy — they build a logical execution plan without processing any data. Actions (show, collect, count, write) trigger actual execution of the entire accumulated transformation chain at once. This matters because Spark's optimizer can reason about and optimize the whole chain together rather than executing each step eagerly and separately — it's fundamentally why Spark can be efficient at scale, and it also explains behavior that surprises people new to Spark, like a print() between transformations not showing intermediate results the way it would in ordinary eager code.
Q: What is a shuffle, and why is it usually the most expensive part of a Spark job?
A: A shuffle is data movement across the network between worker nodes, required whenever an operation (a join, a groupBy) needs rows with matching keys to be co-located on the same node, but those rows started on different nodes. It's expensive because it involves network I/O and disk I/O for spilling intermediate data, which is orders of magnitude slower than in-memory computation — minimizing unnecessary shuffles (via broadcast joins, sensible partitioning) is one of the most impactful things you can do for Spark job performance.
Q: When would you use a broadcast join instead of a regular join?
A: When one side of the join is small enough to fit comfortably in memory on every worker node — broadcasting sends a full copy to every worker, avoiding the shuffle a regular join would require entirely. Spark's optimizer auto-broadcasts tables below a configurable size threshold, but explicit broadcasting is useful when you know a table is small and want to guarantee the optimization, especially in cases where automatic size estimation might not trigger it (after certain transformations that obscure the true size from the optimizer, for instance).
Q: How would you diagnose why a specific Spark job is running slowly?
A: Start with the Spark UI, not guesswork — look at the stage-level breakdown for which stage is actually taking the most time, check for shuffle read/write volume (an unexpectedly large shuffle often points to a missing broadcast opportunity or poor partitioning), and check task-level timing within a stage for skew (a small number of tasks taking dramatically longer than the rest, pointing to data skew on specific keys). df.explain() complements this by showing whether the optimizer is actually doing what you expect (a broadcast join actually happening, a filter being pushed down early). Diagnostic process, not a memorized list of possible causes, is the actual signal interviewers are looking for here.
Q: What's data skew, and how would you handle it?
A: Skew is an uneven distribution of data across partitions, usually caused by a small number of keys having disproportionately many rows — the partition(s) with those keys take much longer to process, and the whole job waits on them. A common mitigation is salting: adding a random suffix to a skewed key to artificially spread its rows across multiple partitions for the expensive operation, then aggregating the results back together afterward. This should be applied based on confirmed skew (visible in the Spark UI as dramatically uneven task timing), not preemptively without evidence.
Q: Explain the difference between the complete, append, and update output modes in Structured Streaming.
A: complete re-outputs the entire aggregated result table on every trigger — only viable when the aggregated result set stays bounded in size. append outputs only new rows since the last trigger, appropriate for non-aggregated data or finalized windowed aggregations. update outputs only the rows that changed since the last trigger. Picking the wrong mode for a given query type is a common, serious mistake — sometimes it errors outright, but sometimes it silently produces technically-valid but semantically-wrong output, which is the more dangerous failure mode since it doesn't announce itself.
Q: When would you cache a DataFrame, and when is caching actually wasteful?
A: Cache when you're going to use the same DataFrame multiple times across different downstream operations — without caching, Spark's lazy evaluation means each subsequent action recomputes the entire transformation chain from scratch, including re-reading source data. Caching a DataFrame you only use once provides no benefit and adds real memory overhead — a common beginner mistake is caching reflexively rather than based on actual, confirmed reuse in the pipeline.