Apache Airflow — Interview Q&A
Q: What's the difference between a DAG, a DAG Run, and a Task Instance?
A: A DAG is the workflow's structural definition — the tasks and their dependencies, defined once in code. A DAG Run is one specific execution of that structure at a specific scheduled (or manually triggered) time. A Task Instance is one specific task, within one specific DAG Run. Airflow tracks state at the task instance level, which is what enables re-running a single failed task without re-running an entire otherwise-successful DAG Run — a precise answer distinguishes all three explicitly, since conflating them is a common way to give an imprecise answer.
Q: Explain catchup and why it matters.
A: If catchup=True, Airflow automatically triggers a DAG Run for every scheduled interval between the DAG's start_date and now that hasn't already run — a DAG with a year-old start_date, deployed today, would immediately trigger hundreds of backfilled runs unless catchup=False is set. This matters because it's a common, real surprise for people new to Airflow, and the safer default for most new DAGs is explicitly setting catchup=False unless automatic backfilling is genuinely intended.
Q: Why does idempotency matter so much specifically in Airflow, more than in a typical one-off script?
A: Because retries and manual re-runs are routine, expected operations in Airflow, not rare exceptions — a task might legitimately run more than once for the same logical date (a transient failure triggering a retry, an operator manually re-triggering a past run to fix a downstream issue). A non-idempotent task (one that appends rather than overwrites, for instance) silently produces incorrect, duplicated results every time this routine re-execution happens, which makes idempotency a core design requirement for Airflow tasks specifically, not just a general best practice.
Q: What are XComs, and what should you NOT use them for?
A: XComs let tasks pass small pieces of data to each other via Airflow's metadata database. They should not be used to pass large datasets between tasks — the metadata database isn't built for that, and doing so is a real, common performance and reliability problem. The correct pattern for passing actual data is writing it to real storage (S3, a database) and passing a reference (a path or ID) via XCom instead of the data itself.
Q: When would you use mode="reschedule" on a sensor instead of the default mode="poke"?
A: For sensors with potentially long wait times — poke mode holds a worker slot occupied for the entire wait, repeatedly checking; reschedule mode releases the worker slot between checks and re-queues the task, which is far more resource-efficient. Using poke mode for a sensor that might wait hours is a real, common way to exhaust available worker capacity in a busy Airflow deployment, since every poking sensor is a worker slot unavailable for other tasks the entire time it's waiting.
Q: How does branching interact with trigger rules, and why is this a common source of confusion?
A: BranchPythonOperator marks the non-chosen path's tasks as "skipped," not simply not-run — and a downstream task's default trigger rule (all_success) treats a skipped upstream task differently than you might assume, which can cause downstream tasks to not run when you expected them to, or vice versa. Trigger rules like none_failed (accepting skipped as fine, only failing on an actual failure) are often what's actually needed for tasks downstream of a branch point — this interaction is genuinely non-obvious and a common real source of unexpected DAG behavior.
Q: How would you decide between LocalExecutor, CeleryExecutor, and KubernetesExecutor for a production deployment?
A: LocalExecutor is appropriate for small-scale deployments where single-machine capacity is genuinely sufficient. CeleryExecutor distributes tasks across a worker pool once that single-machine capacity isn't enough, without needing per-task container isolation. KubernetesExecutor runs each task in its own pod, providing strong resource isolation and the ability to give different tasks different resource allocations or even different images — valuable when tasks have meaningfully different resource/dependency needs, at the cost of real added operational complexity (running Airflow on Kubernetes, pod startup overhead per task). The decision should follow actual scale and isolation requirements, not be made preemptively based on which sounds most sophisticated.