Task Dependencies & the Bitshift Operators in Airflow

The >> and << operators you see in nearly every Airflow DAG aren’t special DAG syntax — they’re Python’s actual bitshift operators, overloaded by Airflow’s BaseOperator class to mean “set as downstream” and “set as upstream” respectively. Understanding that they’re just syntactic sugar over set_upstream()/set_downstream() method calls demystifies a lot of dependency-graph code that otherwise looks like magic.

The Two Equivalent Forms

# Bitshift form (idiomatic, most common)
extract >> transform >> load

# Explicit method form (identical result)
extract.set_downstream(transform)
transform.set_downstream(load)

Both produce the exact same dependency graph. The bitshift form won out as the convention because it reads left-to-right the same way the data flows.

Fan-Out: One Task, Multiple Downstream

extract >> [validate_schema, validate_row_counts, validate_nulls]

All three validation tasks become downstream of extract and run in parallel (as parallel as your executor’s available worker capacity allows) once extract succeeds — none of them depend on each other.

Fan-In: Multiple Tasks, One Downstream

[validate_schema, validate_row_counts, validate_nulls] >> load

load won’t start until all three validation tasks have succeeded — by default, a task only runs once every one of its upstream dependencies has completed successfully (this default is controlled by the task’s trigger_rule, covered below).

Combining Both

extract >> [validate_schema, validate_row_counts, validate_nulls] >> load >> notify

This chains a fan-out into a fan-in into a straight line — a genuinely common real-world shape: one extraction, several independent checks running in parallel, a join point, then a final notification step.

Cross Dependencies with cross_downstream

Sometimes every task in one group needs to depend on every task in another group — not a single fan-in/fan-out, but a full cross product:

from airflow.models.baseoperator import cross_downstream

cross_downstream([extract_a, extract_b], [transform_a, transform_b])
# equivalent to: extract_a >> transform_a, extract_a >> transform_b,
#                extract_b >> transform_a, extract_b >> transform_b

Writing this manually with bitshift operators for anything beyond a 2x2 grid gets unwieldy fast — cross_downstream (and its sibling chain, for building long linear or partially-parallel sequences programmatically) exist specifically to avoid that.

Trigger Rules: Changing “All Must Succeed”

By default, a task’s trigger_rule is all_success — every upstream task must succeed for it to run. This can be overridden:

from airflow.operators.python import PythonOperator

cleanup = PythonOperator(
    task_id="cleanup",
    python_callable=do_cleanup,
    trigger_rule="all_done",   # runs regardless of whether upstream tasks succeeded or failed
)

Common trigger rules: all_success (default), all_failed, all_done (runs once upstream tasks finish, regardless of outcome — the standard choice for cleanup/notification tasks that should run “no matter what”), one_failed (useful for alerting tasks), and none_failed.

Common Mistakes

Assuming fan-out tasks run truly simultaneously. They’re only eligible to run in parallel — actual concurrency is bounded by executor worker capacity and DAG/task concurrency limits (max_active_tasks, pool slots). With one worker, “parallel” tasks still run one at a time.

Forgetting to set trigger_rule="all_done" on cleanup/notification tasks, which then silently never run at all if any upstream task in the DAG fails — exactly the scenario where you’d most want the notification to fire.

Building deeply nested dependency logic with manual >> chains instead of chain()/cross_downstream() when the pattern is regular — it becomes unreadable and error-prone past a handful of tasks.

Frequently Asked Questions

Can a task depend on a task in a different DAG? Yes, via ExternalTaskSensor, which polls another DAG’s task for completion — this is a cross-DAG dependency, not a same-DAG bitshift dependency.

What happens if I create a circular dependency? Airflow will raise a AirflowDagCycleException at DAG parse time — DAGs are, by definition, acyclic, and Airflow enforces this rather than allowing it to fail silently at runtime.

Does dependency order in the file matter? No — what matters is the >>/<< relationships you declare, not the order task objects are defined in the Python file; Airflow builds the graph from the relationships, not from source-code position.

Summary

>> and << are just readable syntax over set_downstream()/set_upstream(), and combined with list syntax for fan-out/fan-in and helpers like chain()/cross_downstream() for regular patterns, they’re expressive enough to represent almost any real pipeline shape. trigger_rule is the other half of the picture — it’s what determines whether “all upstream must succeed” actually holds for a given task, which matters most for cleanup and alerting tasks that need to run regardless of what happened upstream.