Core Airflow Operators: PythonOperator, BashOperator & EmptyOperator

An operator is a template for a single task — it defines what kind of work gets done (run a Python function, execute a shell command, wait for a file) while the DAG defines when and in what order. Airflow ships dozens of operators, but nearly every pipeline is built primarily from a small handful of foundational ones.

BashOperator

Runs a shell command as a subprocess. Straightforward, and useful as a thin wrapper around any existing script or CLI tool.

from airflow.operators.bash import BashOperator

run_script = BashOperator(
    task_id="run_etl_script",
    bash_command="python /opt/scripts/etl.py --date {{ ds }}",
)

{{ ds }} is a Jinja template that Airflow renders at task execution time into the run’s logical date (YYYY-MM-DD) — bash_command (and many other operator string fields) are Jinja-templated by default, which is how DAGs pass runtime context into shell commands without hardcoding dates.

A subtlety worth knowing: BashOperator’s exit-code handling treats any non-zero exit code as task failure, which is exactly bash’s own convention — so a script that exits 1 on error “just works” for retry/alerting purposes with no extra plumbing.

PythonOperator

Runs a Python callable directly, in-process (or in a subprocess/container depending on executor) — no shell wrapping required.

from airflow.operators.python import PythonOperator

def process_batch(batch_size, **context):
    print(f"Processing {batch_size} records for {context['ds']}")

process = PythonOperator(
    task_id="process_batch",
    python_callable=process_batch,
    op_kwargs={"batch_size": 500},
)

**context captures Airflow’s runtime context (logical date, DAG run, task instance, and more) if your function needs it — you don’t have to declare every context key individually. op_kwargs passes your own explicit arguments alongside it.

A real gotcha: whatever you pass via op_kwargs/op_args must be serializable, because with some executors (notably KubernetesExecutor or when using PythonVirtualenvOperator) the callable and its arguments are pickled to be sent to a separate process. Passing something like an open database connection as an argument will fail — pass connection parameters and open the connection inside the function instead.

EmptyOperator

Does literally nothing at runtime — its only purpose is to exist as a node in the DAG graph, most commonly to represent a join point where several parallel branches need to converge before downstream work continues.

from airflow.operators.empty import EmptyOperator

start = EmptyOperator(task_id="start")
join = EmptyOperator(task_id="join_all_extracts")

start >> [extract_a, extract_b, extract_c] >> join >> load

Without join, load would need three separate upstream dependencies listed individually — EmptyOperator makes the graph’s structure visually and semantically clearer, which matters more than it sounds once a DAG has dozens of tasks.

Choosing Between BashOperator and PythonOperator

Prefer PythonOperator (or the @task TaskFlow decorator — see The TaskFlow API) when the logic is genuinely Python and benefits from Airflow’s native context, error handling, and XCom integration. Prefer BashOperator when you’re wrapping an existing, independently-maintained script or CLI tool where re-implementing its logic in Python would just be needless duplication — calling dbt run or a compiled binary via BashOperator is entirely reasonable.

Provider-Specific Operators

Beyond these foundational three, most real pipelines also use provider operators — pre-built integrations for specific systems (S3ToRedshiftOperator, BigQueryInsertJobOperator, SnowflakeOperator, and hundreds more), covered in Airflow Providers & Connections. The core operators above are the building blocks; providers are what make Airflow genuinely useful for talking to real infrastructure without writing that integration code yourself.

Common Mistakes

Wrapping every single shell command in its own task. Excessive task granularity adds scheduling overhead and clutters the graph view without adding real value — group tightly-coupled steps into one task where splitting them provides no independent retry/monitoring benefit.

Passing unpicklable objects through op_kwargs. Open file handles, database connections, and most third-party client objects should be constructed inside the callable, not passed in from outside.

Using BashOperator to invoke Python scripts that could just be PythonOperator callables, losing native XCom integration and Airflow’s Python-level exception handling and traceback capture in the process.

Frequently Asked Questions

Does PythonOperator run in the same process as the scheduler? No — it runs within the worker process handling that task (a Celery worker, a Kubernetes pod, or a local subprocess depending on executor), never inside the scheduler itself.

Can I return a value from a PythonOperator callable? Yes — the return value is automatically pushed to XCom, making it available to downstream tasks; see XComs & Variables.

Is EmptyOperator the same as the older DummyOperator? Yes — DummyOperator was renamed to EmptyOperator in Airflow 2.4+; both names may appear in code depending on the codebase’s Airflow version history, but EmptyOperator is the current one.

Summary

BashOperator, PythonOperator, and EmptyOperator cover the vast majority of what a DAG’s structural skeleton needs: run a command, run a function, or mark a structural join point. Everything more specialized — talking to a specific cloud service, a specific database, a specific API — is handled by provider operators built on the same underlying BaseOperator contract.