Airflow Sensors & Deferrable Operators

A sensor is a special kind of operator that waits for a condition to become true before letting downstream tasks proceed — a file to appear, another DAG’s task to finish, a database row to exist, an HTTP endpoint to return a specific status. Sensors are how Airflow pipelines react to external events rather than only running on a fixed clock schedule.

Common Built-In Sensors

from airflow.sensors.filesystem import FileSensor
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
from airflow.sensors.external_task import ExternalTaskSensor

wait_for_file = FileSensor(
    task_id="wait_for_upload",
    filepath="/data/incoming/orders.csv",
    poke_interval=60,
    timeout=60 * 60,
)

wait_for_s3_object = S3KeySensor(
    task_id="wait_for_s3_export",
    bucket_key="s3://exports/orders/{{ ds }}/complete.flag",
    poke_interval=120,
)

wait_for_upstream_dag = ExternalTaskSensor(
    task_id="wait_for_upstream",
    external_dag_id="orders_extract",
    external_task_id="load_complete",
)

Every sensor shares the same two key parameters: poke_interval (how often to check the condition) and timeout (how long to keep checking before failing the task entirely) — without a timeout, a sensor waiting on a condition that never arrives will run forever, silently occupying resources.

The Problem with Classic “Poke Mode” Sensors

By default, a sensor runs in poke mode: it occupies a worker slot for its entire waiting period, checking the condition on each poke_interval, sleeping in between. A sensor waiting up to an hour for a file, checking every 60 seconds, ties up a worker slot for that full hour even though it’s actively “doing work” for a fraction of a second per check. At scale — dozens of sensors waiting on slow upstream systems — this can exhaust worker capacity entirely, a well-known failure mode called sensor starvation.

Deferrable Operators: The Fix

Airflow 2.2+ introduced deferrable operators (and deferrable-capable sensors), which solve this by releasing the worker slot entirely while waiting, handing the “keep checking” responsibility to the lightweight triggerer process instead — an asyncio event loop that can efficiently manage thousands of concurrent waits without needing thousands of worker slots.

wait_for_file = FileSensor(
    task_id="wait_for_upload",
    filepath="/data/incoming/orders.csv",
    poke_interval=60,
    timeout=60 * 60,
    mode="reschedule",   # releases the worker slot between pokes, instead of blocking it
)

mode="reschedule" is a middle ground available on all sensors — it frees the worker slot between individual pokes (rather than the whole waiting period), at the cost of slightly more scheduling overhead per check. True deferrable operators (built on BaseSensorOperator’s async support, or written explicitly as @task.sensor / deferrable custom operators) go further, using the triggerer’s event loop instead of polling on a schedule at all for sensors that support event-based waiting.

Choosing a Mode

Common Mistakes

Leaving dozens of long-running sensors in default poke mode, exhausting the worker pool and causing unrelated DAGs to queue behind them — a classic Airflow scaling incident.

Omitting timeout, leaving a sensor able to wait indefinitely for a condition that, due to an upstream failure, will simply never occur — always set a timeout matched to a realistic worst-case wait.

Using a sensor at all when a simpler dependency (e.g., ExternalTaskSensor replaced by a proper cross-DAG trigger, or a direct upstream task dependency) would do — sensors add polling overhead and latency; only reach for one when genuinely waiting on an external, non-Airflow-controlled condition.

Frequently Asked Questions

Do all sensors support deferrable/async mode? No — it depends on the specific sensor’s implementation; check the provider’s documentation for a deferrable=True parameter or an async-capable version of the sensor you need.

Does reschedule mode delay how quickly the sensor reacts once the condition is true? Marginally — there’s a small scheduling overhead reacquiring a slot versus already holding one in poke mode, but for waits measured in minutes or longer this is negligible.

Is the triggerer required for reschedule mode? No — reschedule works via the standard scheduler/executor mechanism; the triggerer is specifically needed for true async deferrable operators, not for reschedule mode.

Summary

Sensors let pipelines react to external conditions rather than only fixed schedules, but the default poke-mode implementation has a real resource cost worth understanding before deploying dozens of them. mode="reschedule" is a low-effort improvement for longer waits, and true deferrable operators — powered by the triggerer covered in Airflow Architecture — are the fully resource-efficient answer where the specific sensor supports them.