Metrics

Airflow can be set up to send metrics to StatsD:

Configuration

Add the following lines to your configuration file e.g. airflow.cfg

[scheduler]
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow

Counters

Name Description
<job_name>_start Number of started <job_name> job, ex. SchedulerJob, LocalTaskJob
<job_name>_end Number of ended <job_name> job, ex. SchedulerJob, LocalTaskJob
operator_failures_<operator_name> Operator <operator_name> failures
operator_successes_<operator_name> Operator <operator_name> successes
ti_failures Overall task instances failures
ti_successes Overall task instances successes
zombies_killed Zombie tasks killed
scheduler_heartbeat Scheduler heartbeats

Gauges

Name Description
collect_dags Seconds taken to scan and import DAGs
dagbag_import_errors DAG import errors
dagbag_size DAG bag size
dag_processing.last_runtime.<dag_file> Seconds spent processing <dag_file> (in most recent iteration)
dag_processing.last_run.seconds_ago.<dag_file> Seconds since <dag_file> was last processed
executor.open_slots Number of of open slots on executor
executor.queued_tasks Number of queued tasks on executor
executor.running_tasks Number of running tasks on executor
pool.starving_tasks.<pool_name> Number of starving tasks in the pool

Timers

Name Description
dagrun.dependency-check.<dag_id> Seconds taken to check DAG dependencies
dag.<dag_id>.<task_id>.duration Seconds taken to finish a task
dagrun.duration.success.<dag_id> Seconds taken for a DagRun to reach success state
dagrun.duration.failed.<dag_id> Seconds taken for a DagRun to reach failed state