Command Line InterfaceΒΆ

Airflow has a very rich command line interface that allows for many types of operation on a DAG, starting services, and supporting development and testing.

usage: airflow [-h]
               {resetdb,render,variables,connections,pause,task_failed_deps,version,trigger_dag,initdb,test,unpause,dag_state,run,list_tasks,backfill,list_dags,kerberos,worker,webserver,flower,scheduler,task_state,pool,serve_logs,clear,upgradedb}
               ...
Required Arguments
subcommand

sub-command help

Possible choices: resetdb, render, variables, connections, pause, task_failed_deps, version, trigger_dag, initdb, test, unpause, dag_state, run, list_tasks, backfill, list_dags, kerberos, worker, webserver, flower, scheduler, task_state, pool, serve_logs, clear, upgradedb

Sub-commands:
resetdb

Burn down and rebuild the metadata database

usage: airflow resetdb [-h] [-y]
Optional Arguments
-y=False, --yes=False
 Do not prompt to confirm reset. Use with care!
render

Render a task instance’s template(s)

usage: airflow render [-h] [-sd SUBDIR] dag_id task_id execution_date
Required Arguments
dag_id The id of the dag
task_id The id of the task
execution_date The execution date of the DAG
Optional Arguments
-sd=/home/docs/airflow/dags, --subdir=/home/docs/airflow/dags
 File location or directory from which to look for the dag
variables

CRUD operations on variables

usage: airflow variables [-h] [-s KEY VAL] [-g KEY] [-j] [-d VAL]
                         [-i FILEPATH] [-e FILEPATH] [-x KEY]
Optional Arguments
-s, --set Set a variable
-g, --get Get value of a variable
-j=False, --json=False
 Deserialize JSON variable
-d, --default Default value returned if variable does not exist
-i, --import Import variables from JSON file
-e, --export Export variables to JSON file
-x, --delete Delete a variable
connections

List/Add/Delete connections

usage: airflow connections [-h] [-l] [-a] [-d] [--conn_id CONN_ID]
                           [--conn_uri CONN_URI] [--conn_extra CONN_EXTRA]
Optional Arguments
-l=False, --list=False
 List all connections
-a=False, --add=False
 Add a connection
-d=False, --delete=False
 Delete a connection
--conn_id Connection id, required to add/delete a connection
--conn_uri Connection URI, required to add a connection
--conn_extra Connection `Extra` field, optional when adding a connection
pause

Pause a DAG

usage: airflow pause [-h] [-sd SUBDIR] dag_id
Required Arguments
dag_id The id of the dag
Optional Arguments
-sd=/home/docs/airflow/dags, --subdir=/home/docs/airflow/dags
 File location or directory from which to look for the dag
task_failed_deps

Returns the unmet dependencies for a task instance from the perspective of the scheduler. In other words, why a task instance doesn’t get scheduled and then queued by the scheduler, and then run by an executor).

usage: airflow task_failed_deps [-h] [-sd SUBDIR]
                                dag_id task_id execution_date
Required Arguments
dag_id The id of the dag
task_id The id of the task
execution_date The execution date of the DAG
Optional Arguments
-sd=/home/docs/airflow/dags, --subdir=/home/docs/airflow/dags
 File location or directory from which to look for the dag
version

Show the version

usage: airflow version [-h]
trigger_dag

Trigger a DAG run

usage: airflow trigger_dag [-h] [-sd SUBDIR] [-r RUN_ID] [-c CONF]
                           [-e EXEC_DATE]
                           dag_id
Required Arguments
dag_id The id of the dag
Optional Arguments
-sd=/home/docs/airflow/dags, --subdir=/home/docs/airflow/dags
 File location or directory from which to look for the dag
-r, --run_id Helps to identify this run
-c, --conf JSON string that gets pickled into the DagRun’s conf attribute
-e, --exec_date
 The execution date of the DAG
initdb

Initialize the metadata database

usage: airflow initdb [-h]
test

Test a task instance. This will run a task without checking for dependencies or recording it’s state in the database.

usage: airflow test [-h] [-sd SUBDIR] [-dr] [-tp TASK_PARAMS]
                    dag_id task_id execution_date
Required Arguments
dag_id The id of the dag
task_id The id of the task
execution_date The execution date of the DAG
Optional Arguments
-sd=/home/docs/airflow/dags, --subdir=/home/docs/airflow/dags
 File location or directory from which to look for the dag
-dr=False, --dry_run=False
 Perform a dry run
-tp, --task_params
 Sends a JSON params dict to the task
unpause

Resume a paused DAG

usage: airflow unpause [-h] [-sd SUBDIR] dag_id
Required Arguments
dag_id The id of the dag
Optional Arguments
-sd=/home/docs/airflow/dags, --subdir=/home/docs/airflow/dags
 File location or directory from which to look for the dag
dag_state

Get the status of a dag run

usage: airflow dag_state [-h] [-sd SUBDIR] dag_id execution_date
Required Arguments
dag_id The id of the dag
execution_date The execution date of the DAG
Optional Arguments
-sd=/home/docs/airflow/dags, --subdir=/home/docs/airflow/dags
 File location or directory from which to look for the dag
run

Run a single task instance

usage: airflow run [-h] [-sd SUBDIR] [-m] [-f] [--pool POOL]
                   [--cfg_path CFG_PATH] [-l] [-A IGNORE_ALL_DEPENDENCIES]
                   [-i] [-I] [--ship_dag] [-p PICKLE]
                   dag_id task_id execution_date
Required Arguments
dag_id The id of the dag
task_id The id of the task
execution_date The execution date of the DAG
Optional Arguments
-sd=/home/docs/airflow/dags, --subdir=/home/docs/airflow/dags
 File location or directory from which to look for the dag
-m=False, --mark_success=False
 Mark jobs as succeeded without running them
-f=False, --force=False
 Ignore previous task instance state, rerun regardless if task already succeeded/failed
--pool Resource pool to use
--cfg_path Path to config file to use instead of airflow.cfg
-l=False, --local=False
 Run the task using the LocalExecutor
-A, --ignore_all_dependencies
 Ignores all non-critical dependencies, including ignore_ti_state and ignore_task_depsstore_true
-i=False, --ignore_dependencies=False
 Ignore task-specific dependencies, e.g. upstream, depends_on_past, and retry delay dependencies
-I=False, --ignore_depends_on_past=False
 Ignore depends_on_past dependencies (but respect upstream dependencies)
--ship_dag=False
 Pickles (serializes) the DAG and ships it to the worker
-p, --pickle Serialized pickle object of the entire dag (used internally)
list_tasks

List the tasks within a DAG

usage: airflow list_tasks [-h] [-t] [-sd SUBDIR] dag_id
Required Arguments
dag_id The id of the dag
Optional Arguments
-t=False, --tree=False
 Tree view
-sd=/home/docs/airflow/dags, --subdir=/home/docs/airflow/dags
 File location or directory from which to look for the dag
backfill

Run subsections of a DAG for a specified date range

usage: airflow backfill [-h] [-t TASK_REGEX] [-s START_DATE] [-e END_DATE]
                        [-m] [-l] [-x] [-a] [-i] [-I] [-sd SUBDIR]
                        [--pool POOL] [-dr]
                        dag_id
Required Arguments
dag_id The id of the dag
Optional Arguments
-t, --task_regex
 The regex to filter specific task_ids to backfill (optional)
-s, --start_date
 Override start_date YYYY-MM-DD
-e, --end_date Override end_date YYYY-MM-DD
-m=False, --mark_success=False
 Mark jobs as succeeded without running them
-l=False, --local=False
 Run the task using the LocalExecutor
-x=False, --donot_pickle=False
 Do not attempt to pickle the DAG object to send over to the workers, just tell the workers to run their version of the code.
-a=False, --include_adhoc=False
 Include dags with the adhoc parameter.
-i=False, --ignore_dependencies=False
 Skip upstream tasks, run only the tasks matching the regexp. Only works in conjunction with task_regex
-I=False, --ignore_first_depends_on_past=False
 Ignores depends_on_past dependencies for the first set of tasks only (subsequent executions in the backfill DO respect depends_on_past).
-sd=/home/docs/airflow/dags, --subdir=/home/docs/airflow/dags
 File location or directory from which to look for the dag
--pool Resource pool to use
-dr=False, --dry_run=False
 Perform a dry run
list_dags

List all the DAGs

usage: airflow list_dags [-h] [-sd SUBDIR] [-r]
Optional Arguments
-sd=/home/docs/airflow/dags, --subdir=/home/docs/airflow/dags
 File location or directory from which to look for the dag
-r=False, --report=False
 Show DagBag loading report
kerberos

Start a kerberos ticket renewer

usage: airflow kerberos [-h] [-kt [KEYTAB]] [--pid [PID]] [-D]
                        [--stdout STDOUT] [--stderr STDERR] [-l LOG_FILE]
                        [principal]
Required Arguments
principal=airflow
 kerberos principal
Optional Arguments
-kt=airflow.keytab, --keytab=airflow.keytab
 keytab
--pid PID file location
-D=False, --daemon=False
 Daemonize instead of running in the foreground
--stdout Redirect stdout to this file
--stderr Redirect stderr to this file
-l, --log-file Location of the log file
worker

Start a Celery worker node

usage: airflow worker [-h] [-p] [-q QUEUES] [-c CONCURRENCY] [--pid [PID]]
                      [-D] [--stdout STDOUT] [--stderr STDERR] [-l LOG_FILE]
Optional Arguments
-p=False, --do_pickle=False
 Attempt to pickle the DAG object to send over to the workers, instead of letting workers run their version of the code.
-q=default, --queues=default
 Comma delimited list of queues to serve
-c=16, --concurrency=16
 The number of worker processes
--pid PID file location
-D=False, --daemon=False
 Daemonize instead of running in the foreground
--stdout Redirect stdout to this file
--stderr Redirect stderr to this file
-l, --log-file Location of the log file
webserver

Start a Airflow webserver instance

usage: airflow webserver [-h] [-p PORT] [-w WORKERS]
                         [-k {sync,eventlet,gevent,tornado}]
                         [-t WORKER_TIMEOUT] [-hn HOSTNAME] [--pid [PID]] [-D]
                         [--stdout STDOUT] [--stderr STDERR]
                         [-A ACCESS_LOGFILE] [-E ERROR_LOGFILE] [-l LOG_FILE]
                         [--ssl_cert SSL_CERT] [--ssl_key SSL_KEY] [-d]
Optional Arguments
-p=8080, --port=8080
 The port on which to run the server
-w=4, --workers=4
 Number of workers to run the webserver on
-k=sync, --workerclass=sync
 

The worker class to use for Gunicorn

Possible choices: sync, eventlet, gevent, tornado

-t=120, --worker_timeout=120
 The timeout for waiting on webserver workers
-hn=0.0.0.0, --hostname=0.0.0.0
 Set the hostname on which to run the web server
--pid PID file location
-D=False, --daemon=False
 Daemonize instead of running in the foreground
--stdout Redirect stdout to this file
--stderr Redirect stderr to this file
-A=-, --access_logfile=-
 The logfile to store the webserver access log. Use ‘-‘ to print to stderr.
-E=-, --error_logfile=-
 The logfile to store the webserver error log. Use ‘-‘ to print to stderr.
-l, --log-file Location of the log file
--ssl_cert Path to the SSL certificate for the webserver
--ssl_key Path to the key to use with the SSL certificate
-d=False, --debug=False
 Use the server that ships with Flask in debug mode
flower

Start a Celery Flower

usage: airflow flower [-h] [-hn HOSTNAME] [-p PORT] [-fc FLOWER_CONF]
                      [-a BROKER_API] [--pid [PID]] [-D] [--stdout STDOUT]
                      [--stderr STDERR] [-l LOG_FILE]
Optional Arguments
-hn=0.0.0.0, --hostname=0.0.0.0
 Set the hostname on which to run the server
-p=5555, --port=5555
 The port on which to run the server
-fc, --flower_conf
 Configuration file for flower
-a, --broker_api
 Broker api
--pid PID file location
-D=False, --daemon=False
 Daemonize instead of running in the foreground
--stdout Redirect stdout to this file
--stderr Redirect stderr to this file
-l, --log-file Location of the log file
scheduler

Start a scheduler instance

usage: airflow scheduler [-h] [-d DAG_ID] [-sd SUBDIR] [-r RUN_DURATION]
                         [-n NUM_RUNS] [-p] [--pid [PID]] [-D]
                         [--stdout STDOUT] [--stderr STDERR] [-l LOG_FILE]
Optional Arguments
-d, --dag_id The id of the dag to run
-sd=/home/docs/airflow/dags, --subdir=/home/docs/airflow/dags
 File location or directory from which to look for the dag
-r, --run-duration
 Set number of seconds to execute before exiting
-n=-1, --num_runs=-1
 Set the number of runs to execute before exiting
-p=False, --do_pickle=False
 Attempt to pickle the DAG object to send over to the workers, instead of letting workers run their version of the code.
--pid PID file location
-D=False, --daemon=False
 Daemonize instead of running in the foreground
--stdout Redirect stdout to this file
--stderr Redirect stderr to this file
-l, --log-file Location of the log file
task_state

Get the status of a task instance

usage: airflow task_state [-h] [-sd SUBDIR] dag_id task_id execution_date
Required Arguments
dag_id The id of the dag
task_id The id of the task
execution_date The execution date of the DAG
Optional Arguments
-sd=/home/docs/airflow/dags, --subdir=/home/docs/airflow/dags
 File location or directory from which to look for the dag
pool

CRUD operations on pools

usage: airflow pool [-h] [-s NAME SLOT_COUNT POOL_DESCRIPTION] [-g NAME]
                    [-x NAME]
Optional Arguments
-s, --set Set pool slot count and description, respectively
-g, --get Get pool info
-x, --delete Delete a pool
serve_logs

Serve logs generate by worker

usage: airflow serve_logs [-h]
clear

Clear a set of task instance, as if they never ran

usage: airflow clear [-h] [-t TASK_REGEX] [-s START_DATE] [-e END_DATE]
                     [-sd SUBDIR] [-u] [-d] [-c] [-f] [-r] [-x]
                     dag_id
Required Arguments
dag_id The id of the dag
Optional Arguments
-t, --task_regex
 The regex to filter specific task_ids to backfill (optional)
-s, --start_date
 Override start_date YYYY-MM-DD
-e, --end_date Override end_date YYYY-MM-DD
-sd=/home/docs/airflow/dags, --subdir=/home/docs/airflow/dags
 File location or directory from which to look for the dag
-u=False, --upstream=False
 Include upstream tasks
-d=False, --downstream=False
 Include downstream tasks
-c=False, --no_confirm=False
 Do not request confirmation
-f=False, --only_failed=False
 Only failed jobs
-r=False, --only_running=False
 Only running jobs
-x=False, --exclude_subdags=False
 Exclude subdags
upgradedb

Upgrade the metadata database to latest version

usage: airflow upgradedb [-h]