Operators and Hooks Reference

ASF: Apache Software Foundation

Airflow supports various software created by Apache Software Foundation.

Software operators and hooks

These integrations allow you to perform various operations within software developed by Apache Software Foundation.

Service name

Guides

Hook

Operator

Sensor

Apache Cassandra

airflow.providers.apache.cassandra.hooks.cassandra

airflow.providers.apache.cassandra.sensors.record, airflow.providers.apache.cassandra.sensors.table

Apache Druid

airflow.providers.apache.druid.hooks.druid

airflow.providers.apache.druid.operators.druid, airflow.providers.apache.druid.operators.druid_check

Apache Hive

airflow.providers.apache.hive.hooks.hive

airflow.providers.apache.hive.operators.hive, airflow.providers.apache.hive.operators.hive_stats

airflow.providers.apache.hive.sensors.named_hive_partition, airflow.providers.apache.hive.sensors.hive_partition, airflow.providers.apache.hive.sensors.metastore_partition

Apache Livy

airflow.providers.apache.livy.hooks.livy

airflow.providers.apache.livy.operators.livy

airflow.providers.apache.livy.sensors.livy

Apache Pig

airflow.providers.apache.pig.hooks.pig

airflow.providers.apache.pig.operators.pig

Apache Pinot

airflow.providers.apache.pinot.hooks.pinot

Apache Spark

How to use

airflow.providers.apache.spark.hooks.spark_jdbc, airflow.providers.apache.spark.hooks.spark_jdbc_script, airflow.providers.apache.spark.hooks.spark_sql, airflow.providers.apache.spark.hooks.spark_submit

airflow.providers.apache.spark.operators.spark_jdbc, airflow.providers.apache.spark.operators.spark_sql, airflow.providers.apache.spark.operators.spark_submit

Apache Sqoop

airflow.providers.apache.sqoop.hooks.sqoop

airflow.providers.apache.sqoop.operators.sqoop

Hadoop Distributed File System (HDFS)

airflow.providers.apache.hdfs.hooks.hdfs

airflow.providers.apache.hdfs.sensors.hdfs

WebHDFS

airflow.providers.apache.hdfs.hooks.webhdfs

airflow.providers.apache.hdfs.sensors.web_hdfs

Azure: Microsoft Azure

Airflow has limited support for Microsoft Azure.

AWS: Amazon Web Services

Airflow has support for Amazon Web Services.

All hooks are based on airflow.providers.amazon.aws.hooks.base_aws.

Service operators and hooks

These integrations allow you to perform various operations within the Amazon Web Services.

Service name

Guide

Hook

Operator

Sensor

AWS Batch

airflow.providers.amazon.aws.hooks.batch_client, airflow.providers.amazon.aws.hooks.batch_waiters

airflow.providers.amazon.aws.operators.batch

AWS DataSync

How to use

airflow.providers.amazon.aws.hooks.datasync

airflow.providers.amazon.aws.operators.datasync

AWS Glue Catalog

airflow.providers.amazon.aws.hooks.glue_catalog

airflow.providers.amazon.aws.sensors.glue_catalog_partition

AWS Glue

airflow.providers.amazon.aws.hooks.glue

airflow.providers.amazon.aws.operators.glue

airflow.providers.amazon.aws.sensors.glue

AWS Lambda

airflow.providers.amazon.aws.hooks.lambda_function

Amazon Athena

airflow.providers.amazon.aws.hooks.athena

airflow.providers.amazon.aws.operators.athena

airflow.providers.amazon.aws.sensors.athena

Amazon CloudFormation

airflow.providers.amazon.aws.hooks.cloud_formation

airflow.providers.amazon.aws.operators.cloud_formation

airflow.providers.amazon.aws.sensors.cloud_formation

Amazon CloudWatch Logs

airflow.providers.amazon.aws.hooks.logs

Amazon DynamoDB

airflow.providers.amazon.aws.hooks.aws_dynamodb

Amazon EC2

airflow.providers.amazon.aws.hooks.ec2

airflow.providers.amazon.aws.operators.ec2_start_instance, airflow.providers.amazon.aws.operators.ec2_stop_instance,

airflow.providers.amazon.aws.sensors.ec2_instance_state

Amazon ECS

How to use

airflow.providers.amazon.aws.operators.ecs

Amazon EMR

How to use

airflow.providers.amazon.aws.hooks.emr

airflow.providers.amazon.aws.operators.emr_add_steps, airflow.providers.amazon.aws.operators.emr_create_job_flow, airflow.providers.amazon.aws.operators.emr_terminate_job_flow, airflow.providers.amazon.aws.operators.emr_modify_cluster

airflow.providers.amazon.aws.sensors.emr_base, airflow.providers.amazon.aws.sensors.emr_job_flow, airflow.providers.amazon.aws.sensors.emr_step

Amazon Kinesis Data Firehose

airflow.providers.amazon.aws.hooks.kinesis

Amazon Redshift

airflow.providers.amazon.aws.hooks.redshift

airflow.providers.amazon.aws.sensors.redshift

Amazon SageMaker

airflow.providers.amazon.aws.hooks.sagemaker

airflow.providers.amazon.aws.operators.sagemaker_base, airflow.providers.amazon.aws.operators.sagemaker_endpoint_config, airflow.providers.amazon.aws.operators.sagemaker_endpoint, airflow.providers.amazon.aws.operators.sagemaker_model, airflow.providers.amazon.aws.operators.sagemaker_training, airflow.providers.amazon.aws.operators.sagemaker_transform, airflow.providers.amazon.aws.operators.sagemaker_tuning

airflow.providers.amazon.aws.sensors.sagemaker_base, airflow.providers.amazon.aws.sensors.sagemaker_endpoint, airflow.providers.amazon.aws.sensors.sagemaker_training, airflow.providers.amazon.aws.sensors.sagemaker_transform, airflow.providers.amazon.aws.sensors.sagemaker_tuning

Amazon Simple Notification Service (SNS)

airflow.providers.amazon.aws.hooks.sns

airflow.providers.amazon.aws.operators.sns

Amazon Simple Queue Service (SQS)

airflow.providers.amazon.aws.hooks.sqs

airflow.providers.amazon.aws.operators.sqs

airflow.providers.amazon.aws.sensors.sqs

Amazon Simple Storage Service (S3)

airflow.providers.amazon.aws.hooks.s3

airflow.providers.amazon.aws.operators.s3_bucket, airflow.providers.amazon.aws.operators.s3_file_transform, airflow.providers.amazon.aws.operators.s3_copy_object, airflow.providers.amazon.aws.operators.s3_delete_objects, airflow.providers.amazon.aws.operators.s3_list

airflow.providers.amazon.aws.sensors.s3_key, airflow.providers.amazon.aws.sensors.s3_prefix

Transfer operators and hooks

These integrations allow you to copy data from/to Amazon Web Services.

Source

Destination

Guide

Operator

All GCP services [1]

Amazon Simple Storage Service (S3)

How to use

airflow.providers.amazon.aws.operators.google_api_to_s3_transfer

Amazon DataSync

Amazon Simple Storage Service (S3)

How to use

airflow.providers.amazon.aws.operators.datasync

Amazon DynamoDB

Amazon Simple Storage Service (S3)

airflow.providers.amazon.aws.operators.dynamodb_to_s3

Amazon Redshift

Amazon Simple Storage Service (S3)

airflow.providers.amazon.aws.operators.redshift_to_s3

Amazon Simple Storage Service (S3)

Amazon Redshift

airflow.providers.amazon.aws.operators.s3_to_redshift

Amazon Simple Storage Service (S3)

Snowflake

airflow.providers.snowflake.operators.s3_to_snowflake

Amazon Simple Storage Service (S3)

Apache Hive

airflow.providers.apache.hive.operators.s3_to_hive

Amazon Simple Storage Service (S3)

Google Cloud Storage (GCS)

How to use

airflow.providers.google.cloud.operators.s3_to_gcs, airflow.providers.google.cloud.operators.cloud_storage_transfer_service

Amazon Simple Storage Service (S3)

SSH File Transfer Protocol (SFTP)

airflow.providers.amazon.aws.operators.s3_to_sftp

Apache Hive

Amazon DynamoDB

airflow.providers.amazon.aws.operators.hive_to_dynamodb

Google Cloud Storage (GCS)

Amazon Simple Storage Service (S3)

airflow.providers.amazon.aws.operators.gcs_to_s3

Internet Message Access Protocol (IMAP)

Amazon Simple Storage Service (S3)

How to use

airflow.providers.amazon.aws.operators.imap_attachment_to_s3

MongoDB

Amazon Simple Storage Service (S3)

airflow.providers.amazon.aws.operators.mongo_to_s3

SSH File Transfer Protocol (SFTP)

Amazon Simple Storage Service (S3)

airflow.providers.amazon.aws.operators.sftp_to_s3

[1] Those discovery-based operators use GoogleDiscoveryApiHook to communicate with Google Services via the Google API Python Client. Please note that this library is in maintenance mode hence it won’t fully support GCP in the future. Therefore it is recommended that you use the custom GCP Service Operators for working with the Google Cloud Platform.

Google

Airflow has support for the Google service.

All hooks are based on airflow.providers.google.common.hooks.base_google.GoogleBaseHook. Some integration also use airflow.providers.google.common.hooks.discovery_api.

See the GCP connection type documentation to configure connections to Google services.

GCP: Google Cloud Platform

Airflow has extensive support for the Google Cloud Platform.

Note

You can learn how to use Google Cloud Platform integrations by analyzing the source code of the Google Cloud Platform example DAGs

Service operators and hooks

These integrations allow you to perform various operations within the Google Cloud Platform.

Service name

Guide

Hook

Operator

Sensor

AutoML

How to use

airflow.providers.google.cloud.hooks.automl

airflow.providers.google.cloud.operators.automl

BigQuery

How to use

airflow.providers.google.cloud.hooks.bigquery

airflow.providers.google.cloud.operators.bigquery

airflow.providers.google.cloud.sensors.bigquery

BigQuery Data Transfer Service

How to use

airflow.providers.google.cloud.hooks.bigquery_dts

airflow.providers.google.cloud.operators.bigquery_dts

airflow.providers.google.cloud.sensors.bigquery_dts

Bigtable

How to use

airflow.providers.google.cloud.hooks.bigtable

airflow.providers.google.cloud.operators.bigtable

airflow.providers.google.cloud.sensors.bigtable

Cloud Build

How to use

airflow.providers.google.cloud.hooks.cloud_build

airflow.providers.google.cloud.operators.cloud_build

Compute Engine

How to use

airflow.providers.google.cloud.hooks.compute

airflow.providers.google.cloud.operators.compute

Cloud Data Loss Prevention (DLP)

airflow.providers.google.cloud.hooks.dlp

airflow.providers.google.cloud.operators.dlp

DataFusion

How to use

airflow.providers.google.cloud.hooks.datafusion

airflow.providers.google.cloud.operators.datafusion

Datacatalog

How to use

airflow.providers.google.cloud.hooks.datacatalog

airflow.providers.google.cloud.operators.datacatalog

Dataflow

airflow.providers.google.cloud.hooks.dataflow

airflow.providers.google.cloud.operators.dataflow

Dataproc

airflow.providers.google.cloud.hooks.dataproc

airflow.providers.google.cloud.operators.dataproc

Datastore

airflow.providers.google.cloud.hooks.datastore

airflow.providers.google.cloud.operators.datastore

Cloud Functions

How to use

airflow.providers.google.cloud.hooks.functions

airflow.providers.google.cloud.operators.functions

Cloud Firestore

How to use

airflow.providers.google.firebase.hooks.firestore

airflow.providers.google.firebase.operators.firestore

Cloud Key Management Service (KMS)

airflow.providers.google.cloud.hooks.kms

Cloud Life Sciences

How to use

airflow.providers.google.cloud.hooks.life_sciences

airflow.providers.google.cloud.operators.life_sciences

Kubernetes Engine

airflow.providers.google.cloud.hooks.kubernetes_engine

airflow.providers.google.cloud.operators.kubernetes_engine

Machine Learning Engine

airflow.providers.google.cloud.hooks.mlengine

airflow.providers.google.cloud.operators.mlengine

Cloud Memorystore

How to use

airflow.providers.google.cloud.hooks.cloud_memorystore

airflow.providers.google.cloud.operators.cloud_memorystore

Natural Language

How to use

airflow.providers.google.cloud.hooks.natural_language

airflow.providers.google.cloud.operators.natural_language

Cloud Pub/Sub

How to use

airflow.providers.google.cloud.hooks.pubsub

airflow.providers.google.cloud.operators.pubsub

airflow.providers.google.cloud.sensors.pubsub

Cloud Spanner

How to use

airflow.providers.google.cloud.hooks.spanner

airflow.providers.google.cloud.operators.spanner

Cloud Speech-to-Text

How to use

airflow.providers.google.cloud.hooks.speech_to_text

airflow.providers.google.cloud.operators.speech_to_text

Cloud SQL

How to use

airflow.providers.google.cloud.hooks.cloud_sql

airflow.providers.google.cloud.operators.cloud_sql

Cloud Stackdriver

How to use

airflow.providers.google.cloud.hooks.stackdriver

airflow.providers.google.cloud.operators.stackdriver

Cloud Storage (GCS)

How to use

airflow.providers.google.cloud.hooks.gcs

airflow.providers.google.cloud.operators.gcs

airflow.providers.google.cloud.sensors.gcs

Storage Transfer Service

How to use

airflow.providers.google.cloud.hooks.cloud_storage_transfer_service

airflow.providers.google.cloud.operators.cloud_storage_transfer_service

airflow.providers.google.cloud.sensors.cloud_storage_transfer_service

Cloud Tasks

airflow.providers.google.cloud.hooks.tasks

airflow.providers.google.cloud.operators.tasks

Cloud Text-to-Speech

How to use

airflow.providers.google.cloud.hooks.text_to_speech

airflow.providers.google.cloud.operators.text_to_speech

Cloud Translation

How to use

airflow.providers.google.cloud.hooks.translate

airflow.providers.google.cloud.operators.translate

Cloud Video Intelligence

How to use

airflow.providers.google.cloud.hooks.video_intelligence

airflow.providers.google.cloud.operators.video_intelligence

Cloud Vision

How to use

airflow.providers.google.cloud.hooks.vision

airflow.providers.google.cloud.operators.vision

Transfer operators and hooks

These integrations allow you to copy data from/to Google Cloud Platform.

Source

Destination

Guide

Operator

All services [1]

Amazon Simple Storage Service (S3)

How to use

airflow.providers.amazon.aws.operators.google_api_to_s3_transfer

Amazon Simple Storage Service (S3)

Google Cloud Storage (GCS)

How to use

airflow.providers.google.cloud.operators.s3_to_gcs, airflow.providers.google.cloud.operators.cloud_storage_transfer_service

Apache Cassandra

Google Cloud Storage (GCS)

airflow.providers.google.cloud.operators.cassandra_to_gcs

Azure Data Lake Storage

Google Cloud Storage (GCS)

airflow.providers.google.cloud.operators.adls_to_gcs

Facebook Ads

Google Cloud Storage (GCS)

How to use

airflow.providers.google.cloud.operators.facebook_ads_to_gcs

Google BigQuery

MySQL

airflow.providers.google.cloud.operators.bigquery_to_mysql

Google BigQuery

Google Cloud Storage (GCS)

airflow.providers.google.cloud.operators.bigquery_to_gcs

Google BigQuery

Google BigQuery

airflow.providers.google.cloud.operators.bigquery_to_bigquery

Cloud Firestore

Google Cloud Storage (GCS)

How to use

airflow.providers.google.firebase.operators.firestore

Google Cloud Storage (GCS)

Amazon Simple Storage Service (S3)

airflow.providers.amazon.aws.operators.gcs_to_s3

Google Cloud Storage (GCS)

Google BigQuery

airflow.providers.google.cloud.operators.gcs_to_bigquery

Google Cloud Storage (GCS)

Google Cloud Storage (GCS)

How to use, How to use

airflow.providers.google.cloud.operators.gcs_to_gcs, airflow.providers.google.cloud.operators.cloud_storage_transfer_service

Google Cloud Storage (GCS)

Google Drive

airflow.providers.google.suite.operators.gcs_to_gdrive

Google Cloud Storage (GCS)

SFTP

How to use

airflow.providers.google.cloud.operators.gcs_to_sftp

Local

Google Cloud Storage (GCS)

How to use

airflow.providers.google.cloud.operators.local_to_gcs

Microsoft SQL Server (MSSQL)

Google Cloud Storage (GCS)

airflow.providers.google.cloud.operators.mssql_to_gcs

MySQL

Google Cloud Storage (GCS)

airflow.providers.google.cloud.operators.mysql_to_gcs

PostgresSQL

Google Cloud Storage (GCS)

airflow.providers.google.cloud.operators.postgres_to_gcs

Presto

Google Cloud Storage (GCS)

How to use

airflow.providers.google.cloud.operators.presto_to_gcs

SFTP

Google Cloud Storage (GCS)

How to use

airflow.providers.google.cloud.operators.sftp_to_gcs

SQL

Google Cloud Storage (GCS)

airflow.providers.google.cloud.operators.sql_to_gcs

Google Spreadsheet

Google Cloud Storage (GCS)

How to use

airflow.providers.google.cloud.operators.sheets_to_gcs

Google Cloud Storage (GCS)

Google Spreadsheet

How to use

airflow.providers.google.suite.operators.gcs_to_sheets

[1] Those discovery-based operators use GoogleDiscoveryApiHook to communicate with Google Services via the Google API Python Client. Please note that this library is in maintenance mode hence it won’t fully support Google in the future. Therefore it is recommended that you use the custom Google Service Operators for working with the Google services.

Other operators and hooks

Guide

Operator

Hook

How to use

airflow.providers.google.cloud.operators.translate_speech

Yandex.Cloud

Airflow has a limited support for the Yandex.Cloud.

See the Yandex.Cloud connection type documentation to configure connections to Yandex.Cloud.

All hooks are based on airflow.providers.yandex.hooks.yandex.YandexCloudBaseHook.

Note

You can learn how to use Yandex.Cloud integrations by analyzing the example DAG

Service operators and hooks

These integrations allow you to perform various operations within the Yandex.Cloud.

Service name

Guide

Hook

Operator

Sensor

Base Classes

How to use

airflow.providers.yandex.hooks.yandex

Data Proc

How to use

airflow.providers.yandex.hooks.yandexcloud_dataproc

airflow.providers.yandex.operators.yandexcloud_dataproc

Service integrations

Service operators and hooks

These integrations allow you to perform various operations within various services.

Service name

Guide

Hook

Operator

Sensor

Atlassian Jira

airflow.providers.jira.hooks.jira

airflow.providers.jira.operators.jira

airflow.providers.jira.sensors.jira

Databricks

airflow.providers.databricks.hooks.databricks

airflow.providers.databricks.operators.databricks

Datadog

airflow.providers.datadog.hooks.datadog

airflow.providers.datadog.sensors.datadog

Pagerduty

airflow.providers.pagerduty.hooks.pagerduty

Dingding

How to use

airflow.providers.dingding.hooks.dingding

airflow.providers.dingding.operators.dingding

Discord

airflow.providers.discord.hooks.discord_webhook

airflow.providers.discord.operators.discord_webhook

Facebook Ads

airflow.providers.facebook.ads.hooks.ads

IBM Cloudant

airflow.providers.cloudant.hooks.cloudant

Jenkins

airflow.providers.jenkins.hooks.jenkins

airflow.providers.jenkins.operators.jenkins_job_trigger

Opsgenie

airflow.providers.opsgenie.hooks.opsgenie_alert

airflow.providers.opsgenie.operators.opsgenie_alert

Qubole

airflow.providers.qubole.hooks.qubole, airflow.providers.qubole.hooks.qubole_check

airflow.providers.qubole.operators.qubole, airflow.providers.qubole.operators.qubole_check

airflow.providers.qubole.sensors.qubole

Salesforce

airflow.providers.salesforce.hooks.salesforce, airflow.providers.salesforce.hooks.tableau

airflow.providers.salesforce.operators.tableau_refresh_workbook

airflow.providers.salesforce.sensors.tableau_job_status

Segment

airflow.providers.segment.hooks.segment

airflow.providers.segment.operators.segment_track_event

Slack

airflow.providers.slack.hooks.slack, airflow.providers.slack.hooks.slack_webhook

airflow.providers.slack.operators.slack, airflow.providers.slack.operators.slack_webhook

Snowflake

airflow.providers.snowflake.hooks.snowflake

airflow.providers.snowflake.operators.snowflake

Vertica

airflow.providers.vertica.hooks.vertica

airflow.providers.vertica.operators.vertica

Zendesk

airflow.providers.zendesk.hooks.zendesk

Software integrations

Software operators and hooks

These integrations allow you to perform various operations using various software.

Service name

Guide

Hook

Operator

Sensor

Celery

airflow.providers.celery.sensors.celery_queue

Docker

airflow.providers.docker.hooks.docker

airflow.providers.docker.operators.docker, airflow.providers.docker.operators.docker_swarm

Elasticsearch

airflow.providers.elasticsearch.hooks.elasticsearch

Exasol

airflow.providers.exasol.hooks.exasol

airflow.providers.exasol.operators.exasol

GNU Bash

How to use

airflow.operators.bash

airflow.sensors.bash

Kubernetes

How to use

airflow.providers.cncf.kubernetes.hooks.kubernetes

airflow.providers.cncf.kubernetes.operators.kubernetes_pod airflow.providers.cncf.kubernetes.operators.spark_kubernetes

airflow.providers.cncf.kubernetes.sensors.spark_kubernetes

Microsoft SQL Server (MSSQL)

airflow.providers.microsoft.mssql.hooks.mssql, airflow.providers.odbc.hooks.odbc

airflow.providers.microsoft.mssql.operators.mssql

ODBC

airflow.providers.odbc.hooks.odbc

MongoDB

airflow.providers.mongo.hooks.mongo

airflow.providers.mongo.sensors.mongo

MySQL

airflow.providers.mysql.hooks.mysql

airflow.providers.mysql.operators.mysql

OpenFaaS

airflow.providers.openfaas.hooks.openfaas

Oracle

airflow.providers.oracle.hooks.oracle

airflow.providers.oracle.operators.oracle

Papermill

How to use

airflow.providers.papermill.operators.papermill

PostgresSQL

airflow.providers.postgres.hooks.postgres

airflow.providers.postgres.operators.postgres

Presto

airflow.providers.presto.hooks.presto

Python

How to use

airflow.operators.python

airflow.sensors.python

Redis

airflow.providers.redis.hooks.redis

airflow.providers.redis.operators.redis_publish

airflow.providers.redis.sensors.redis_pub_sub, airflow.providers.redis.sensors.redis_key

Samba

airflow.providers.samba.hooks.samba

Singularity

airflow.providers.singularity.operators.singularity

SQLite

airflow.providers.sqlite.hooks.sqlite

airflow.providers.sqlite.operators.sqlite

Protocol integrations