Operators and Hooks Reference

ASF: Apache Software Foundation

Airflow supports various software created by Apache Software Foundation.

AWS: Amazon Web Services

Airflow has support for Amazon Web Services.

All hooks are based on airflow.contrib.hooks.aws_hook.

Service operators and hooks

These integrations allow you to perform various operations within the Amazon Web Services.

Service name

Hook

Operators

Sensors

AWS Batch

airflow.contrib.operators.awsbatch_operator

AWS DataSync

airflow.providers.amazon.aws.hooks.datasync

airflow.providers.amazon.aws.operators.datasync

AWS Glue Catalog

airflow.contrib.hooks.aws_glue_catalog_hook

airflow.contrib.sensors.aws_glue_catalog_partition_sensor

AWS Lambda

airflow.providers.amazon.aws.hooks.lambda_function

Amazon Athena

airflow.providers.amazon.aws.hooks.athena

airflow.providers.amazon.aws.operators.athena

airflow.providers.amazon.aws.sensors.athena

Amazon CloudWatch Logs

airflow.contrib.hooks.aws_logs_hook

Amazon DynamoDB

airflow.contrib.hooks.aws_dynamodb_hook

Amazon EC2

airflow.contrib.operators.ecs_operator

Amazon EMR

airflow.contrib.hooks.emr_hook

airflow.contrib.operators.emr_add_steps_operator, airflow.contrib.operators.emr_create_job_flow_operator, airflow.contrib.operators.emr_terminate_job_flow_operator

airflow.contrib.sensors.emr_base_sensor, airflow.contrib.sensors.emr_job_flow_sensor, airflow.contrib.sensors.emr_step_sensor

Amazon Kinesis Data Firehose

airflow.providers.amazon.aws.hooks.kinesis

Amazon Redshift

airflow.providers.amazon.aws.hooks.redshift

airflow.providers.amazon.aws.sensors.redshift

Amazon SageMaker

airflow.contrib.hooks.sagemaker_hook

airflow.contrib.operators.sagemaker_base_operator, airflow.contrib.operators.sagemaker_endpoint_config_operator, airflow.contrib.operators.sagemaker_endpoint_operator, airflow.contrib.operators.sagemaker_model_operator, airflow.contrib.operators.sagemaker_training_operator, airflow.contrib.operators.sagemaker_transform_operator, airflow.contrib.operators.sagemaker_tuning_operator

airflow.contrib.sensors.sagemaker_base_sensor, airflow.contrib.sensors.sagemaker_endpoint_sensor, airflow.contrib.sensors.sagemaker_training_sensor, airflow.contrib.sensors.sagemaker_transform_sensor, airflow.contrib.sensors.sagemaker_tuning_sensor

Amazon Simple Notification Service (SNS)

airflow.providers.amazon.aws.hooks.sns

airflow.providers.amazon.aws.operators.sns

Amazon Simple Queue Service (SQS)

airflow.providers.amazon.aws.hooks.sqs

airflow.providers.amazon.aws.operators.sqs

airflow.providers.amazon.aws.sensors.sqs

Amazon Simple Storage Service (S3)

airflow.providers.amazon.aws.hooks.s3

airflow.operators.s3_file_transform_operator, airflow.contrib.operators.s3_copy_object_operator, airflow.contrib.operators.s3_delete_objects_operator, airflow.contrib.operators.s3_list_operator

airflow.sensors.s3_key_sensor, airflow.sensors.s3_prefix_sensor

Transfer operators and hooks

These integrations allow you to copy data from/to Amazon Web Services.

Source

Destination

Guide

Operators

All GCP services [1]

Amazon Simple Storage Service (S3)

airflow.operators.google_api_to_s3_transfer

Amazon DataSync

Amazon Simple Storage Service (S3)

airflow.providers.amazon.aws.operators.datasync

Amazon DynamoDB

Amazon Simple Storage Service (S3)

airflow.contrib.operators.dynamodb_to_s3

Amazon Redshift

Amazon Simple Storage Service (S3)

airflow.operators.redshift_to_s3_operator

Amazon Simple Storage Service (S3)

Amazon Redshift

airflow.operators.s3_to_redshift_operator

Amazon Simple Storage Service (S3)

Apache Hive

airflow.operators.s3_to_hive_operator

Amazon Simple Storage Service (S3)

Google Cloud Storage (GCS)

How to use

airflow.contrib.operators.s3_to_gcs_operator, airflow.gcp.operators.cloud_storage_transfer_service

Amazon Simple Storage Service (S3)

SSH File Transfer Protocol (SFTP)

airflow.contrib.operators.s3_to_sftp_operator

Apache Hive

Amazon DynamoDB

airflow.contrib.operators.hive_to_dynamodb

Google Cloud Storage (GCS)

Amazon Simple Storage Service (S3)

airflow.operators.gcs_to_s3

Internet Message Access Protocol (IMAP)

Amazon Simple Storage Service (S3)

airflow.contrib.operators.imap_attachment_to_s3_operator

MongoDB

Amazon Simple Storage Service (S3)

airflow.contrib.operators.mongo_to_s3

SSH File Transfer Protocol (SFTP)

Amazon Simple Storage Service (S3)

airflow.contrib.operators.sftp_to_s3_operator

[1] Those discovery-based operators use airflow.gcp.hooks.discovery_api.GoogleDiscoveryApiHook to communicate with Google Services via the Google API Python Client. Please note that this library is in maintenance mode hence it won’t fully support GCP in the future. Therefore it is recommended that you use the custom GCP Service Operators for working with the Google Cloud Platform.

GCP: Google Cloud Platform

Airflow has extensive support for the Google Cloud Platform.

See the GCP connection type documentation to configure connections to GCP.

All hooks are based on airflow.gcp.hooks.base.GoogleCloudBaseHook.

Note

You can learn how to use GCP integrations by analyzing the source code of the particular example DAGs.

Service operators and hooks

These integrations allow you to perform various operations within the Google Cloud Platform.

Service name

Guide

Hook

Operators

Sensors

AutoML

How to use

airflow.gcp.hooks.automl

airflow.gcp.operators.automl

BigQuery

airflow.gcp.hooks.bigquery

airflow.gcp.operators.bigquery

airflow.gcp.sensors.bigquery

BigQuery Data Transfer Service

How to use

airflow.gcp.hooks.bigquery_dts

airflow.gcp.operators.bigquery_dts

airflow.gcp.sensors.bigquery_dts

Bigtable

How to use

airflow.gcp.hooks.bigtable

airflow.gcp.operators.bigtable

airflow.gcp.sensors.bigtable

Cloud Build

How to use

airflow.gcp.hooks.cloud_build

airflow.gcp.operators.cloud_build

Compute Engine

How to use

airflow.gcp.hooks.compute

airflow.gcp.operators.compute

Cloud Data Loss Prevention (DLP)

airflow.gcp.hooks.dlp

airflow.gcp.operators.dlp

Dataflow

airflow.gcp.hooks.dataflow

airflow.gcp.operators.dataflow

Dataproc

airflow.gcp.hooks.dataproc

airflow.gcp.operators.dataproc

Datastore

airflow.gcp.hooks.datastore

airflow.gcp.operators.datastore

Cloud Functions

How to use

airflow.gcp.hooks.functions

airflow.gcp.operators.functions

Cloud Key Management Service (KMS)

airflow.gcp.hooks.kms

Kubernetes Engine

airflow.gcp.hooks.kubernetes_engine

airflow.gcp.operators.kubernetes_engine

Machine Learning Engine

airflow.gcp.hooks.mlengine

airflow.gcp.operators.mlengine

Cloud Memorystore

How to use

airflow.gcp.hooks.cloud_memorystore

airflow.gcp.operators.cloud_memorystore

Natural Language

How to use

airflow.providers.google.cloud.hooks.natural_language

airflow.providers.google.cloud.operators.natural_language

Cloud Pub/Sub

How to use

airflow.providers.google.cloud..hooks.pubsub

airflow.providers.google.cloud..operators.pubsub

airflow.providers.google.cloud..sensors.pubsub

Cloud Spanner

How to use

airflow.gcp.hooks.spanner

airflow.gcp.operators.spanner

Cloud Speech-to-Text

How to use

airflow.gcp.hooks.speech_to_text

airflow.gcp.operators.speech_to_text

Cloud SQL

How to use

airflow.gcp.hooks.cloud_sql

airflow.gcp.operators.cloud_sql

Cloud Storage (GCS)

How to use

airflow.gcp.hooks.gcs

airflow.gcp.operators.gcs

airflow.gcp.sensors.gcs

Storage Transfer Service

How to use

airflow.gcp.hooks.cloud_storage_transfer_service

airflow.gcp.operators.cloud_storage_transfer_service

airflow.gcp.sensors.cloud_storage_transfer_service

Cloud Tasks

airflow.gcp.hooks.tasks

airflow.gcp.operators.tasks

Cloud Text-to-Speech

How to use

airflow.gcp.hooks.text_to_speech

airflow.gcp.operators.text_to_speech

Cloud Translation

How to use

airflow.gcp.hooks.translate

airflow.gcp.operators.translate

Cloud Video Intelligence

How to use

airflow.gcp.hooks.video_intelligence

airflow.gcp.operators.video_intelligence

Cloud Vision

How to use

airflow.providers.google.cloud.hooks.vision

airflow.providers.google.cloud.operators.vision

Transfer operators and hooks

These integrations allow you to copy data from/to Google Cloud Platform.

Source

Destination

Guide

Operators

All services [1]

Amazon Simple Storage Service (S3)

airflow.operators.google_api_to_s3_transfer

Amazon Simple Storage Service (S3)

Google Cloud Storage (GCS)

How to use

airflow.contrib.operators.s3_to_gcs_operator, airflow.gcp.operators.cloud_storage_transfer_service

Apache Cassandra

Google Cloud Storage (GCS)

airflow.operators.cassandra_to_gcs

Azure Data Lake Storage

Google Cloud Storage (GCS)

airflow.operators.adls_to_gcs

Google BigQuery

MySQL

airflow.operators.bigquery_to_mysql

Google BigQuery

Cloud Storage (GCS)

airflow.operators.bigquery_to_gcs

Google BigQuery

Google BigQuery

airflow.operators.bigquery_to_bigquery

Google Cloud Storage (GCS)

Amazon Simple Storage Service (S3)

airflow.operators.gcs_to_s3

Google Cloud Storage (GCS)

Google BigQuery

airflow.operators.gcs_to_bq

Google Cloud Storage (GCS)

Google Cloud Storage (GCS)

How to use, How to use

airflow.operators.gcs_to_gcs, airflow.gcp.operators.cloud_storage_transfer_service

Google Cloud Storage (GCS)

Google Drive

airflow.contrib.operators.gcs_to_gdrive_operator

Google Cloud Storage (GCS)

SFTP

How to use

airflow.operators.gcs_to_sftp

Local

Google Cloud Storage (GCS)

airflow.operators.local_to_gcs

Microsoft SQL Server (MSSQL)

Google Cloud Storage (GCS)

airflow.operators.mssql_to_gcs

MySQL

Google Cloud Storage (GCS)

airflow.operators.mysql_to_gcs

PostgresSQL

Google Cloud Storage (GCS)

airflow.operators.postgres_to_gcs

SFTP

Google Cloud Storage (GCS)

How to use

airflow.providers.google.cloud.operators.sftp_to_gcs

SQL

Cloud Storage (GCS)

airflow.operators.sql_to_gcs

[1] Those discovery-based operators use airflow.gcp.hooks.discovery_api.GoogleDiscoveryApiHook to communicate with Google Services via the Google API Python Client. Please note that this library is in maintenance mode hence it won’t fully support GCP in the future. Therefore it is recommended that you use the custom GCP Service Operators for working with the Google Cloud Platform.

Service integrations

Service operators and hooks

These integrations allow you to perform various operations within various services.

Service name

Guide

Hook

Operators

Sensors

Atlassian Jira

airflow.contrib.hooks.jira_hook

airflow.contrib.operators.jira_operator

airflow.contrib.sensors.jira_sensor

Databricks

airflow.contrib.hooks.databricks_hook

airflow.contrib.operators.databricks_operator

Datadog

airflow.contrib.hooks.datadog_hook

airflow.contrib.sensors.datadog_sensor

Pagerduty

airflow.contrib.hooks.pagerduty_hook

Dingding

How to use

airflow.contrib.hooks.dingding_hook

airflow.contrib.operators.dingding_operator

Discord

airflow.contrib.hooks.discord_webhook_hook

airflow.contrib.operators.discord_webhook_operator

Google Campaign Manager

How to use

airflow.providers.google.marketing_platform.hooks.campaign_manager

airflow.providers.google.marketing_platform.operators.campaign_manager

airflow.providers.google.marketing_platform.sensors.campaign_manager

Google Display&Video 360

How to use

airflow.providers.google.marketing_platform.hooks.display_video

airflow.providers.google.marketing_platform.operators.display_video

airflow.providers.google.marketing_platform.sensors.display_video

Google Drive

airflow.contrib.hooks.gdrive_hook

Google Search Ads 360

How to use

airflow.providers.google.marketing_platform.hooks.search_ads

airflow.providers.google.marketing_platform.operators.search_ads

airflow.providers.google.marketing_platform.sensors.search_ads

Google Spreadsheet

airflow.gcp.hooks.gsheets

IBM Cloudant

airflow.contrib.hooks.cloudant_hook

Jenkins

airflow.contrib.hooks.jenkins_hook

airflow.contrib.operators.jenkins_job_trigger_operator

Opsgenie

airflow.contrib.hooks.opsgenie_alert_hook

airflow.contrib.operators.opsgenie_alert_operator

Qubole

airflow.contrib.hooks.qubole_hook, airflow.contrib.hooks.qubole_check_hook

airflow.contrib.operators.qubole_operator, airflow.contrib.operators.qubole_check_operator

airflow.contrib.sensors.qubole_sensor

Salesforce

airflow.contrib.hooks.salesforce_hook

Segment

airflow.contrib.hooks.segment_hook

airflow.contrib.operators.segment_track_event_operator

Slack

airflow.hooks.slack_hook, airflow.contrib.hooks.slack_webhook_hook

airflow.operators.slack_operator, airflow.contrib.operators.slack_webhook_operator

Snowflake

airflow.contrib.hooks.snowflake_hook

airflow.contrib.operators.snowflake_operator

Vertica

airflow.contrib.hooks.vertica_hook

airflow.contrib.operators.vertica_operator

Zendesk

airflow.hooks.zendesk_hook

Transfer operators and hooks

These integrations allow you to perform various operations within various services.

Source

Destination

Guide

Operators

Google Cloud Storage (GCS)

Google Drive

airflow.contrib.operators.gcs_to_gdrive_operator

Vertica

Apache Hive

airflow.contrib.operators.vertica_to_hive

Vertica

MySQL

airflow.contrib.operators.vertica_to_mysql

Software integrations

Protocol integrations