Debussy Airflow

A set of Airflow Hooks, Operators and Sensors for Debussy Concert,

the core component of Debussy Framework

⚠️Not exactly a provider

This package provides operators, sensors, and hooks that enables running Debussy Concert on Apache Airflow. However, it's not a traditional Airflow provider package in the sense Debussy Concert is not a tool, but rather a framework for data pipeline development and code generation. Therefore, our hooks, operators and sensors acts as interfaces, extending other Airflow providers, and abstracting their usage.

Installation

You can install this package via

pip install debussy-airflow

Prerequisites

An environment running apache-airflow 2.2.0+ with the following requirements (already managed as dependencies):

- apache-airflow-providers-google == 8.0.0
- facebook-business == 13.0.0
- mysql-connector-python == 8.0.24
- paramiko == 2.8.1

Modules

Hooks

• db_api_hook
• facebook_ads_hook
• http_api_hook
• http_hook
• storage_hook

Operators

• basic_operator
• db_to_storage_operator
• facebook_ads_to_storage_operator
• http_operator
• rest_api_to_storage_operator
• storage_to_rdbms_operator
• storage_to_storage_operator

Sensors

• http_sensor

Examples

See the examples directory for an example DAG.

Issues

Please submit issues and pull requests in our official repo: https://github.com/DotzInc/debussy_airflow

We are happy to hear from you. Please email any feedback to the authors at [email protected].

Project Contributors and Maintainers

This project is built with active contributions from:

• Lawrence Fernandes
• Nilton Duarte
• Maycon Francis

This project is formatted via black:

pip install black
black .