How to hide globals in an Airflow DAG definition file?

airflow

Airflow has a fairly strange way of registering DAGs and tasks. They’re put into the global namespace of the DAG definition file.

dag = DAG(dag_id='foo', start_date=start_date)
MyOperator(dag=dag, task_id='foo')

Airflow then comes along and finds them.

When importing that file however, as you do when unit testing, it’s not ideal to have those global objects created.

The solution is to protect that code with an if statement:

if __name__.startswith('unusual_prefix'):
    dag = DAG(dag_id='foo', start_date=start_date)
    MyOperator(dag=dag, task_id='foo')

This is Airflow’s equivalent of if __name__ == "__main__".

Airflow will still find your DAG as normal, however that code inside the block won’t be executed when the module is imported.