Since you use the task decorator on task1(), what PythonVirtualenvOperator gets instead is an Airflow operator (and not the function task1()). The following DAG will work as expected: from corators import dagįrom airflow.operators. PythonVirtualenvOperator expects a function to be executed as an argument to its pythoncallable parameter. So, you have to do all necessary imports inside the function. You need to remove that task decorator.Īlso, task1() will be "cut out" from the DAG and executed in a virtual environment on its own. PythonVirtualenvOperator expects a function to be executed as an argument to its python_callable parameter. Question: How do I properly utilize the PythonVirtualenvOperator in DAGs built on the Airflow 2.x TaskFlow API? The decorator allows you to create dynamically a new virtualenv with custom libraries and even a different Python version to run your function. It seems that, even though I'm installing the apache-airflow package into the virtual environment, it's not finding the TaskFlow API types. The simplest approach is to create dynamically (every time a task is run) a separate virtual environment on the same machine, you can use the task.virtualenv decorator. T_task1 = PythonVirtualenvOperator(python_callable=task1, system_site_packages=False, requirements=, task_id='trevor')ĭAG executes successfully in a Python virtual environment. This set of kwargs correspond exactly to what you can use in your jinja templates. 'owner': schedule_interval=None, start_date=days_ago(2), tags = ) See also For more information on how to use this operator, take a look at the guide: PythonOperator When running your callable, Airflow will pass a set of keyword arguments that can be used in your function. ![]() Here's what my sample DAG looks like: from time import sleepįrom import PythonVirtualenvOperator ![]() I'm trying to figure out how to use the PythonVirtualenvOperator inside of a DAG that I'm creating, using the TaskFlow API in Apache Airflow 2.0.1.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |