How to add template variable in the filename of an EmailOperator task? (Airflow)

2024/10/11 2:20:43

I can't seem to get this to work.

I am trying to send daily a given file, whose name is like 'file_{{ds_nodash}}.csv'.

The problem is that I can't seem to add this name as the filename, since it seems it cant be used. In the text of the email or the subject works perfectly, not not on the name.

Here is the dag as an example:

local_file = 'file-{{ds_nodash}}.csv'send_stats_csv = EmailOperator(task_id='send-stats-csv',to=['[email protected]'],subject='Subject - {{ ds }}',html_content='Here is the new file.',files=[local_file],dag=dag)

Error code: No such file or directory: u'file-{{ds_nodash}}.csv'

If i write it literally, with its given date, it works flawlessly.

Where am I wrong? How should I go about this?

Any help would be appreciated.

Thanks.

P.D. Copy paste from airflow's documentation - "The Airflow engine passes a few variables by default that are accessible in all templates". https://airflow.incubator.apache.org/code.html

If I understood correctly, these variables are accessible in execution, so if i am executing the dag, the file should be found right? I've tried both testing the task or backfilling the dag with no success.

Answer

Airflow Operators define what fields are template fields. For the EmailOperator only the subject and html_content fields are set as templates.

class EmailOperator(BaseOperator):...template_fields = ('subject', 'html_content')template_ext = ('.html',)

See: https://airflow.incubator.apache.org/_modules/email_operator.html

From the Airflow Gotcha's Page (https://gtoonstra.github.io/etl-with-airflow/gotchas.html)

Not all parameters in operators are templated, so you cannot use Jinja templates everywhere. The Jinja templates only work for those fields in operators where it’s listed in the template_fields...

To get this to work, you would have to derive a new class from EmailOperator and add in templating for the files array.

https://en.xdnf.cn/q/69820.html

Related Q&A

Disadvantage of Python eggs?

Are there any disadvantages about using eggs through easy-install compared to the "traditional" packages/modules/libs?

Does Google App Engine Flex support Pipfile?

For App Engine Standard the explicitly state that they do not support Pipfiles and immediately block you from pushing your project if it contains a Pipfile. In searching the documentation, I dont see a…

Keras sees my GPU but doesnt use it when training a neural network

My GPU is not used by Keras/TensorFlow.To try to make my GPU working with tensorflow, I installed tensorflow-gpu via pip (I am using Anaconda on Windows)I have nvidia 1080tiprint(tf.test.is_gpu_availab…

Identify if there are two of the same character adjacent to eachother

Ive been asked to create a program that identifies if a password is valid or not. The one part I am struggling with is identifying whether there are two of the same character adjacent to each other. He…

Extract lined table from scanned document opencv python

I want to extract the information from a scanned table and store it a csv. Right now my table extraction algorithm does the following steps.Apply skew correction Apply a gaussian filter for denoising. …

Nested Python C Extensions/Modules?

How do I compile a C-Python module such that it is local to another? E.g. if I have a module named "bar" and another module named "mymodule", how do I compile "bar" so th…

ImportError: No module named sysconfig--cant get pip working

Im really struggling with pip on a RedHat 6.9 system. Every time I tried to use pip, I got ImportError: No module named sysconfigI tried Googling for solutions. I dont have apt-get and cant seem to get…

Convert Dataframe to a Dictionary with List Values

Suppose I have a Dataframe df :Label1 Label2 Label3 key1 col1value1 col2value1 key2 col1value2 col2value2 key3 col1value3 col2value3dict1 = df.set_index(Label1).to_dic…

Efficiently count all the combinations of numbers having a sum close to 0

I have following pandas dataframe df column1 column2 list_numbers sublist_column x y [10,-6,1,-4] a b [1,3,7,-2] p q [6,2,-3,-3.…

What is the equivalent to iloc for dask dataframe?

I have a situation where I need to index a dask dataframe by location. I see that there is not an .iloc method available. Is there an alternative? Or am I required to use label-based indexing?For …