Apache Airflow - customize logging format

2024/10/12 8:16:17

Is it possible to customize the format that Airflow uses for logging?

I tried adding a LOG_FORMAT variable in $AIRFLOW_HOME/airflow.cfg, but it doesn't seem to take effect

LOG_FORMAT = "%(asctime)s logLevel=%(levelname)s logger=%(name)s - %(message)s"
Answer

You need to change the settings.py file in the airflow package to change the log format

  • Update settings.py (after LOGGING_LEVEL add below line):

    LOG_FORMAT = os.path.expanduser(conf.get('core', 'LOG_FORMAT'))

  • Update airflow.cfg configuration file: Add line under [core]:

    LOG_FORMAT = "%(asctime)s logLevel=%(levelname)s logger=%(name)s - %(message)s"

  • Restart webserver and scheduler services

    Try the approach as mentioned in the answer here to change logging level

https://en.xdnf.cn/q/69672.html

Related Q&A

How can I set up Celery to call a custom worker initialization?

I am quite new to Celery and I have been trying to setup a project with 2 separate queues (one to calculate and the other to execute). So far, so good. My problem is that the workers in the execute que…

Why does print(__name__) give builtins?

Im using pycharm.2017.1.2. I installed anaconda2 with py3 environment. in Pycharm, Im using Python3 interpreter, and the code is simply:print(__name__)In Python console in Pycharm, it prints builtins.I…

List Comprehensions and Conditions?

I am trying to see if I can make this code better using list comprehensions. Lets say that I have the following lists:a_list = [HELLO,FOO,FO1BAR,ROOBAR,SHOEBAR]regex_list = [lambda x: re.search(rFOO,…

Python in operator time complexity on range()

I have the following function:def foo(length, num):return num in range(length)Whats the time complexity of this function? Noting that range() creates a Range object on Python 3, will the time complexi…

Pandas read data from a secure FTP server in Python 3

I am looking for a neat solution to read data (using either read_csv or read_sas) to a Pandas Dataframe from a secure FTP server in Python 3. All the examples I can find are many lines and some for Pyt…

How to read XML header in Python

How can I read the header of an XML document in Python 3?Ideally, I would use the defusedxml module as the documentation states that its safer, but at this point (after hours of trying to figure this …

Shift interpolation does not give expected behaviour

When using scipy.ndimage.interpolation.shift to shift a numpy data array along one axis with periodic boundary treatment (mode = wrap), I get an unexpected behavior. The routine tries to force the firs…

HEX decoding in Python 3.2

In Python 2.x Im able to do this:>>> 4f6c6567.decode(hex_codec) OlegBut in Python 3.2 I encounter this error:>>> b4f6c6567.decode(hex_codec) Traceback (most recent call last):File &qu…

How do I access session data in Jinja2 templates (Bottle framework on app engine)?

Im running the micro framework Bottle on Google App Engine. Im using Jinja2 for my templates. And Im using Beaker to handle the sessions. Im still a pretty big Python newbie and am pretty stoked I g…

What is a dimensional range of [-1,0] in Pytorch?

So Im struggling to understand some terminology about collections in Pytorch. I keep running into the same kinds of errors about the range of my tensors being incorrect, and when I try to Google for a …