Read Celery configuration from Python properties file

2024/10/10 4:23:51

I have an application that needs to initialize Celery and other things (e.g. database). I would like to have a .ini file that would contain the applications configuration. This should be passed to the application at runtime.

development.init:

[celery]
broker=amqp://localhost/
backend=amqp://localhost/
task.result.expires=3600[database]
# database config
# ...

celeryconfig.py:

from celery import Celery
import ConfigParserconfig = ConfigParser.RawConfigParser()
config.read(...) # Pass this from the command line somehowcelery = Celery('myproject.celery',broker=config.get('celery', 'broker'),backend=config.get('celery', 'backend'),include=['myproject.tasks'])# Optional configuration, see the application user guide.
celery.conf.update(CELERY_TASK_RESULT_EXPIRES=config.getint('celery', 'task.result.expires')
)# Initialize database, etc.if __name__ == '__main__':celery.start()

To start Celery, I call:

celery worker --app=myproject.celeryconfig -l info

Is there anyway to pass in the config file without doing something ugly like setting a environment variable?

Answer

Alright, I took Jordan's advice and used a env variable. This is what I get in celeryconfig.py:

celery import Celery
import os
import sys
import ConfigParserCELERY_CONFIG = 'CELERY_CONFIG'if not CELERY_CONFIG in os.environ:sys.stderr.write('Missing env variable "%s"\n\n' % CELERY_CONFIG)sys.exit(2)configfile = os.environ['CELERY_CONFIG']if not os.path.isfile(configfile):sys.stderr.write('Can\'t read file: "%s"\n\n' % configfile)sys.exit(2)config = ConfigParser.RawConfigParser()
config.read(configfile)celery = Celery('myproject.celery',broker=config.get('celery', 'broker'),backend=config.get('celery', 'backend'),include=['myproject.tasks'])# Optional configuration, see the application user guide.
celery.conf.update(CELERY_TASK_RESULT_EXPIRES=config.getint('celery', 'task.result.expires'),
)if __name__ == '__main__':celery.start()

To start Celery:

$ export CELERY_CONFIG=development.ini
$ celery worker --app=myproject.celeryconfig -l info
https://en.xdnf.cn/q/69935.html

Related Q&A

numpys tostring/fromstring --- what do I need to specify to restore the array

Given a raw binary representation of a numpy array, what is the complete set of metadata needed to unambiguously restore the array? For example, >>> np.fromstring( np.array([42]).tostring())…

How to limit width of column headers in Pandas

How can I limit the column width within Pandas when displaying dataframes, etc? I know about display.max_colwidth but it doesnt affect column names. Also, I do not want to break the names up, but rath…

Django + Auth0 JWT authentication refusing to decode

I am trying to implement Auth0 JWT-based authentication in my Django REST API using the django-rest-framework. I know that there is a JWT library available for the REST framework, and I have tried usin…

How to measure the angle between 2 lines in a same image using python opencv?

I have detected a lane boundary line which is not straight using hough transform and then extracted that line separately. Then blended with another image that has a straight line. Now I need to calcula…

How to modify variables in another python file?

windows 10 - python 3.5.2Hi, I have the two following python files, and I want to edit the second files variables using the code in the first python file.firstfile.pyfrom X.secondfile import *def edit(…

SciPy optimizer ignores one of the constraints

I am trying to solve an optimization problem where I need to create a portfolio that with a minimum tracking error from benchmark portfolio and its subject to some constraints:import scipy.optimize as …

How can one use HashiCorp Vault in Airflow?

I am starting to use Apache Airflow and I am wondering how to effectively make it use secrets and passwords stored in Vault. Unfortunately, search does not return meaningful answers beyond a yet-to-be-…

List all words in a dictionary that start with user input

How would a go about making a program where the user enters a string, and the program generates a list of words beginning with that string?Ex: User: "abd" Program:abdicate, abdomen, abduct..…

Python version of C#s conditional operator (?)

I saw this question but it uses the ?? operator as a null check, I want to use it as a bool true/false test.I have this code in Python:if self.trait == self.spouse.trait:trait = self.trait else:trait…

Python String Replace Error

I have a python script that keeps returning the following error:TypeError: replace() takes at least 2 arguments (1 given)I cannot for the life of me figure out what is causing this.Here is part of my c…