Multithreading for Python Django

2024/11/19 11:44:20

Some functions should run asynchronously on the web server. Sending emails or data post-processing are typical use cases.

What is the best (or most pythonic) way write a decorator function to run a function asynchronously?

My setup is a common one: Python, Django, Gunicorn or Waitress, AWS EC2 standard Linux

For example, here's a start:

from threading import Threaddef postpone(function):def decorator(*args, **kwargs):t = Thread(target = function, args=args, kwargs=kwargs)t.daemon = Truet.start()return decorator

desired usage:

@postpone
def foo():pass #do stuff
Answer

I've continued using this implementation at scale and in production with no issues.

Decorator definition:

def start_new_thread(function):def decorator(*args, **kwargs):t = Thread(target = function, args=args, kwargs=kwargs)t.daemon = Truet.start()return decorator

Example usage:

@start_new_thread
def foo():#do stuff

Over time, the stack has updated and transitioned without fail.

Originally Python 2.4.7, Django 1.4, Gunicorn 0.17.2, now Python 3.6, Django 2.1, Waitress 1.1.

If you are using any database transactions, Django will create a new connection and this needs to be manually closed:

from django.db import connection@postpone
def foo():#do stuffconnection.close()
https://en.xdnf.cn/q/26447.html

Related Q&A

run a crontab job using an anaconda env

I want to have a cron job execute a python script using an already existing anaconda python environment called my_env. The only thing I can think to do is have the cron job run a script called my_scri…

pandas - Merging on string columns not working (bug?)

Im trying to do a simple merge between two dataframes. These come from two different SQL tables, where the joining keys are strings:>>> df1.col1.dtype dtype(O) >>> df2.col2.dtype dtyp…

Making a chart bigger in size

Im trying to get a bigger chart. However, the figure method from matplotlib does not seem to be working properly. I get a message, which is not an error: <matplotlib.figure.Figure at 0xa25f7f0>My…

index of non NaN values in Pandas

From Pandas data frame, how to get index of non "NaN" values?My data frame isA b c 0 1 q1 1 1 2 NaN 3 2 3 q2 3 3 4 q1 NaN 4 5 q2 7And I want the…

How can I type-check variables in Python? [duplicate]

This question already has answers here:Whats the canonical way to check for type in Python?(16 answers)Closed 3 months ago.I have a Python function that takes a numeric argument that must be an intege…

Py_INCREF/DECREF: When

Is one correct in stating the following:If a Python object is created in a C function, but the function doesnt return it, no INCREF is needed, but a DECREF is. [false]If the function does return it, yo…

pop/remove items out of a python tuple

I am not sure if I can make myself clear but will try.I have a tuple in python which I go through as follows (see code below). While going through it, I maintain a counter (lets call it n) and pop item…

Difference between frompyfunc and vectorize in numpy

What is the difference between vectorize and frompyfunc in numpy?Both seem very similar. What is a typical use case for each of them?Edit: As JoshAdel indicates, the class vectorize seems to be built…

Jupyter notebook command does not work on Mac

I installed jupyter using pip on my macbook air. Upon trying to execute the command jupyter notebook, I get an error jupyter: notebook is not a Jupyter commandI used the --h option to get a listing of …

Recursively compare two directories to ensure they have the same files and subdirectories

From what I observe filecmp.dircmp is recursive, but inadequate for my needs, at least in py2. I want to compare two directories and all their contained files. Does this exist, or do I need to build …