Parallel processing from a command queue on Linux (bash, python, ruby... whatever)

2024/11/19 9:33:30

I have a list/queue of 200 commands that I need to run in a shell on a Linux server.

I only want to have a maximum of 10 processes running (from the queue) at once. Some processes will take a few seconds to complete, other processes will take much longer.

When a process finishes I want the next command to be "popped" from the queue and executed.

Does anyone have code to solve this problem?

Further elaboration:

There's 200 pieces of work that need to be done, in a queue of some sort. I want to have at most 10 pieces of work going on at once. When a thread finishes a piece of work it should ask the queue for the next piece of work. If there's no more work in the queue, the thread should die. When all the threads have died it means all the work has been done.

The actual problem I'm trying to solve is using imapsync to synchronize 200 mailboxes from an old mail server to a new mail server. Some users have large mailboxes and take a long time tto sync, others have very small mailboxes and sync quickly.

Answer

On the shell, xargs can be used to queue parallel command processing. For example, for having always 3 sleeps in parallel, sleeping for 1 second each, and executing 10 sleeps in total do

echo {1..10} | xargs -d ' ' -n1 -P3 sh -c 'sleep 1s' _

And it would sleep for 4 seconds in total. If you have a list of names, and want to pass the names to commands executed, again executing 3 commands in parallel, do

cat names | xargs -n1 -P3 process_name

Would execute the command process_name alice, process_name bob and so on.

https://en.xdnf.cn/q/26455.html

Related Q&A

How do I select and store columns greater than a number in pandas? [duplicate]

This question already has answers here:How do I select rows from a DataFrame based on column values?(17 answers)Closed 28 days ago.I have a pandas DataFrame with a column of integers. I want the rows …

Plotting transparent histogram with non transparent edge

I am plotting a histogram, and I have three datasets which I want to plot together, each one with different colours and linetype (dashed, dotted, etc). I am also giving some transparency, in order to s…

Cross-platform desktop notifier in Python

I am looking for Growl-like, Windows balloon-tip-like notifications library in Python. Imagine writing code like:>>> import desktopnotifier as dn >>> dn.notify(Title, Long description…

set object is not JSON serializable [duplicate]

This question already has answers here:How to JSON serialize sets? [duplicate](12 answers)Closed 9 years ago.When I try to run the following code:import jsond = {testing: {1, 2, 3}} json_string = json…

Python Sqlite3: INSERT INTO table VALUE(dictionary goes here)

I would like to use a dictionary to insert values into a table, how would I do this? import sqlite3db = sqlite3.connect(local.db) cur = db.cursor()cur.execute(DROP TABLE IF EXISTS Media)cur.execute(CR…

Count the uppercase letters in a string with Python

I am trying to figure out how I can count the uppercase letters in a string. I have only been able to count lowercase letters:def n_lower_chars(string):return sum(map(str.islower, string))Example of w…

Multithreading for Python Django

Some functions should run asynchronously on the web server. Sending emails or data post-processing are typical use cases.What is the best (or most pythonic) way write a decorator function to run a func…

run a crontab job using an anaconda env

I want to have a cron job execute a python script using an already existing anaconda python environment called my_env. The only thing I can think to do is have the cron job run a script called my_scri…

pandas - Merging on string columns not working (bug?)

Im trying to do a simple merge between two dataframes. These come from two different SQL tables, where the joining keys are strings:>>> df1.col1.dtype dtype(O) >>> df2.col2.dtype dtyp…

Making a chart bigger in size

Im trying to get a bigger chart. However, the figure method from matplotlib does not seem to be working properly. I get a message, which is not an error: <matplotlib.figure.Figure at 0xa25f7f0>My…