Run several python programs at the same time

2024/7/7 7:18:28

I have python script run.py:

def do(i):# doing something with i, that takes timestart_i = sys.argv[1]
end_i = sys.argv[2]
for i in range(start_i, end_i):do(i)

Then I run this script:

python run.py 0 1000000

After 30 minutes script is completed. But, it's too long for me.

So, I create bash script run.sh:

python run.py 0 200000 &
python run.py 200000 400000 &
python run.py 400000 600000 &
python run.py 600000 800000 &
python run.py 800000 1000000

Then I run this script:

bash run.sh

After 6 minutes script is completed. Rather good. I'm happy.

But I think, there is another way to solve the problem (without creating bash script), isn't there?

Answer

You're looking for the multiprocessing package, and especially the Pool class:

from multiprocessing import Pool
p = Pool(5)  # like in your example, running five separate processes
p.map(do, range(start_i, end_i))

Besides consolidating this into a single command, this has other advantages over your approach of calling python run.py 0 200000 & etc. If some processes take longer than others (and therefore, python run.py 0 200000 might finish before the others), this will make sure all 5 threads keep working until all of them are done.

Note that depending on your computer's architecture, running too many processes at the same time might slow them all down (for starters, it depends on how many cores your processor has, as well as what else you are running at the same time).

https://en.xdnf.cn/q/73353.html

Related Q&A

Using python, what is the most accurate way to auto determine a users current timezone

I have verified that dateutils.tz.tzlocal() does not work on heroku and even if it did, wouldnt it just get the tz from the OS of the computer its on, not necessarly the users?Short of storing a users…

ImportError: cannot import name ParseMode from telegram

I am trying to create a telegram bot. The code i am trying to execute is : from telegram import ParseModeBut it is throwing up this error: ImportError: cannot import name ParseMode from telegram (C:\Pr…

Executing bash with subprocess.Popen

Im trying to write a wrapper for a bash session using python. The first thing I did was just try to spawn a bash process, and then try to read its output. like this:from subprocess import Popen, PIPE b…

Attribute error when attempting to get a value for field

Im working with the django rest framework and the serializer Im trying to use is creating errors. Im trying to do something like https://gist.github.com/anonymous/7463dce5b0bfcf9b6767 but I still get t…

Why did I have problems with alembic migrations

Project structue(only directory with DB migrations):--db_manage:alembic.ini--alembic:env.pyscript.py.makoREADME--versions:#migration filesWhen I try to run command: python db_manage/alembic/env.py, I h…

Python and App Engine project structure

I am relatively new to python and app engine, and I just finished my first project. It consists of several *.py files (usually py file for every page on the site) and respectively temple files for each…

Text object in matplotlib doesnt respond to zooming properly

. Hello, everyone.I recently try to add text object in my plot. But when I zoom into the text, the text size remains unchanged. What I want to have is that the text size will increase when I zoom in an…

Python unit testing code which calls OS/Module level python functions

I have a python module/script which does a few of theseAt various nested levels inside the script I take command line inputs, validate them, apply sensible defaults I also check if a few directories ex…

How do I connect/disconnect/configure a wireless network in python?

Im looking to see if there is a way to connect or disconnect to a wireless network in python, preferably a way that would work for both public and secured networks if I supplied the password. If I can …

CSRF protection on AJAX authentication in Flask

Id like to AJAXify both a login and a signup form on a site. Up to now Ive been using WTForms mainly for its built-in CSRF protetion, but for this project I didnt feel like it was worth it -- an extra …