From subprocess.Popen to multiprocessing

2024/10/4 9:27:43

I got a function that invokes a process using subprocess.Popen in the following way:

    def func():...process = subprocess.Popen(substr, shell=True, stdout=subprocess.PIPE)timeout = {"value": False}timer = Timer(timeout_sec, kill_proc, [process, timeout])timer.start()for line in process.stdout:lines.append(line)timer.cancel()if timeout["value"] == True:return 0...

I call this function from other function using a loop (e.g from range(1,100) ) , how can I make multiple calls to the function with multiprocessing? that each time several processes will run in parallel

The processes doesn't depend on each other, the only constraint is that each process would be 'working' on only one index (e.g no two processes will work on index 1)

Thanks for your help

Answer

Just add the index to your Popen call and create a worker pool with as many CPU cores you have available.

import multiprocessingdef func(index):....process = subprocess.Popen(substr + " --index {}".format(index), shell=True, stdout=subprocess.PIPE)....if __name__ == '__main__':p = multiprocessing.Pool(multiprocessing.cpu_count())p.map(func, range(1, 100))
https://en.xdnf.cn/q/70624.html

Related Q&A

Assigning float as a dictionary key changes its precision (Python)

I have a list of floats (actually its a pandas Series object, if it changes anything) which looks like this:mySeries:... 22 16.0 23 14.0 24 12.0 25 10.0 26 3.1 ...(So elements…

Installing jpype in Mountain Lion

I am trying to install jpype in Mountain Lion. I followed all the steps suggested in this post: How to install JPype on OS X Lion to use with Neo4j?However, there is a glitch with Mountain Lion. I hav…

Most efficient way to index words in a document?

This came up in another question but I figured it is best to ask this as a separate question. Give a large list of sentences (order of 100 thousands):[ "This is sentence 1 as an example", &qu…

python libclang bindings on Windows fail to initialize a translation unit from sublime text

Short description: using libclang to autocomplete code does not work with python that comes bundled with Sublime Text 3.Details: A small verifiable example is in the repo on GithubIn essence, there is …

How to create a simple Gradient Descent algorithm

Im studying simple machine learning algorithms, beginning with a simple gradient descent, but Ive got some trouble trying to implement it in python. Here is the example Im trying to reproduce, Ive got …

login_required decorator on a class based view in django

I have a working class based view. But when adding @login_required I get the error:AttributeError: function object has no attribute as_viewSomething is happening to the ResultListView here:from django.…

Generic way to get primary key from declaratively defined instance in SQLAlchemy

Does SQLAlchemy offer a generic way to get the primary key from a declaratively defined instance, so that if:Base = declarative_base()class MyClass(Base):__tablename__ = mytablekey = Column(Integer, pr…

Add column after another column

How can I add a column after another column to a database using Alembic or SQLAlchemy? That would be equivalent to this SQL clause: ALTER TABLE foo CHANGE COLUMN bar bar COLUMN_DEFINITION_HERE AFTER …

Scrapy Images Downloading

My spider runs without displaying any errors but the images are not stored in the folder here are my scrapy files:Spider.py:import scrapy import re import os import urlparse from scrapy.spiders import …

A full and minimal example for a class (not method) with Python C Extension?

Everywhere, I can easily find an example about writing a method with Python C Extensions and use it in Python. Like this one: Python 3 extension example$ python3 >>> import hello >>> …