How does thread pooling works, and how to implement it in an async/await env like NodeJS?

2024/10/13 16:16:32

I need to run a function int f(int i) with 10_000 parameters and it takes around 1sec to execute due to I/O time.
In a language like Python, I can use threads (or async/await, I know, but I'll talk about it later) to parallelize this task.
If I want to always have 10 running threads, and to split the task between them, I can use ThreadingPool :

def f(p):x = [...]return xp = ThreadPool()
xs = p.map(f, range(10_000))

But how does it work ? If I want to implement a similar thing with, let's say NodeJS and f = http("www.google.com", callback), where should I begin ? What's the algorithms for this kind of problem ?
Again, I'd like to get 10 requests at the same time, and when one is finished the next one should start.

What I've been thinking so far (ugly because the callback is starting a new call to the f() function):

queue = ["www.google.com", "www.facebook.com"]
var f = function(url) {http.get(url, (e) => {const newUrl = queue.pop();f(newUrl);});
};for (var i = 0; i < 10; i++) {f(queue.pop());
}
Answer

Reimplementation of that Bluebird function I linked to:

const mapWithConcurrency = async (values, concurrency, fn) => {
let i = 0;
let results = values.map(() => null);
const work = async () => {
while (i < values.length) {
const current = i++;
results[current] = await fn(values[current]);
}
};
await Promise.all(Array.from({length: concurrency}, work));
return results;
};
mapWithConcurrency(Array.from({length: 30 * 15}, (_, i) => i), 10, async i => {
const el = document.body.appendChild(document.createElement('i'));
el.style.left = 5 * (i % 30) + 'px';
el.style.top = 5 * (i / 30 | 0) + 'px';
await new Promise(resolve => { setTimeout(resolve, Math.random() * 500); });
el.style.background = 'black';
return 2 * i;
}).then(results => {
console.log(results.length, results.every((x, i) => x === 2 * i));
});
i {
background: grey;
transition: background 0.3s ease-out;
position: absolute;
width: 5px;
height: 5px;
}

https://en.xdnf.cn/q/69515.html

Related Q&A

Calculate centroid of entire GeoDataFrame of points

I would like to import some waypoints/markers from a geojson file. Then determine the centroid of all of the points. My code calculates the centroid of each point not the centroid of all points in the …

Flask-Babel localized strings within js

Im pretty new to both Python and Flask (with Jinja2 as template engine) and I am not sure I am doing it the right way. I am using Flask-Babel extension to add i18n support to my web application. I want…

a (presumably basic) web scraping of http://www.ssa.gov/cgi-bin/popularnames.cgi in urllib

I am very new to Python (and web scraping). Let me ask you a question. Many website actually do not report its specific URLs in Firefox or other browsers. For example, Social Security Admin shows popul…

Why is tuple being returned?

I have the following:tableNumber = session.query(TABLE.TABLESNUMBER).filter_by(TABLESID=self.TABLESID).first() return str(tableNumber)This is my TABLE class:class TABLE(Base):.... TABLESID =…

How to assert both UserWarning and SystemExit in pytest

Assert UserWarning and SystemExit in pytestIn my application I have a function that when provided with wrong argument values will raise a UserWarnings from warnings module and then raises SystemExit fr…

Distinguish button_press_event from drag and zoom clicks in matplotlib

I have a simple code that shows two subplots, and lets the user left click on the second subplot while recording the x,y coordinates of those clicks.The problem is that clicks to select a region to zoo…

String reversal in Python

I have taken an integer input and tried to reverse it in Python but in vain! I changed it into a string but still I am not able to. Is there any way to reverse it ? Is there any built-in function?I a…

Python: passing functions as arguments to initialize the methods of an object. Pythonic or not?

Im wondering if there is an accepted way to pass functions as parameters to objects (i.e. to define methods of that object in the init block).More specifically, how would one do this if the function de…

Encrypt and Decrypt by AES algorithm in both python and android

I have python and android code for AES encryption. When I encrypt a text in android, it decrypt on python successfully but it can’t decrypt in android side. Do anyone have an idea?Python code :impo…

How to conditionally assign values to tensor [masking for loss function]?

I want to create a L2 loss function that ignores values (=> pixels) where the label has the value 0. The tensor batch[1] contains the labels while output is a tensor for the net output, both have a …