fastest way to load images in python for processing

2024/9/19 20:26:24

I want to load more than 10000 images in my 8gb ram in the form of numpy arrays.So far I have tried cv2.imread,keras.preprocessing.image.load_image,pil,imageio,scipy.I want to do it the fastest way possible but I can't figure out which on is it.

Answer

One of the fastest ways is to get your multiprocessors to do your job in Parallel. It brings multiple processors to work on your tasks at the same time when concurrent running isn't an issue. Now the example below is just a simple sketch of how it might look, you can practice with small functions and then integrate them with your own code :

from multiprocessing import Process#this is the function to be parallelizeddef image_load_here(image_path):pass if __name__ == '__main__':#Start the multiprocesses and provide your dataset.p = Process(target=image_load_here,['img1', 'img2', 'img3', 'img4'])p.start()p.join()

Feel free to write, ill try to help.

https://en.xdnf.cn/q/72391.html

Related Q&A

How to access server response when Python requests library encounters the retry limit

I am using the Python requests library to implement retry logic. Here is a simple script I made to reproduce the problem that I am having. In the case where we run out of retries, I would like to be ab…

Matplotlib patch with holes

The following code works. The problem is I dont know exactly why it works. The code draws a circle patch (using PathPatch) with a triangle cutout from the centre. My guess is that the inner triangle is…

Convert sha256 digest to UUID in python

Given a sha256 hash of a str in python: import hashlibhash = hashlib.sha256(foobar.encode(utf-8))How can the hash be converted to a UUID? Note: there will obviously be a many-to-one mapping of hexdige…

Drag and Drop QLabels with PyQt5

Im trying to drag and drop a Qlabel on another Qlabel with PyQt5:from PyQt5.QtWidgets import QApplication, QWidget, QToolTip, QPushButton, QMessageBox, QHBoxLayout, QVBoxLayout, QGridLayout,QFrame, QCo…

replace block within {{ super() }}

I have a base template which includes a block for the default <head> content. Within the head block, theres a block for the <title>.For example, in the base file I would have:<head>{%…

Change Timezone for Date object Python

Hello I am using Pythonanywhere and when I call from datetime import *print date.today().dayIt is printing a different day than the day it is where I live (Austin, Texas). I figured it is because there…

Multiprocessing Pool - how to cancel all running processes if one returns the desired result?

Given the following Python code: import multiprocessingdef unique(somelist):return len(set(somelist)) == len(somelist)if __name__ == __main__:somelist = [[1,2,3,4,5,6,7,8,9,10,11,12,13,2], [1,2,3,4,5],…

Pandas: Product of specific columns

Finding the product of all columns in a dataframe is easy:df[Product] = df.product(axis=1)How can I specify which column names (not column numbers) to include in the product operation?From the help pa…

instagram.bind.InstagramClientError: Unable to parse response, not valid JSON

Made myself a simple Instagram client to make authenticated requests to their API. However, running it keeps throwing the following errorTraceback (most recent call last):File "request-ig-data.py&…

Using jinja to send data to Javascript

I have Python code, in which Im using jinja to send data to a template in Flask. I can access the code just find in HTML, but when I try displaying the data in Javascript, it doesnt work. For example, …