I have a lot of image files in a folder (5M+). These images are of different sizes. I want to resize these images to 128x128.

I used the following function in a loop to resize in Python using OpenCV

def read_image(img_path):# print(img_path)img = cv2.imread(img_path)img = cv2.resize(img, (128, 128))return imgfor file in tqdm(glob.glob('train-images//*.jpg')):img = read_image(file)img = cv2.imwrite(file, img)

But it will take more than 7 hours to complete. I was wondering whether there are any method to speed up this process.

Can I implement parallel processing to do this efficiently with dask or something.? If so how is it possible.?

If you are absolutely intent on doing this in Python, then please just disregard my answer. If you are interested in getting the job done simply and fast, read on...

I would suggest GNU Parallel if you have lots of things to be done in parallel and even more so as CPUs become "fatter" with more cores rather than "taller" with higher clock rates (GHz).

At its simplest, you can use ImageMagick just from the command line in Linux, macOS and Windows like this to resize a bunch of images:

magick mogrify -resize 128x128\! *.jpg

If you have hundreds of images, you would be better running that in parallel which would be:

parallel magick mogrify -resize 128x128\! ::: *.jpg

If you have millions of images, the expansion of *.jpg will overflow your shell's command buffer, so you can use the following to feed the image names in on stdin instead of passing them as parameters:

find -iname \*.jpg -print0 | parallel -0 -X --eta magick mogrify -resize 128x128\!

There are two "tricks" here:

• I use find ... -print0 along with parallel -0 to null-terminate filenames so there are no problems with spaces in them,

• I use parallel -X which means, rather than start a whole new mogrify process for each image, GNU Parallel works out how many filenames mogrify can accept, and gives it that many in batches.

I commend both tools to you.

Whilst the ImageMagick aspects of the above answer work on Windows, I don't use Windows and I am unsure about using GNU Parallel there. I think it maybe runs under git-bash and/or maybe under Cygwin - you could try asking a separate question - they are free!

As regards the ImageMagick part, I think you can get a listing of all the JPEG filenames in a file using this command:

DIR /S /B *.JPG > filenames.txt

You can then probably process them (not in parallel) like this:

magick mogrify -resize 128x128\! @filenames.txt

And if you find out how to run GNU Parallel on Windows, you can probably process them in parallel using something like this:

parallel --eta -a filenames.txt magick mogrify -resize 128x128\!