How to put many numpy files in one big numpy file without having memory error?

2024/10/15 3:14:00

I follow this question Append multiple numpy files to one big numpy file in python in order to put many numpy files in one big file, the result is:

import matplotlib.pyplot as plt 
import numpy as np
import glob
import os, sys
fpath ="path_Of_my_final_Big_File"
npyfilespath ="path_of_my_numpy_files"   
os.chdir(npyfilespath)
npfiles= glob.glob("*.npy")
npfiles.sort()
all_arrays = np.zeros((166601,8000))
for i,npfile in enumerate(npfiles):all_arrays[i]=np.load(os.path.join(npyfilespath, npfile))
np.save(fpath, all_arrays)
data = np.load(fpath)
print data
print data.shape

I have thousands of files, by using this code, I have always a memory error, so I can't have my result file. How to resolve this error? How to read, write and append int the final numpy file by file, ?

Answer

Try to have a look to np.memmap. You can instantiateall_arrays:

all_arrays = np.memmap("all_arrays.dat", dtype='float64', mode='w+', shape=(166601,8000))

from the documentation:

Memory-mapped files are used for accessing small segments of large files on disk, without reading the entire file into memory.

You will be able to access all the array, but the operating system will take care of loading the part that you actually need. Read carefully the documentation page and note that from the performance point of view you can decide whether the file should be stored column-wise or row-wise.

https://en.xdnf.cn/q/117876.html

Related Q&A

scraping : nested url data scraping

I have a website name https://www.grohe.com/in In that page i want to get one type of bathroom faucets https://www.grohe.com/in/25796/bathroom/bathroom-faucets/grandera/ In that page there are multiple…

How to trigger an action once on overscroll in Kivy?

I have a ScrollView thats supposed to have an update feature when you overscroll to the top (like in many apps). Ive found a way to trigger it when the overscroll exceeds a certain threshold, but it tr…

Python - Print Each Sentence On New Line

Per the subject, Im trying to print each sentence in a string on a new line. With the current code and output shown below, whats the syntax to return "Correct Output" shown below?Codesentenc…

pyinstaller struct.error: unpack requires a bytes object of length 16 [closed]

Closed. This question needs debugging details. It is not currently accepting answers.Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to repro…

Getting the quarter where recession start and recession ends along with the quarter of minimum gdp

Quarter: GDP: GDP change: change 1999q3 9 -- ------ 1999q4 10 1 increase 2000q1 9 -1 decline 2000q2 8 -1 de…

Inherit view and adding fields

I want to add my 2 fields boatlenght and fuelcapacity under price list in product form view but they are not showing up. What did i miss.<?xml version="1.0" encoding="utf-8"?&g…

Linux and python: Combining multiple wave files to one wave file

I am looking for a way that I can combine multiple wave files into one wave file using python and run it on linux. I dont want to use any add on other than the default shell command line and default py…

How does the in operator determine membership? [duplicate]

This question already has answers here:Set "in" operator: uses equality or identity?(5 answers)Closed 7 years ago.How does the in operator work for Python? In the example below I have two n…

Python Automatically ignore unicode string

Ive been searching to automatically import some files but since Im on Windows i got the unicode error (because of the "C:\Users\..."). Ive been looking to correct this error and found some h…

How to obtain currency rates from this website converter widget python

How can I implement the currency rates on this website and keep the currencies up to date so that i can access them in python from this website and input and output values and currencies types. I need …