Add text to end of line without loading file

2024/10/4 15:27:10

I need to store information into a very big file, in form of many dictionaries. That's not so important, is just to say that I tried to first get all the data into these dictionaries and I run out of memory (~60Gb).

Fine, so I want to add the data in the file without actually loading it in memory, by doing a loop on the lines and attaching to each line a bit of text. Is that possible? if so, how?

Answer

Did you try any code yourself at all, what were your findings? You might go along the following approach:

with open('/tmp/bigfile.new', 'w') as output:with open('/tmp/bigfile', 'r') as input:while True:line = input.readline().strip()if not line:breakline += ' Look ma, no hands!'print(line, file=output)

Except of course that instead of 'look ma no hands' you'd have your extra dictionary ;-)

https://en.xdnf.cn/q/70592.html

Related Q&A

How does one use `dis.dis` to analyze performance?

Im trying to use pythons dis library to experiment with & understand performance. Below is an experiment i tried, with the results.import disdef myfunc1(dictionary):t = tuple(dictionary.items())ret…

How do I require HTTPS for this Django view?

(r^login/?$,django.contrib.auth.views.login,{template_name:login.html, authentication_form:CustomAuthenticationForm}),How do I add HTTPS required to this? I usually have a decorator for it..But in th…

How many times a number appears in a numpy array

I need to find a way to count how many times each number from 0 to 9 appears in a random matrix created using np.random.randint()import numpy as np p = int(input("Length of matrix: ")) m = np…

python: How to remove values from 2 lists based on whats in 1 list

I have 2 lists of numbers, one called xVar and the other called yVar. I will use these 2 elements to plot X and Y values on a graph. They both have the same number of elements. Normally, I would jus…

merge two dataframe columns into 1 in pandas

I have 2 columns in my data frame and I need to merge it into 1 single columnIndex A Index B 0 A 0 NAN 1 NAN 1 D 2 B 2 …

Upsample and Interpolate a NumPy Array

I have an array, something like:array = np.arange(0,4,1).reshape(2,2)> [[0 12 3]]I want to both upsample this array as well as interpolate the resulting values. I know that a good way to upsample an…

How to extract text from table in image?

I have data which in a structured table image. The data is like below:I tried to extract the text from this image using this code:import pytesseract from PIL import Imagevalue=Image.open("data/pic…

numpy.savetxt tuple index out of range?

Im trying to write a few lines into a text file, and heres the code I used:import numpy as np# Generate some test data data = np.arange(0.0,1000.0,50.0)with file(test.txt, w) as outfile: outfile.w…

Retrieve list of USB items using Python

How can I retrieve the items plugged into the computer through USB using python? Ive searched around on here and found some old examples which dont appear to work anymore as they are over 5 years old.…

History across ipdb sessions

This question has been asked before, but I couldnt find a good answer. So, I am trying to ask again.I would like my ipdb to remember commands across sessions. Right now, it can pull up commands execute…