Is there a really efficient (FAST) way to read large text files in python?

2024/10/5 15:20:52

I am looking to open and fetch data from a large text file in python as fast as possible (It almost has 62603143 lines - size 550MB). As I don't want to stress my computer, I am doing it by following way ,

import time
start = time.time()
for line in open(filePath):#considering data as last element in fileif data in line:do_something(data)
end = time.time()
print "processing time = %s" % (count, end-start)

But as I am doing by above method its taking almost 18 seconds to read full file ( My computer has Intel i3 processor and 4 GB RAM ). Likewise if file size is more it is taking more time and considering user point of view its very large. I read lot of opinions on forums, referred multiple Stack Overflow questions but didn't get the fast and efficient way to read and fetch the data from large files. Is there really any way in Python to read large text files in few seconds?

Answer

No, there is no faster way of processing a file line by line, not from Python.

Your bottleneck is your hardware, not how you read the file. Python is already doing everything it can (using a buffer to read the file in larger chunks before splitting into newlines).

I suggest upgrading your disk to an SSD.

https://en.xdnf.cn/q/119915.html

Related Q&A

How to extract all K*K submatrix of matrix with or without NumPy?

This is my input: row=6 col=9 6 9 s b k g s y w g f r g y e q j j a s s m s a s z s l e u s q u e h s s s g s f h s s e s g x d r h g y s s sThis is my code: r=int(input()) c=int(input()) n=min(r,c) k=…

How to scrape multiple result having same tags and class

My code is accurate for single page but when I run this code for multiple records using for loop and if there are some data missing like person then (as I used index no[1] and [2] for person variable ,…

Is there an alternative for sys.exit() in python?

try:x="blaabla"y="nnlfa" if x!=y:sys.exit()else:print("Error!") except Exception:print(Exception)Im not asking about why it is throwing an error. I know that it raises e…

Adding items to Listbox in Python Tkinter

I would like my Listbox widget to be updated upon clicking of a button. However I encountered a logic error. When I click on the button, nothing happens. No errors at all.listOfCompanies: [[1, ], [2, -…

Policy based design in Python [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.Want to improve this question? Add details and clarify the problem by editing this post.Closed 9 years ago.Improve…

Exception raised: cannot import name manual_seed from torch

im trying to run the AutoClean project on my device (heres my code): import random from AutoClean import AutoClean import pandas as pddef __init__(self, pipeline, resultat ):self.pipeline = pipelinesel…

Compile a C/C++ Program and store standard output in a File via Python

Lets say I have a C/C++ file named userfile.c. Using Python, how can I invoke the local gcc compiler so that the file is compiled and an executable is made? More specifically, I would like to provide …

How to swap maximums with the minimums? (python)

Is there a method to swap the maximum and the minimum of a list? The list will be as follows and the program has to be continued so that it will print the maximum swapped with the minimum, the second …

python object attributes and methods

In python all data is object and any object should have attributes and methods. Does somebody know python object without any attributes and methods?>>> len(dir(1)) 64

How to retrieve nested data with BeautifulSoup?

I have the below webpage source: </li><li class="cl-static-search-result" title="BELLO HONDA ACCORD &quot;95 MIL MILLAS&quot;. REALMENTE COMO NUEVO"><a href=&…