Python memory limit [duplicate]

2024/10/5 16:31:22

So clearly there cannot be unlimited memory in Python. I am writing a script that creates lists of dictionaries. But each list has between 250K and 6M objects (there are 6 lists).

Is there a way to actually calculate (possibly based on the RAM of the machine) the maximum memory and the memory required to run the script?

The actual issue I came across:

In running one of the scripts that populates a list with 1.25-1.5 million dictionaries, when it hits 1.227... it simply stops, but returns no error let alone MemoryError. So I am not even sure if this is a memory limit. I have print statements so I can watch what is going on, and it seems to buffer forever as nothing is printing and up until that specific section, the code is running a couple thousand lines per second. Any ideas as to what is making it stop? Is this memory or something else?

Answer

If you have that many objects you need to store, you need store them on disk, not in memory. Consider using a database.

If you import the sys module you will have access to the function sys.getsizeof(). You will have to look at each object of the list and for each dictionary compute the value for every key. For more on this see this previous question - In-memory size of a Python structure.

https://en.xdnf.cn/q/119051.html

Related Q&A

Automating Gmail login in Python

I am writing a Python program that can login Gmail.The purpose of this program is to check whether the username/password combination exists and is correct.Since this program is to test the the username…

Reversing order in incrementing digits

I have a list of numbers, and Im trying to do the following in a way as efficient as possible.For each consecutively incrementing chunk in the list I have to reverse its order.This is my attempt so far…

why python selenium get the empty page_source?

I try to simulate the buy item operation on the link below. (Need login in first)taobao_item_linkAnd after you click the button below. img_link:The link will jump to a new link.But if I print out the p…

How do i print the repetition output using regex it prints only first match

I have task where I need to print the occurrence and count and non-occurrence and count import resequence = 1222311m = re.search(r(\d)\1+,sequence)print(m) Exptected output : (1, 1) (3, 2) (1, 3) (2…

How do I get rid of all roles of the user discord.py

I was making my mute command and I thought of getting rid of all the members role but I dont know how to get rid of all the members roles I even tried for role in member.roles:await member.remove_roles…

How to pick the rows which contains all the keywords? [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.Want to improve this question? Add details and clarify the problem by editing this post.Closed 6 years ago.Improve…

Extract HTML Tables With Similar Data from Different Sources with Different Formatting - Python

I am trying to scrape HTML tables from two different HTML sources. Both are very similar, each table includes the same data but they may be structured differently, with different column names etc. For …

AttributeError: NoneType object has no attribute replace_with

I am getting the following error:Traceback (most recent call last):File "2.py", line 22, in <module>i.string.replace_with(i.string.replace(u\xa0, -)) AttributeError: NoneType object has…

How to expand out a Pyspark dataframe based on column?

How do I expand a dataframe based on column values? I intend to go from this dataframe:+---------+----------+----------+ |DEVICE_ID| MIN_DATE| MAX_DATE| +---------+----------+----------+ | 1|…

How can I trigger my python script to automatically run via a ping?

I wrote a script that recurses through a set of Cisco routers in a network, and gets traffic statistics. On the router itself, I have it ping to the loopback address of my host PC, after a traffic thre…