How does Pythonic garbage collection with numpy array appends and deletes?

2024/11/15 18:32:31

I am trying to adapt the underlying structure of plotting code (matplotlib) that is updated on a timer to go from using Python lists for the plot data to using numpy arrays. I want to be able to lower the time step for the plot as much as possible, and since the data may get up into the thousands of points, I start to lose valuable time fast if I can't. I know that numpy arrays are preferred for this sort of thing, but I am having trouble figuring out when I need to think like a Python programmer and when I need to think like a C++ programmer maximize my efficiency of memory access.

It says in the scipy.org docs for the append() function that it returns a copy of the arrays appended together. Do all these copies get garbage-collected properly? For example:

import numpy as npa = np.arange(10)
a = np.append(a,10)
print a

This is my reading of what is going on on the C++-level, but if I knew what I was talking about, I wouldn't be asking the question, so please correct me if I'm wrong! =P

First a block of 10 integers gets allocated, and the symbol a points to the beginning of that block. Then a new block of 11 integers is allocated, for a total of 21 ints (84 bytes) being used. Then the a pointer is moved to the start of the 11-int block. My guess is that this would result in the garbage-collection algorithm decrementing the reference count of the 10-int block to zero and de-allocating it. Is this right? If not, how do I ensure I don't create overhead when appending?

I also am not sure how to properly delete a numpy array when I am done using it. I have a reset button on my plots that just flushes out all the data and starts over. When I had lists, this was done using del data[:]. Is there an equivalent function for numpy arrays? Or should I just say data = np.array([]) and count on the garbage collector to do the work for me?

Answer

The point of automatic memory management is that you don't think about it. In the code that you wrote, the copies will be garbage-collected fine (it's nigh on impossible to confuse Python's memory management). However, because np.append is not in-place, the code will create a new array in memory (containing the concatenation of a and 10) and then the variable a will be updated to point to this new array. Since a now no longer points to the original array, which had a refcount of 1, its refcount is decremented to 0 and it will be cleaned up automatically. You can use gc.collect to force a full cleanup.

Python's strength does not lie in fine-tuning memory access, although it is possible to optimise. You are probably best sorted pre-allocating a (using e.g. a = np.zeros( <size> )); if you need finer tuning than that it starts to get a bit hairy. You could have a look at the Cython + Numpy tutorial for a very neat and easy way to integrate C with Python for efficiency.

Variables in Python just point to the location where their contents are stored; you can del any variable and it will decrease the reference count of its target by one. The target will be cleaned automatically after its reference count hits zero. The moral of this is, don't worry about cleaning up your memory. It will happen automatically.

https://en.xdnf.cn/q/71757.html

Related Q&A

What is the default if I install virtualenv using pip and pip3 respectively?

I used sudo pip install virtualenv, then when I run virtualenv ENV in a directory, I get a Python 2 virtual enviroment.If I use pip3 install virtualenv to install virtualenv again, will it override the…

Flask Admin ModelView different fields between CREATE and EDIT

In a Flask app using Flask Admin I would like to be able to define different form fields in the Edit section of a ModelView than those in the Create section.The form_columns setting applies to both Cre…

Python compilation error: LONG_BIT definition appears wrong for platform

This error message is not unknown, I have already reinstalled many packages, but so far not yet found a solution.I get the following error from the command pip install cryptography/usr/include/python2.…

Randomly sampling lines from a file

I have a csv file which is ~40gb and 1800000 lines. I want to randomly sample 10,000 lines and print them to a new file.Right now, my approach is to use sed as:(sed -n $vars < input.txt) > output…

Is Pythons hashlib.sha256(x).hexdigest() equivalent to Rs digest(x,algo=sha256)

Im not a python programmer, but Im trying to translate some Python code to R. The piece of python code Im having trouble with is:hashlib.sha256(x).hexdigest()My interpretation of this code is that the…

How to do Histogram Equalization on specific area

I have a image and I want to do HE or CLAHE on specific area of the image. I already have a mask for the image. Is there any possible way to do so?

Timing out a multiprocessing function

I need to set a time limit on a python function which use some multiprocessing stuff (I dont know if it matters). Something like this:function(a_list):p1 = Process(a_list[0:len(a_list/2)])p2 = Process(…

How can I get the actual axis limits when using ax.axis(equal)?

I am using ax.axes(equal) to make the axis spacing equal on X and Y, and also setting xlim and ylim. This over-constrains the problem and the actual limits are not what I set in ax.set_xlim() or ax.set…

Python rarfile package: fail to open files

So I was trying to archive a .rar file using rarfile library in Python, but it keeps saying "failed to open". Am using Mac OS X El Capitan, python 2.7. Any help would be appreciated, thanks.O…

Sublime Text 3 Python Interactive Console? [duplicate]

This question already has answers here:Cant send input to running program in Sublime Text(5 answers)Closed 7 years ago.I have been using a lot of sublime text 3 to write python. However, whenever a pro…