Turn off list reflection in Numba

2024/10/15 17:18:56

I'm trying to accelerate my code using Numba. One of the arguments I'm passing into the function is a mutable list of lists. When I try changing one of the sublists, I get this error:

Failed in nopython mode pipeline (step: nopython mode backend)cannot reflect element of reflected container: reflected list(reflected list(int64))

I don't actually care about reflecting changes I make to the native list into the original Python list. How do I go about telling Numba not to reflect the changes? The documentation is pretty vague regarding list reflection in Numba.

Thanks,

Answer

Quoting directly from the docs:

In nopython mode, Numba does not operate on Python objects. list are compiled into an internal representation. Any list arguments must be converted into this representation on the way in to nopython mode and their contained elements must be restored in the original Python objects via a process called reflection.

Reflection is required to maintain the same semantics as found in regular Python code. However, the reflection process can be expensive for large lists and it is not supported for lists that contain reflected data types. Users cannot use list-of-list as an argument because of this limitation.

Your best bet would be to give a 2D numpy array of shape len(ll) x max(len(x) for x in ll), ll being the list of lists. I myself use something like this to achieve this, and then pass the arr, lengths to the njit compiled function:

def make_2D_array(lis):"""Funciton to get 2D array from a list of lists"""n = len(lis)lengths = np.array([len(x) for x in lis])max_len = np.max(lengths)arr = np.zeros((n, max_len))for i in range(n):arr[i, :lengths[i]] = lis[i]return arr, lengths

HTH.

https://en.xdnf.cn/q/69258.html

Related Q&A

Identifying large bodies of text via BeautifulSoup or other python based extractors

Given some random news article, I want to write a web crawler to find the largest body of text present, and extract it. The intention is to extract the physical news article on the page. The original p…

Pandas reindex and interpolate time series efficiently (reindex drops data)

Suppose I wish to re-index, with linear interpolation, a time series to a pre-defined index, where none of the index values are shared between old and new index. For example# index is all precise times…

How do you set the box width in a plotly box in python?

I currently have the following;y = time_h time_box = Box(y=y,name=Time (hours),boxmean=True,marker=Marker(color=green),boxpoints=all,jitter=0.5,pointpos=-2.0 ) layout = Layout(title=Time Box, ) fig = F…

how do you install django older version using easy_install?

I just broke my environment because of django 1.3. None of my sites are able to run. So, i decided to use virtualenv to set virtual environment with different python version as well as django.But, seem…

Whats difference between findall() and iterfind() of xml.etree.ElementTree

I write a program using just like belowfrom xml.etree.ElementTree import ETxmlroot = ET.fromstring([my xml content])for element in xmlroot.iterfind(".//mytag"):do some thingit works fine on m…

How to convert string dataframe column to datetime as format with year and week?

Sample Data:Week Price 2011-31 1.58 2011-32 1.9 2011-33 1.9 2011-34 1.9I have a dataframe like above and I wanna convert Week column type from string to datetime.My Code:data[Date_Time…

Tensorflow - ValueError: Shape must be rank 1 but is rank 0 for ParseExample/ParseExample

I have a .tfrecords file of the Ubuntu Dialog Corpus. I am trying to read in the whole dataset so that I can split the contexts and utterances into batches. Using tf.parse_single_example I was able to …

Navigating Multi-Dimensional JSON arrays in Python

Im trying to figure out how to query a JSON array in Python. Could someone show me how to do a simple search and print through a fairly complex array please?The example Im using is here: http://eu.bat…

Numpy, apply a list of functions along array dimension

I have a list of functions of the type:func_list = [lambda x: function1(input),lambda x: function2(input),lambda x: function3(input),lambda x: x]and an array of shape [4, 200, 200, 1] (a batch of image…

Database first Django models

In ASP.NET there is entity framework or something called "database first," where entities are generated from an existing database. Is there something similar for Django? I usually work with …