How does numpy broadcasting perform faster?

2024/10/3 4:33:06

In the following question, https://stackoverflow.com/a/40056135/5714445

Numpy's broadcasting provides a solution that's almost 6x faster than using np.setdiff1d() paired with np.view(). How does it manage to do this?

And using A[~((A[:,None,:] == B).all(-1)).any(1)] speeds it up even more. Interesting, but raises yet another question. How does this perform even better?

Answer

I would try to answer the second part of the question.

So, with it we are comparing :

A[np.all(np.any((A-B[:, None]), axis=2), axis=0)]  (I)

and

A[~((A[:,None,:] == B).all(-1)).any(1)]

To compare with a matching perspective against the first one, we could write down the second approach like this -

A[(((~(A[:,None,:] == B)).any(2))).all(1)]         (II)

The major difference when considering performance, would be the fact that with the first one, we are getting non-matches with subtraction and then checking for non-zeros with .any(). Thus, any() is made to operate on an array of non-boolean dtype array. In the second approach, instead we are feeding it a boolean array obtained with A[:,None,:] == B.

Let's do a small runtime test to see how .any() performs on int dtype vs boolean array -

In [141]: A = np.random.randint(0,9,(1000,1000)) # An int arrayIn [142]: %timeit A.any(0)
1000 loops, best of 3: 1.43 ms per loopIn [143]: A = np.random.randint(0,9,(1000,1000))>5 # A boolean arrayIn [144]: %timeit A.any(0)
10000 loops, best of 3: 164 µs per loop

So, with close to 9x speedup on this part, we see a huge advantage to use any() with boolean arrays. This I think was the biggest reason to make the second approach faster.

https://en.xdnf.cn/q/70767.html

Related Q&A

python check if utf-8 string is uppercase

I am having trouble with .isupper() when I have a utf-8 encoded string. I have a lot of text files I am converting to xml. While the text is very variable the format is static. words in all caps should…

Failed building wheel for Twisted in Windows 10 python 3

Im trying to install rasa-core on my windows 10 machine.When installing with pip install, I get: Failed building wheel for TwistedThe same error appears when trying to install Twisted separately.How co…

How to set a fill color between two vertical lines

Using matplotlib, we can "trivially" fill the area between two vertical lines using fill_between() as in the example: https://matplotlib.org/3.2.1/gallery/lines_bars_and_markers/fill_between_…

Pythonic way to read file line by line?

Whats the Pythonic way to go about reading files line by line of the two methods below?with open(file, r) as f:for line in f:print lineorwith open(file, r) as f:for line in f.readlines():print lineOr …

Python Selenium clicking next button until the end

This is a follow up question from my first question, and I am trying to scrape a website and have Selenium click on next (until it cant be clicked) and collect the results as well.This is the html tag …

How can i detect one word with speech recognition in Python

I know how to detect speech with Python but this question is more specific: How can I make Python listening for only one word and then returns True if Python could recognize the word.I know, I could ju…

Finding matching and nonmatching items in lists

Im pretty new to Python and am getting a little confused as to what you can and cant do with lists. I have two lists that I want to compare and return matching and nonmatching elements in a binary form…

How to obtain better results using NLTK pos tag

I am just learning nltk using Python. I tried doing pos_tag on various sentences. But the results obtained are not accurate. How can I improvise the results ?broke = NN flimsy = NN crap = NNAlso I am …

Pandas apply on rolling with multi-column output

I am working on a code that would apply a rolling window to a function that would return multiple columns. Input: Pandas Series Expected output: 3-column DataFrame def fun1(series, ):# Some calculation…

Exceptions for the whole class

Im writing a program in Python, and nearly every method im my class is written like this: def someMethod(self):try:#...except someException:#in case of exception, do something here#e.g display a dialog…