I am performing a large number of these calculations:

A == A[np.newaxis].T

where A is a dense numpy array which frequently has common values.

For benchmarking purposes we can use:

n = 30000
A = np.random.randint(0, 1000, n)
A == A[np.newaxis].T

When I perform this calculation, I run into memory issues. I believe this is because the output isn't in more efficient bitarray or np.packedbits format. A secondary concern is we are performing twice as many comparisons as necessary, since the resulting Boolean array is symmetric.

The questions I have are:

1. Is it possible to produce the Boolean numpy array output in a more memory efficient fashion without sacrificing speed? The options I know about are bitarray and np.packedbits, but I only know how to apply these after the large Boolean array is created.
2. Can we utilise the symmetry of our calculation to halve the number of comparisons processed, again without sacrificing speed?

I will need to be able to perform & and | operations on Boolean arrays output. I have tried bitarray, which is super-fast for these bitwise operations. But it is slow to pack np.ndarray -> bitarray and then unpack bitarray -> np.ndarray.

[Edited to provide clarification.]

Here's one with numba to give us a NumPy boolean array as output -

from numba import njit@njit
def numba_app1(idx, n, s, out):for i,j in zip(idx[:-1],idx[1:]):s0 = s[i:j]c = 0for p1 in s0[c:]:for p2 in s0[c+1:]:out[p1,p2] = 1out[p2,p1] = 1c += 1return outdef app1(A):s = A.argsort()b = A[s]n = len(A)idx = np.flatnonzero(np.r_[True,b[1:] != b[:-1],True])out = np.zeros((n,n),dtype=bool)numba_app1(idx, n, s, out)out.ravel()[::out.shape[1]+1] = 1return out

Timings -

In [287]: np.random.seed(0)...: n = 30000...: A = np.random.randint(0, 1000, n)# Original soln
In [288]: %timeit A == A[np.newaxis].T
1 loop, best of 3: 317 ms per loop# @Daniel F's soln-1 that skips assigning lower diagonal in output
In [289]: %timeit sparse_outer_eq(A)
1 loop, best of 3: 450 ms per loop# @Daniel F's soln-2 (complete one)
In [291]: %timeit sparse_outer_eq(A)
1 loop, best of 3: 634 ms per loop# Solution from this post
In [292]: %timeit app1(A)
10 loops, best of 3: 66.9 ms per loop