I have a large correlation matrix in a pandas python DataFrame: df (342, 342).
How do I take the mean, sd, etc. of all of the numbers in the upper triangle not including the 1's along the diagonal?
Thank you.
I have a large correlation matrix in a pandas python DataFrame: df (342, 342).
How do I take the mean, sd, etc. of all of the numbers in the upper triangle not including the 1's along the diagonal?
Thank you.
Another potential one line answer:
In [1]: corr
Out[1]:a b c d e
a 1.000000 0.022246 0.018614 0.022592 0.008520
b 0.022246 1.000000 0.033029 0.049714 -0.008243
c 0.018614 0.033029 1.000000 -0.016244 0.049010
d 0.022592 0.049714 -0.016244 1.000000 -0.015428
e 0.008520 -0.008243 0.049010 -0.015428 1.000000In [2]: corr.values[np.triu_indices_from(corr.values,1)].mean()
Out[2]: 0.016381
Edit: added performance metrics
Performance of my solution:
In [3]: %timeit corr.values[np.triu_indices_from(corr.values,1)].mean()
10000 loops, best of 3: 48.1 us per loop
Performance of Theodros Zelleke's one-line solution:
In [4]: %timeit corr.unstack().ix[zip(*np.triu_indices_from(corr, 1))].mean()
1000 loops, best of 3: 823 us per loop
Performance of DSM's solution:
In [5]: def method1(df):...: df2 = df.copy()...: df2.values[np.tril_indices_from(df2)] = np.nan...: return df2.unstack().mean()...:In [5]: %timeit method1(corr)
1000 loops, best of 3: 242 us per loop