I have a sparse dataframe with duplicate indices. How can I merge the same-indexed rows in a way that I keep all the non-NaN data from the conflicting rows?
I know that you can achieve something very close with the built-in drop_duplicates
function, but you can only keep either the first or the last row with the same index:
df.reset_index().drop_duplicates(subset='index', keep='first').set_index('index').sort_index()
What I'd need is all non-nan values, from any of the conflicting rows.
Before:
After: