Pandas dataframe boolean mask on multiple columns

2024/9/24 9:27:24

I have a dataframe (df) containing several columns with an actual measure and corresponding number of columns (A,B,...) with an uncertainty (dA, dB, ...) for each of these columns:

   A    B    dA      dB
0 -1    3    0.31    0.08
1  2   -4    0.263   0.357
2  5    5    0.382   0.397
3 -4   -0.5  0.33    0.115

I apply a function to find values in the measurement columns that are valid according to my definition

df[["A","B"]].apply(lambda x: x.abs()-5*df['d'+x.name] > 0)

This will return a boolean array:

     A          B 
0    False      True
1    True       True
2    True       True
3    True       False

I would like to use this array to select rows in dataframe for which the condition is true within a single column, e.g. A -> row 1-3, and also find rows where the condition is true for all the input columns, e.g. row 1 and 2. Is there an efficient way to do this with pandas?

Answer

You can use the results of your apply statement to boolean index select from the original dataframe:

results = df[["A","B"]].apply(lambda x: x.abs()-5*df['d'+x.name] > 0)

Which returns your boolean array above:

       A      B
0  False   True
1   True   True
2   True   True
3   True  False

Now, you can use this array to select rows from your original datafame as follows:

Select where A is True:

df[results.A]A    B     dA     dB
1  2 -4.0  0.263  0.357
2  5  5.0  0.382  0.397
3 -4 -0.5  0.330  0.115

Select where either A or B are true:

df[results.any(axis=1)]A    B     dA     dB
0 -1  3.0  0.310  0.080
1  2 -4.0  0.263  0.357
2  5  5.0  0.382  0.397
3 -4 -0.5  0.330  0.115

Select where all the columns true:

df[results.all(axis=1)]A    B     dA     dB
1  2 -4.0  0.263  0.357
2  5  5.0  0.382  0.397
https://en.xdnf.cn/q/71717.html

Related Q&A

Which of these scripting languages is more appropriate for pen-testing? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.Clo…

Keras: Optimal epoch selection

Im trying to write some logic that selects the best epoch to run a neural network in Keras. My code saves the training loss and the test loss for a set number of epochs and then picks the best fitting …

error in loading pickle

Not able to load a pickle file. I am using python 3.5import pickle data=pickle.load(open("D:\\ud120-projects\\final_project\\final_project_dataset.pkl", "r"))TypeError: a bytes-lik…

How to test if a webpage is an image

Sorry that the title wasnt very clear, basically I have a list with a whole series of urls, with the intention of downloading the ones that are pictures. Is there anyway to check if the webpage is an i…

Generic detail view ProfileView must be called with either an object pk or a slug

Im new to Django 2.0 and im getting this error when visiting my profile page view. Its working with urls like path(users/<int:id>) but i wanted to urls be like path(<username>). Not sure wh…

Python Pandas group datetimes by hour and count row

This is my transaction dataframe, where each row mean a transaction :date station 30/10/2017 15:20 A 30/10/2017 15:45 A 31/10/2017 07:10 A 31/10/2017 07:25 B 31/10/2017 07:55 …

Get Bokehs selection in notebook

Id like to select some points on a plot (e.g. from box_select or lasso_select) and retrieve them in a Jupyter notebook for further data exploration. How can I do that?For instance, in the code below, …

Upload CSV file into Microsoft Azure storage account using python

I am trying to upload a .csv file into Microsoft Azure storage account using python. I have found C-sharp code to write a data to blob storage. But, I dont know C# language. I need to upload .csv file …

GeoDjango: How can I get the distance between two points?

My Profile model has this field:location = models.PointField(geography=True, dim=2, srid=4326)Id like to calculate the distance between the two of these locations (taking into account that the Earth is…

Reading direct access binary file format in Python

Background:A binary file is read on a Linux machine using the following Fortran code:parameter(nx=720, ny=360, nday=365) c dimension tmax(nx,ny,nday),nmax(nx,ny,nday)dimension tmin(nx,ny,nday),nmin(nx,…