pandas filter by multiple columns NULL

2024/10/18 15:05:07

I have a pandas dataframe like:

df = pd.DataFrame({'Last_Name': ['Smith', None, 'Brown'], 'First_Name': ['John', None, 'Bill'],'Age': [35, 45, None]})

And could manually filter it using:

df[df.Last_Name.isnull() & df.First_Name.isnull()]

but this is annoying as I need to write a lot of duplicate code for each column/condition. It is not maintainable if there is a large number of columns. Is it possible to write a function which generates this python code for me?

Some Background: My pandas dataframe is based on an initial SQL-based multi-dimensional Aggregation (grouping-sets) https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-multi-dimensional-aggregation.html so always some different columns are NULL. Now, I want to efficiently select these different groups and analyze them separately in pandas.

Answer

Use filter:

df[df.filter(like='_Name').isna().all(1)]Last_Name First_Name   Age
1      None       None  45.0

Or, if you want more flexibility, specify a list of column names.

cols = ['First_Name', 'Last_Name']
df[df[cols].isna().all(1)]Last_Name First_Name   Age
1      None       None  45.0
https://en.xdnf.cn/q/73121.html

Related Q&A

Closed IPython Notebook that was running code

How it works? I got some code running in an IPython Notebook. Some iterative work. Accidentally I closed the browser with the running Notebook, but going back to the IPython Dashboard I see that this …

os.popen().read() - charmap decoding error

I have already read UnicodeDecodeError: charmap codec cant decode byte X in position Y: character maps to <undefined>. While the error message is similar, the code is completely different, becau…

multiprocessing numpy not defined error

I am using the following test code:from pathos.multiprocessing import ProcessingPool as Pool import numpydef foo(obj1, obj2):a = obj1**2b = numpy.asarray(range(1,5))return obj1, bif __name__ == __main_…

Convert mp4 sound to text in python

I want to convert a sound recording from Facebook Messenger to text. Here is an example of an .mp4 file send using Facebooks API: https://cdn.fbsbx.com/v/t59.3654-21/15720510_10211855778255994_5430581…

pandas map one column to the combination of two columns

I am working with a DataFrame which looks like thisList Numb Name 1 1 one 1 2 two 2 3 three 4 4 four 3 5 fiveand I am trying to compute…

Pyspark command not recognised

I have anaconda installed and also I have downloaded Spark 1.6.2. I am using the following instructions from this answer to configure spark for Jupyter enter link description hereI have downloaded and …

Sliding window in Python for GLCM calculation

I am trying to do texture analysis in a satellite imagery using GLCM algorithm. The scikit-image documentation is very helpful on that but for GLCM calculation we need a window size looping over the im…

Keras multi-label image classification with F1-score

I am working on a multi-label image classification problem with the evaluation being conducted in terms of F1-score between system predicted and ground truth labels.Given that, should I use loss="…

Thread safe locale techniques

Were currently writing a web application based on a threaded python web server framework (cherrypy) and would like to simultaneously support users from multiple locales.The locale module doesnt appear …

How to draw image from raw bytes using ReportLab?

All the examples I encounter in the internet is loading the image from url (either locally or in the web). What I want is to draw the image directly to the pdf from raw bytes.UPDATE:@georgexsh Here is …