How do I replace NA with NaN in a Pandas DataFrame?

2024/10/6 1:46:06

Some columns in my DataFrame have instances of <NA> which are of type pandas._libs.missing.NAType.

I'd like to replace them with NaN using np.nan.

I have seen questions where the instances of <NA> can be replaced when using pd.read_csv().

But since my Pandas DataFrame is created from a Spark DataFrame I do not use the pd.read_csv() function.

Please Advise.

Answer

I didn't have any luck with the replace solution but was able to convert <NA> to np.nan by converting the column to float - df['my_col'].astype(float).

https://en.xdnf.cn/q/70415.html

Related Q&A

concatenation of two or more base64 strings in python

Im tring to concatenate two strings encoded to base64 but it doesnt really work, just prints the first string in concatanation:q = base64.b64encode("StringA") print q # prints an encoded stri…

How to find shared library used by a python module?

I am debugging a python program based on pygtk and I want to make sure that the program is using the right shared library.pygtk is a GTK+ wrapper for python. I have already compiled GTK+ using jhbuild …

Python groupby doesnt work as expected [duplicate]

This question already has answers here:itertools.groupby() not grouping correctly(3 answers)Closed 5 years ago.I am trying to read an excel spreadsheet that contains some columns in following format:co…

Dask: create strictly increasing index

As is well documented, Dask creates a strictly increasing index on a per partition basis when reset_index is called, resulting in duplicate indices over the whole set. What is the best way (e.g. comput…

Installing hunspell package

Im looking forward to install the hunspell package using pip, but it throws the following error:Collecting hunspellUsing cached hunspell-0.4.1.tar.gz Building wheels for collected packages: hunspellRun…

Flask-Restful taking over exception handling from Flask during non debug mode

Ive used Flasks exception handling during development (@app.errorhander(MyException)) which worked fine even for exceptions coming from Flask-Restful endpoints.However, I noticed that when switching to…

Fetching data with snowflake connector throws EmptyPyArrowIterator error

I use python snowflake connector in my python script (plotly dash app) and today the app stopped working without me changing the code. I tried a couple of things to find out what might be the issue and…

What does epochs mean in Doc2Vec and train when I have to manually run the iteration?

I am trying to understand the epochs parameter in the Doc2Vec function and epochs parameter in the train function. In the following code snippet, I manually set up a loop of 4000 iterations. Is it requ…

TensorFlow 2.0 How to get trainable variables from tf.keras.layers layers, like Conv2D or Dense

I have been trying to get the trainable variables from my layers and cant figure out a way to make it work. So here is what I have tried:I have tried accessing the kernel and bias attribute of the Dens…

Convert Excel row,column indices to alphanumeric cell reference in python/openpyxl

I want to convert the row and column indices into an Excel alphanumeric cell reference like A1. Im using python and openpyxl, and I suspect theres a utility somewhere in that package that does this, bu…

Latest Q&A