Intersection of multiple pandas dataframes

2024/9/16 19:21:00

I have a number of dataframes (100) in a list as:

frameList = [df1,df2,..,df100]

Each dataframe has the two columns DateTime, Temperature.

I want to intersect all the dataframes on the common DateTime column and get all their Temperature columns combined/merged into one big dataframe: Temperature from df1, Temperature from df2, Temperature from df3, .., Temperature from df100.

(pandas merge doesn't work as I'd have to compute multiple (99) pairwise intersections).

Answer

Use pd.concat, which works on a list of DataFrames or Series.

pd.concat(frameList, axis=1, join='inner')

This is better than using pd.merge, as pd.merge will copy the data pairwise every time it is executed. pd.concat copies only once. However, pd.concat only merges based on an axes, whereas pd.merge can also merge on (multiple) columns.

https://en.xdnf.cn/q/72670.html

Related Q&A

docker with pycharm 5

I try to build a docker-based development box for our django app. Its running smoothly.None of my teammembers will care about that until there is a nice IDE integration, therefore I play the new and sh…

How to make a simple Python REST server and client?

Im attempting to make the simplest possible REST API server and client, with both the server and client being written in Python and running on the same computer.From this tutorial:https://blog.miguelgr…

Histogram fitting with python

Ive been surfing but havent found the correct method to do the following.I have a histogram done with matplotlib:hist, bins, patches = plt.hist(distance, bins=100, normed=True)From the plot, I can see …

Subtract each row of matrix A from every row of matrix B without loops

Given two arrays, A (shape: M X C) and B (shape: N X C), is there a way to subtract each row of A from each row of B without using loops? The final output would be of shape (M N X C).Example A = np.ar…

Programmatically setting access control limits in mosquitto

I am working on an application that will use mqtt. I will be using the python library. I have been leaning towards using mosquitto but can find no way of programmatically setting access control limits …

Optimizing cartesian product between two Pandas Dataframe

I have two dataframes with the same columns:Dataframe 1:attr_1 attr_77 ... attr_8 userID John 1.2501 2.4196 ... 1.7610 Charles 0.0000 1.0618 ... 1.4813 Genarit…

Tensorflow: open a PIL.Image?

I have a script that obscures part of an image and runs it through a prediction net to see which parts of the image most strongly influence the tag prediction. To do this, I open a local image with PIL…

Django: Saving to DB from form example

It seems I had difficulty finding a good source/tutorial about saving data to the DB from a form. And as it progresses, I am slowly getting lost. I am new to Django, and please guide me. I am getting e…

eval(input()) in python 2to3

From the Python 2to3 doc:input:Converts input(prompt) to eval(input(prompt))I am currently trying to learn Python 3 after a few years working with Python 2. Can anybody please explain why the tool inse…

Post XML file using Python

Im new to Python and in need of some help. My aim is to send some XML with a post request to a URL, which is going to trigger a SMS being sent. I have a small XML document that I want to post to the UR…