Joining Two Different Dataframes on Timestamp

2024/11/13 9:41:19

Say I have two dataframes:

df1:                          df2:
+-------------------+----+    +-------------------+-----+
|  Timestamp        |data|    |  Timestamp        |stuff|
+-------------------+----+    +-------------------+-----+
|2019/04/02 11:00:01| 111|    |2019/04/02 11:00:14|  101|
|2019/04/02 11:00:15| 222|    |2019/04/02 11:00:15|  202|
|2019/04/02 11:00:29| 333|    |2019/04/02 11:00:16|  303|
|2019/04/02 11:00:30| 444|    |2019/04/02 11:00:30|  404|
+-------------------+----+    |2019/04/02 11:00:31|  505|+-------------------+-----+

Without looping through every row of df2, I am trying to join the two dataframes based on the timestamp. So for every row in df2, it will "add" data from df1 that was at that particular time. In this example, the resulting dataframe would be:

Adding df1 data to df2:
+-------------------+-----+----+
|  Timestamp        |stuff|data|
+-------------------+-----+----+
|2019/04/02 11:00:14|  101| 111|
|2019/04/02 11:00:15|  202| 222|
|2019/04/02 11:00:16|  303| 222|
|2019/04/02 11:00:30|  404| 444|
|2019/04/02 11:00:31|  505|None|
+-------------------+-----+----+

Looping through each row of df2 then comparing to each df1 is very inefficient. Is there another way?

Answer

Use merge_asof:

df1['Timestamp'] = pd.to_datetime(df1['Timestamp'])
df2['Timestamp'] = pd.to_datetime(df2['Timestamp'])df = pd.merge_asof(df2, df1, on='Timestamp')
print (df)Timestamp  stuff  data
0 2019-04-02 11:00:14    101   111
1 2019-04-02 11:00:15    202   222
2 2019-04-02 11:00:16    303   222
3 2019-04-02 11:00:30    404   444

Also is possible change order df1 with df2 and add parameter direction='forward':

df = pd.merge_asof(df1, df2, on='Timestamp', direction='forward')
print (df)Timestamp  data  stuff
0 2019-04-02 11:00:01   111  101.0
1 2019-04-02 11:00:15   222  202.0
2 2019-04-02 11:00:29   333  404.0
3 2019-04-02 11:00:30   444  404.0
4 2019-04-02 11:00:31   505    NaN

#default direction='backward'
df = pd.merge_asof(df1, df2, on='Timestamp')
print (df)Timestamp  data  stuff
0 2019-04-02 11:00:01   111    NaN
1 2019-04-02 11:00:15   222  202.0
2 2019-04-02 11:00:29   333  303.0
3 2019-04-02 11:00:30   444  404.0
4 2019-04-02 11:00:31   505  404.0
https://en.xdnf.cn/q/72346.html

Related Q&A

Find if the array contain a 2 next to a 2

I am stuck on this problemGiven an array of ints, return True if the array contains a 2 next to a 2 somewhere.has22([1, 2, 2]) → True has22([1, 2, 1, 2]) → False has22([2, 1, 2]) → FalseI know the b…

AttributeError: xml.etree.ElementTree.Element object has no attribute encode

Im trying to make a desktop notifier, and for that Im scraping news from a site. When I run the program, I get the following error.news[child.tag] = child.encode(utf8) AttributeError: xml.etree.Element…

How to parse code (in Python)?

I need to parse some special data structures. They are in some somewhat-like-C format that looks roughly like this:Group("GroupName") {/* C-Style comment */Group("AnotherGroupName")…

Using OpenCV detectMultiScale to find my face

Im pretty sure I have the general theme correct, but Im not finding any faces. My code reads from c=cv2.VideoCapture(0), i.e. the computers videocamera. I then have the following set up to yield where …

Get marginal effects for sklearn logistic regression

I want to get the marginal effects of a logistic regression from a sklearn modelI know you can get these for a statsmodel logistic regression using .get_margeff(). Is there nothing for sklearn? I want…

How to use win32com.client.constants with MS Word?

Whats wrong with this code? Why win32com.client.constants doesnt have attribute wdWindowStateMinimize?>>> import win32com.client >>> w=win32com.client.Dispatch("Word.Applicatio…

How to properly patch boto3 calls in unit test

Im new to Python unit testing, and I want to mock calls to the boto3 3rd party library. Heres my stripped down code:real_code.py:import boto3def temp_get_variable(var_name):return boto3.client(ssm).ge…

import a github into jupyter notebook directly?

Hey Im creating a jupyter notebook, would like to install: https://github.com/voice32/stock_market_indicators/blob/master/indicators.py which is a python program not sure how to do it directly so anybo…

Django : Call a method only once when the django starts up

I want to initialize some variables (from the database) when Django starts. I am able to get the data from the database but the problem is how should I call the initialize method . And this should be o…

Mocking instance attributes

Please help me understand why the following doesnt work. In particular - instance attributes of a tested class are not visible to Pythons unittest.Mock.In the example below bar instance attribute is no…