Pandas DatetimeIndex vs to_datetime discrepancies

2024/9/8 8:54:49

I'm trying to convert a Pandas Series of epoch timestamps to human-readable times. There are at least two obvious ways to do this: pd.DatetimeIndex and pd.to_datetime(). They seem to work in quite different ways:

In [1]: import pandas as pdIn [3]: nanos = pd.Series([1462282258000000000, 1462282258100000000, 1462282258200000000])In [4]: pd.to_datetime(nanos)
Out[4]: 
0   2016-05-03 13:30:58.000
1   2016-05-03 13:30:58.100
2   2016-05-03 13:30:58.200
dtype: datetime64[ns]In [5]: pd.DatetimeIndex(nanos)
Out[5]: 
DatetimeIndex([       '2016-05-03 13:30:58', '2016-05-03 13:30:58.100000','2016-05-03 13:30:58.200000'],dtype='datetime64[ns]', freq=None)

With to_datetime(), the display resolution is milliseconds, and .000 is printed on whole seconds. With DatetimeIndex, the display resolution is microseconds (which I like), but the decimal part is completely omitted on whole seconds.

Then, try converting the time zone:

In [12]: pd.DatetimeIndex(nanos).tz_localize('UTC')                   
Out[12]: 
DatetimeIndex([       '2016-05-03 13:30:58+00:00','2016-05-03 13:30:58.100000+00:00','2016-05-03 13:30:58.200000+00:00'],dtype='datetime64[ns, UTC]', freq=None)In [13]: pd.to_datetime(nanos).tz_localize('UTC')  
TypeError: index is not a valid DatetimeIndex or PeriodIndex

This is strange: the timezone functions don't work with a plain datetime Series, only with a DatetimeIndex. Why would that be? The tz_localize() method exists and is documented here: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.tz_localize.html

I've tried Pandas 0.17.0 and 0.18.1 with the same results.

I'm not trying to make an actual index, so all else being equal I would have expected to use to_datetime() - I just can't get time zone methods to work with it.

Answer

There is 1 way to convert things, pd.to_datetime(), yes you can directly construct a DatetimeIndex, but it is restrictive on purpose, while to_datetime is quite flexible.

So to_datetime will give you a similar object to what you input, if you input an array-like, then you will get a DatetimeIndex, input a Series you will get a Series.

In [5]: nanos = [1462282258000000000, 1462282258100000000, 1462282258200000000]

By default it will convert with a unit='ns' which lines up here

In [7]: pd.to_datetime(nanos)
Out[7]: DatetimeIndex(['2016-05-03 13:30:58', '2016-05-03 13:30:58.100000', '2016-05-03 13:30:58.200000'], dtype='datetime64[ns]', freq=None)

So one thing we could do is make a Series out of this. The index is INTEGER here, the values are Datetimes.

In [10]: s = Series(pd.to_datetime(nanos))In [11]: s
Out[11]: 
0   2016-05-03 13:30:58.000
1   2016-05-03 13:30:58.100
2   2016-05-03 13:30:58.200
dtype: datetime64[ns]

You then can use the .dt accessor to operate on the values. Series.tz_localize operates on the index.

In [12]: s.dt.tz_localize('US/Eastern')
Out[12]: 
0          2016-05-03 13:30:58-04:00
1   2016-05-03 13:30:58.100000-04:00
2   2016-05-03 13:30:58.200000-04:00
dtype: datetime64[ns, US/Eastern]
https://en.xdnf.cn/q/73201.html

Related Q&A

Slicing a circle in equal segments, Python

I have a set of close of 10,000 points on the sky. They are plotted using the RA (right ascension) and DEC (declination) on the sky. When plotted, they take the shape of a circle. What I would like to …

Pyautogui screenshot. Where does it go? How to save and find later?

I am learning from Al Sweigarts you tube video for automating the boring stuff. I got to the part of taking screenshots. He didnt really explain in his video so I tested things out. I found that it tak…

How to get pip to point to newer version of Python

I have two versions of Python installed on my centOS server. [ethan@demo ~]$ python2.6 --version Python 2.6.6 [ehtan@demo ~]$ python --version Python 2.7.3The older version (2.6) is required by some es…

Connect JS client with Python server

Im relatively new to JS and Python, so this is probably a beginners question. Im trying to send a string from a JS client to Python Server (and then send the string to another Python client).This is my…

Pip does not acknowledge Cython

I just installed pip and Python via home-brew on a fresh Mac OS installation.First of all, my pip is not installing dependencies at all - which forces me to re-run pip install tables 3 times and every …

Is it me or is pygame.key.get_pressed() not working?

okay, so I am making a basic space-ship game.I cant get rotation to work because it scrambles the bitmap, but thats for another question.Should I even use a gif? any other filetype suggestions?back t…

Find out if a python script is running in IDLE or terminal/command prompt

Is there a way to find out if the python script is running in the IDLE interpreter or the terminal?Works cross-platform if possible, or if needed a different way for each platform.Work with Python 2 a…

How to map coordinates in AxesImage to coordinates in saved image file?

I use matplotlib to display a matrix of numbers as an image, attach labels along the axes, and save the plot to a PNG file. For the purpose of creating an HTML image map, I need to know the pixel coor…

Why am I getting IOError: [Errno 13] Permission denied?

I am creating Log file for the code but I am getting the following error :[Tue Jun 11 17:22:59 2013] [error] [client 127.0.0.1] import mainLCF [Tue Jun 11 17:22:59 2013] [error] [client 127.0.0.1] …

Difference between bytearray and list

What is the difference between bytearray and for example, a list or tuple?As the name suggests, bytearray must be an array that carries byte objects. In python, it seems that bytes and str are treate…