How to store Dataframe data to Firebase Storage?

2024/11/18 12:44:30

Given a pandas Dataframe which contains some data, what is the best to store this data to Firebase?

Should I convert the Dataframe to a local file (e.g. .csv, .txt) and then upload it on Firebase Storage, or is it also possible to directly store the pandas Dataframe without conversion? Or are there better best practices?

Update 01/03 - So far I've come with this solution, which requires writing a csv file locally, then reading it in and uploading it and then deleting the local file. I doubt however that this is the most efficient method, thus I would like to know if it can be done better and quicker?

import os
import firebase_admin
from firebase_admin import db, storagecred   = firebase_admin.credentials.Certificate(cert_json)
app    = firebase_admin.initialize_app(cred, config)
bucket = storage.bucket(app=app)def upload_df(df, data_id):"""Upload a Dataframe as a csv to Firebase Storage:return: storage_ref"""# Storage location + extensionstorage_ref = data_id + ".csv"# Store locallydf.to_csv(data_id)# Upload to Firebase Storageblob    = bucket.blob(storage_ref)with open(data_id,'rb') as local_file:blob.upload_from_file(local_file)# Delete locallyos.remove(data_id)return storage_ref
Answer

With python-firebase and to_dict:

postdata = my_df.to_dict()# Assumes any auth/headers you need are already taken care of.
result = firebase.post('/my_endpoint', postdata, {'print': 'pretty'})
print(result)
# Snapshot info

You can get the data back using the snapshot info and endpoint, and reestablish the df with from_dict(). You could adapt this solution to SQL and JSON solutions, which pandas also has support for.

Alternatively and depending on where you script executes from, you might consider treating firebase as a db and using the dbapi from firebase_admin (check this out.)

As for whether it's according to best practice, it's difficult to say without knowing anything about your use case.

https://en.xdnf.cn/q/71060.html

Related Q&A

Multiple characters in Python ord function

Programming beginner here. (Python 2.7)Is there a work around for using more than a single character for Pythons ord function?For example, I have a hex string \xff\x1a which Id like the decimal value …

Get minimum x and y from 2D numpy array of points

Given a numpy 2D array of points, aka 3D array with size of the 3rd dimension equals to 2, how do I get the minimum x and y coordinate over all points? Examples:First:I edited my original example, sin…

Extract Text with its Font Details (Style,Size,color,Italic etc) from a PDF in Python [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic…

How to keep track of status with multiprocessing and pool.map?

Im setting up a multiprocessing module for the first time, and basically, I am planning to do something along the lines offrom multiprocessing import pool pool = Pool(processes=102) results = pool.map(…

How to get time 17:00:00 today or yesterday?

If 17:00:00 today is already passed, then it should be todays date, otherwise - yesterdays. Todays time I get with:test = datetime.datetime.now().replace(hour=17,minute=0,second=0,microsecond=0)But I d…

PyMongo Aggregate how to get executionStats

I am trying to get executionStats of a Particular mongo aggregate query. I run db.command but that doesnt give "execution status"This is what I am trying to do. how to get Python Mongo Aggreg…

Is it possible to do parallel reads on one h5py file using multiprocessing?

I am trying to speed up the process of reading chunks (load them into RAM memory) out of a h5py dataset file. Right now I try to do this via the multiprocessing library. pool = mp.Pool(NUM_PROCESSES) g…

Where is a django validator functions return value stored?

In my django app, this is my validator.py from django.core.exceptions import ValidationError from django.core.validators import URLValidatordef validate_url(value):url_validator = URLValidator()url_inv…

Modifying YAML using ruamel.yaml adds extra new lines

I need to add an extra value to an existing key in a YAML file. Following is the code Im using.with open(yaml_in_path, r) as f:doc, ind, bsi = load_yaml_guess_indent(f, preserve_quotes=True) doc[phase1…

How to get the background color of a button or label (QPushButton, QLabel) in PyQt

I am quite new to PyQt. Does anyone tell me how to get the background color of a button or label (QPushButton, QLabel) in PyQt.