saving csv file to s3 using boto3

2024/11/15 20:34:35

I am trying to write and save a CSV file to a specific folder in s3 (exist). this is my code:

from io import BytesIO
import pandas as pd
import boto3
s3 = boto3.resource('s3')d = {'col1': [1, 2], 'col2': [3, 4]}
df = pd.DataFrame(data=d)csv_buffer = BytesIO()bucket = 'bucketName/folder/'
filename = "test3.csv"
df.to_csv(csv_buffer)
content = csv_buffer.getvalue()def to_s3(bucket,filename,content):s3.Object(bucket,filename).put(Body=content)to_s3(bucket,filename,content)

this is the error that I get:

Invalid bucket name "bucketName/folder/": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$"

I also tried :

bucket = bucketName/folder

and:

bucket = bucketName
key = folder/
s3.Object(bucket,key,filename).put(Body=content)

Any suggestions?

Answer

Saving into s3 buckets can be also done with upload_file with an existing .csv file:

import boto3
s3 = boto3.resource('s3')bucket = 'bucket_name'
filename = 'file_name.csv'
s3.meta.client.upload_file(Filename = filename, Bucket= bucket, Key = filename)
https://en.xdnf.cn/q/71844.html

Related Q&A

httplib2, how to set more than one cookie?

As you are probably aware, more often than not, an HTTP server will send more than just a session_id cookie; however, httplib2 handles cookies with a dictionary, like this:response, content = http.requ…

FTP upload file works manually, but fails using Python ftplib

I installed vsFTP in a Debian box. When manually upload file using ftp command, its ok. i.e, the following session works:john@myhost:~$ ftp xxx.xxx.xxx.xxx 5111 Connected to xxx.xxx.xxx.xxx. 220 Hello,…

Baktracking function which calculates change exceeds maximum recursion depth

Im trying to write a function that finds all possible combinations of coins that yield a specified amount, for example it calculates all possible way to give change for the amount 2 British pounds from…

How to interface a NumPy complex array with C function using ctypes?

I have a function in C that takes an array of complex floats and does calculations on them in-place.:/* foo.c */ void foo(cmplx_float* array, int length) {...}The complex float struct looks like this:t…

How to access predefined environment variables in conda environment.yml?

I wish to share an environment.yml file for others to reproduce the same setup as I have. The code we use depends on the environment variable $PWD. I wish to set a new env variable in the environment.y…

Python enclosing scope variables with lambda function

I wrote this simple code:def makelist():L = []for i in range(5):L.append(lambda x: i**x)return Lok, now I callmylist = makelist()because the enclosing scope variable is looked up when the nested functi…

Overloading + to support tuples

Id like to be able to write something like this in python:a = (1, 2) b = (3, 4) c = a + b # c would be (4, 6) d = 3 * b # d would be (9, 12)I realize that you can overload operators to work with custom…

Extracting particular text associated value from an image

I have an image, and from the image I want to extract key and value pair details.As an example, I want to extract the value of "MASTER-AIRWAYBILL NO:" I have written to extract the entire te…

Installing pip in Pycharm 2016.3

I upgraded to the new version of Pycharm. In the terminal, it says bash-3.2$ instead of my username. When I tried to install a library, it said that pip command is not found:bash: pip: command not foun…

How to store real-time chat messages in database?

I am using mysqldb for my database currently, and I need to integrate a messaging feature that is in real-time. The chat demo that Tornado provides does not implement a database, (whereas the blog does…