How to install Numpy and Pandas for AWS Lambdas?

2024/9/8 8:50:36

Problem: I wanted to use Numpy and Pandas in my AWS lambda function. I am working on Windows 10 with PyCharm. My function compiles and works fine on local machine, however, as soon as package it up and deploy on AWS, it breaks down giving errors in importing the numpy and pandas packages. I tried reinstalling both packages and then redeploying however, error remained the same.

StackOverFlow Solutions: Other people are having similar issues and fellow users have suggested that this is mainly compatibility issue, because Python libraries are compiled on Windows whereas, AWS Lambda runs on linux machines.

Question: What's the best way to create a deployment package for AWS on windows 10? Is there a way I can specify targeted platform while installing packages through PIP. Apparently there is an option in pip with tag --platform but I cannot figure out how to use it. Any helps?

Answer

Like often there is more than one way to come to a solution.

The preferred way imho is to use AWS lambda layers, because it separates the functional code from the dependencies. The basics are explained here.

  1. Get all your dependencies. Like you mentioned correctly, pandas and numpy have to be compiled for the AMI Linux. This can be done with the tool: "serverless python requirements" or with a docker container based on this image. A more detailed instruction can be found here.
  2. Put the dependencies in a folder called python.
  3. zip the whole folder e.g. with the preinstalled windows zipping tool.
  4. Upload the zip file to AWS as a layer: Go to AWS Lambda, from the left choose Layers and "Create a new layer".
  5. After you saved the layer, go to your Lambda Function and choose "Layers". Click "Add a layer" choose your newly created layer and click on save. Now your function should not get import errors anymore.
https://en.xdnf.cn/q/73309.html

Related Q&A

vim highlighting everything in red

I added a print line to a python script while the script was executing, and now all the text is highlighted in red when I open the file. Opening and closing the file doesnt get rid of it. Opening a sec…

Using python modules in node.js

Is it possible to create a glue that makes it possible for python modules (more specifically, library bindings) to be used in node.js? Some of the data structures can be directly mapped to V8 objects …

Using PHP to call Virtualenv’ed Python Script

Last night I spent 5.5 hours trying make PHP execute and receive the output of Virtualenv’ed Python script. Nothing worked; except for scripts that were not Virtualenv’ed.What I am trying to do:I am …

How do I find my program name?

ProblemI am unable to write to a different log than the default one using syslog. I am unsure if maybe my app name is wrong in my configuration. Do "program name" and "process name"…

Is it possible to automate the execution of a Python script using Microsoft Flow?

I want to execute a snippet of python code based on some trigger using Microsoft-Flow. Is there a way to do this? Basically I am exploring on Powerapps and Microsoft-Flow. I have data in powerapp, I c…

Python: Selecting numbers with associated probabilities [duplicate]

This question already has answers here:Closed 13 years ago.Possible Duplicates:Random weighted choiceGenerate random numbers with a given (numerical) distribution I have a list of list which contains …

Weights and Biases: Login and network errors

I recently installed Weights and Biases (wandb) for recording the metrics of my machine learning projects. Everything worked fine when connected to wandb cloud instance or when I used a local docker im…

Python: Opening a file without creating a lock

Im trying to create a script in Python to back up some files. But, these files could be renamed or deleted at any time. I dont want my script to prevent that by locking the file; the file should be abl…

Tensorflow dataset questions about .shuffle, .batch and .repeat

I had a question about the use of batch, repeat and shuffle with tf.Dataset.It is not clear to me exactly how repeat and shuffle are used. I understand that .batch will dictate how many training exampl…

How to sort in python with multiple conditions?

I have a list with sublists as follows:result = [ [helo, 10], [bye, 50], [yeah, 5], [candy,30] ]I want to sort this with three conditions: first, by highrest integer in index 2 of sublist, then by leng…