Deploy AWS Lambda from a Windows PC

So my requirement was simple. Deploy AWS Lambda from a Windows PC such that once a file is uploaded to a bucket, an SFTP put is etablished to a remote server.

Some background. While trying to do this entirely in my webapp, I immediately ran into the problem of the transfer time exceeding the allowable wait time for an HTTP request by Heroku (30 seconds).

The other problem was that since I was transferring the file immediately to S3, reliably getting a handle on the file and its name was proving difficult as there was no hook that wouldn’t require significant code

Lastly, although the only reason why I was even considering this was because I had relatively few users, I knew it was very poor design. I naturally would have run this aspect – uploading file via FTP, using a redis-backed queue.

So, I knew what AWS lambda was. I had read about how devs were using it, and I knew it was a very viable option right now. I was already uploading files to s3, why not push the files using AWS lambda via SFTP to the remote servers?

After writing my script, I immediately deployed my code, but then immediately ran into an error; something longer but which read like cannot import name '_bcrypt'

After reading back and forth, I discovered it was due to the disparity between how python packages are installed on linux vs windows. Lambda runs on a linux based AMI, and the deployment packages had to originate from same.

So essentially, you cannot prepare and deploy your AWS Lambda function from a windows PC.

So here’s how I went about getting it done

1. I logged into a debian 8 server running on linode. Did an update and upgrade.
2. I ran sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev to install the crypto and dev libs.
3. Because I planned to use the Python3.6 lambda runtime, I had to ensure I was using Python 3.6. This version of debian comes shipped with Python2.7.9 (as python) and Python3.4.2 (as python3).
4. To get around this quickly, I installed pyenv, then ran pyenv install 3.6.0
5. With pyenv installed, run pyenv local 3.6.0 and to confirm you’re running python 3.6.0, run python –version.
6. Create a new virtualenv with python -m venv myownenv
7. Then install all the packages you want. In my case, I needed pysftp.
8. Once done, navigate into /lib/python3.6/site-packages
9. Once inside site-packages, run zip -R ~/s3_to_sftp.zip . (please note the “.”. using . copies all files, including hidden files.)
10. In my case, I did not have need to do the same for /lib64

The zip file s3_to_sftp is ready to be deployed. Remember to copy in your function script(s) to the zip file.

References

• http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example-deployment-pkg.html#with-s3-example-deployment-pkg-python
• https://pythonvibes.wordpress.com/2016/12/09/ftp-and-sftp-through-lambda/comment-page-1/#comment-3
• https://github.com/pyenv/pyenv