Deploy AWS Lambda from a Windows PC

So my requirement was simple. Deploy AWS Lambda from a Windows PC such that once a file is uploaded to a bucket, an SFTP put is etablished to a remote server.

Some background. While trying to do this entirely in my webapp, I immediately ran into the problem of the transfer time exceeding the allowable wait time for an HTTP request by Heroku (30 seconds).

The other problem was that since I was transferring the file immediately to S3, reliably getting a handle on the file and its name was proving difficult as there was no hook that wouldn’t require significant code

Lastly, although the only reason why I was even considering this was because I had relatively few users, I knew it was very poor design. I naturally would have run this aspect – uploading file via FTP, using a redis-backed queue.

So, I knew what AWS lambda was. I had read about how devs were using it, and I knew it was a very viable option right now. I was already uploading files to s3, why not push the files using AWS lambda via SFTP to the remote servers?

After writing my script, I immediately deployed my code, but then immediately ran into an error; something longer but which read like cannot import name '_bcrypt'

After reading back and forth, I discovered it was due to the disparity between how python packages are installed on linux vs windows. Lambda runs on a linux based AMI, and the deployment packages had to originate from same.

So essentially, you cannot prepare and deploy your AWS Lambda function from a windows PC.

So here’s how I went about getting it done

  1. I logged into a debian 8 server running on linode. Did an update and upgrade.
  2. I ran sudo apt-get install zip build-essential libffi-dev python3-dev to install the crypto and dev libs
  3. Because I planned to use the Python3.6 lambda runtime, I had to ensure I was using Python 3.6. This version of debian comes shipped with Python2.7.9 (as python) and Python3.4.2 (as python3).
  4. To get around this quickly, I installed pyenv, then ran pyenv install 3.6.0
  5. With pyenv installed, run pyenv local 3.6.0 and to confirm you’re running python 3.6.0, run python –version.
  6. Create a new virtualenv with python -m venv myownenv
  7. Then install all the packages you want. In my case, I needed pysftp.
  8. Once done, navigate into /lib/python3.6/site-packages
  9. Once inside site-packages, run zip -R ~/s3_to_sftp.zip . (please note the “.”. using . copies all files, including hidden files.)
  10. In my case, I did not have need to do the same for /lib64

The zip file s3_to_sftp is ready to be deployed. Remember to copy in your function script(s) to the zip file.

 

References