So my requirement was simple. Deploy AWS Lambda from a Windows PC such that once a file is uploaded to a bucket, an SFTP put is etablished to a remote server.
Some background. While trying to do this entirely in my webapp, I immediately ran into the problem of the transfer time exceeding the allowable wait time for an HTTP request by Heroku (30 seconds).
The other problem was that since I was transferring the file immediately to S3, reliably getting a handle on the file and its name was proving difficult as there was no hook that wouldn’t require significant code
Lastly, although the only reason why I was even considering this was because I had relatively few users, I knew it was very poor design. I naturally would have run this aspect – uploading file via FTP, using a redis-backed queue.
So, I knew what AWS lambda was. I had read about how devs were using it, and I knew it was a very viable option right now. I was already uploading files to s3, why not push the files using AWS lambda via SFTP to the remote servers?
After writing my script, I immediately deployed my code, but then immediately ran into an error; something longer but which read like
cannot import name '_bcrypt'
After reading back and forth, I discovered it was due to the disparity between how python packages are installed on linux vs windows. Lambda runs on a linux based AMI, and the deployment packages had to originate from same.
So essentially, you cannot prepare and deploy your AWS Lambda function from a windows PC.
So here’s how I went about getting it done
- I logged into a debian 8 server running on linode. Did an update and upgrade.
- I ran
sudo apt-get install zip build-essential libffi-dev python3-dev to install the crypto and dev libs
- Because I planned to use the Python3.6 lambda runtime, I had to ensure I was using Python 3.6. This version of debian comes shipped with Python2.7.9 (as python) and Python3.4.2 (as python3).
- To get around this quickly, I installed pyenv, then ran
pyenv install 3.6.0
- With pyenv installed, run
pyenv local 3.6.0 and to confirm you’re running python 3.6.0, run python –version.
- Create a new virtualenv with
python -m venv myownenv
- Then install all the packages you want. In my case, I needed pysftp.
- Once done, navigate into /lib/python3.6/site-packages
- Once inside site-packages, run
zip -R ~/s3_to_sftp.zip . (please note the “.”. using . copies all files, including hidden files.)
- In my case, I did not have need to do the same for /lib64
The zip file s3_to_sftp is ready to be deployed. Remember to copy in your function script(s) to the zip file.