SFTP File System Backed by S3
These are instructions to set up SFTP file server backed by S3 and to restrict user access to their own bucket. There are pros and cons of storing your data in S3. For some use-cases it might make sense and for others it might not. Selecting appropriate storage is important for your file system to function properly. For example, one of the variable that matters is how frequently you will be updating the files e.t.c. Consider appropriate storage based on your needs.
- Launch EC2 Instance using Amazon Linux AMI and download the private key. (You can use any other OS, following instructions might vary depending on OS)
- The private key will be in ‘.pem’ format. If using windows, convert private key into ‘.ppk’ format using PuttyGen.
- Using PuttyGen, Generate Public key against the same private key and save on your file system. We will need it later.
- SSH into ec2-instance using putty. The username for Linux AMI is ‘ec2-user’.
- Create user which will be used to access SFTP server.
sudo adduser testuser
- Create appropriate directories for this new user and save public key in '.ssh' directory. You can distribute your private key to users who need access to SFTP server and they will use this key to authenticate. Here, I am using same key pair which I used while launching ec2-instance. A better approach might be to generate another set of key pair and distribute this new private key to users who will be using SFTP server.
cd ../testuser mkdir .ssh chmod 700 .ssh touch .ssh/authorized_keys chmod 600 .ssh/authorized_keys
- Save public key (which we generated as part of Step 3) in this file.
- Switch back to the root account and open the sshd config file.
sudo nano /etc/ssh/sshd_configFind following statement and comment it.
Subsystem sftp /usr/libexec/openssh/sftp-serverAnd add following statement.
Subsystem sftp internal-sftpAlso add the following block at the end of file.
Match User testuser ChrootDirectory %h ForceCommand internal-sftp AllowTcpForwarding no
- Restart the service.
sudo service sshd restart
Create directory as place holder for bucket.
sudo mkdir /home/testuser/data sudo chown ec2-user:testuser /home/testuser/data
Install s3fu plugin to mount bucket. Please see this github page of plugin for more information.
sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel git clone https://github.com/s3fs-fuse/s3fs-fuse.git cd s3fs-fuse ./autogen.sh ./configure make sudo make install echo yourAccessKey:yourSecretKey > /etc/passwd-s3fs chmod 600 /etc/passwd-s3fs mkdir /home/testuser/data/<yourBucket> echo s3fs#<yourBucket> /home/testclient/data/testsqsfundrecs fuse _netdev,rw,nosuid,nodev,allow_other 0 0 >> /etc/fstab mount -a
- Make sure that access key and secret key have access to bucket which you are trying to access
- Permission and Ownership of testuser directories are appropriately set (for authentication and for syncing bucket data)