SFTP File System Backed by S3

These are instructions to set up SFTP file server backed by S3 and to restrict user access to their own bucket. There are pros and cons of storing your data in S3. For some use-cases it might make sense and for others it might not. Selecting appropriate storage is important for your file system to function properly. For example, one of the variable that matters is how frequently you will be updating the files e.t.c. Consider appropriate storage based on your needs.
  1. Launch EC2 Instance using Amazon Linux AMI and download the private key. (You can use any other OS, following instructions might vary depending on OS)
  2. The private key will be in ‘.pem’ format. If using windows, convert private key into ‘.ppk’ format using PuttyGen.
  3. Using PuttyGen, Generate Public key against the same private key and save on your file system. We will need it later.
  4. SSH into ec2-instance using putty. The username for Linux AMI is ‘ec2-user’.
  5. Create user which will be used to access SFTP server.
    sudo adduser testuser 
  6. Create appropriate directories for this new user and save public key in '.ssh' directory. You can distribute your private key to users who need access to SFTP server and they will use this key to authenticate. Here, I am using same key pair which I used while launching ec2-instance. A better approach might be to generate another set of key pair and distribute this new private key to users who will be using SFTP server.
    cd ../testuser
    mkdir .ssh
    chmod 700 .ssh
    touch .ssh/authorized_keys
    chmod 600 .ssh/authorized_keys
    
  7. Save public key (which we generated as part of Step 3) in this file.
    nano .ssh/authorized_keys
  8. Switch back to the root account and open the sshd config file.
    sudo nano /etc/ssh/sshd_config
    Find following statement and comment it.
    Subsystem sftp /usr/libexec/openssh/sftp-server
    And add following statement.
    Subsystem sftp internal-sftp
    Also add the following block at the end of file.
    Match User testuser
        ChrootDirectory %h
        ForceCommand internal-sftp
        AllowTcpForwarding no
    
  9. Restart the service.
    sudo service sshd restart
  10. Create directory as place holder for bucket.
    sudo mkdir /home/testuser/data
    sudo chown ec2-user:testuser /home/testuser/data
    

  11. Install s3fu plugin to mount bucket. Please see this github page of plugin for more information.
    sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
    git clone https://github.com/s3fs-fuse/s3fs-fuse.git
    cd s3fs-fuse
    ./autogen.sh
    ./configure
    make 
    sudo make install
    echo yourAccessKey:yourSecretKey > /etc/passwd-s3fs
    chmod 600 /etc/passwd-s3fs
    mkdir /home/testuser/data/<yourBucket>
    echo s3fs#<yourBucket> /home/testclient/data/testsqsfundrecs fuse _netdev,rw,nosuid,nodev,allow_other 0 0 >> /etc/fstab 
    mount -a 
    
Troubleshooting:
  1. Make sure that access key and secret key have access to bucket which you are trying to access
  2. Permission and Ownership of testuser directories are appropriately set (for authentication and for syncing bucket data)

Comments

Popular posts from this blog

Practice Questions - AWS Solutions Architect - Associate Certification

Continuous Integration using AWS CodePipeline (GitHub to Elastic BeanStalk)

AWS Parameter Store