SFTP File System Backed by S3

These are instructions to set up SFTP file server backed by S3 and to restrict user access to their own bucket. There are pros and cons of storing your data in S3. For some use-cases it might make sense and for others it might not. Selecting appropriate storage is important for your file system to function properly. For example, one of the variable that matters is how frequently you will be updating the files e.t.c. Consider appropriate storage based on your needs.
  1. Launch EC2 Instance using Amazon Linux AMI and download the private key. (You can use any other OS, following instructions might vary depending on OS)
  2. The private key will be in ‘.pem’ format. If using windows, convert private key into ‘.ppk’ format using PuttyGen.
  3. Using PuttyGen, Generate Public key against the same private key and save on your file system. We will need it later.
  4. SSH into ec2-instance using putty. The username for Linux AMI is ‘ec2-user’.
  5. Create user which will be used to access SFTP server. 
    sudo adduser testuser
  6. Create appropriate directories for this new user and save public key in '.ssh' directory. You can distribute your private key to users who need access to SFTP server and they will use this key to authenticate. Here, I am using same key pair which I used while launching ec2-instance. A better approach might be to generate another set of key pair and distribute this new private key to users who will be using SFTP server.
    cd ../testuser
mkdir .ssh
chmod 700 .ssh
touch .ssh/authorized_keys
chmod 600 .ssh/authorized_keys
  7. Save public key (which we generated as part of Step 3) in this file. 
    nano .ssh/authorized_keys
  8. Switch back to the root account and open the sshd config file. 
    sudo nano /etc/ssh/sshd_config
    Find following statement and comment it.
    Subsystem sftp /usr/libexec/openssh/sftp-server
    And add following statement. 
    Subsystem sftp internal-sftp
    Also add the following block at the end of file. 
    Match User testuser
    ChrootDirectory %h
    ForceCommand internal-sftp
    AllowTcpForwarding no
  9. Restart the service. 
    sudo service sshd restart
  10. Create directory as place holder for bucket.
    sudo mkdir /home/testuser/data
sudo chown ec2-user:testuser /home/testuser/data

  11. Install s3fu plugin to mount bucket. Please see this github page of plugin for more information.
    sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse
./autogen.sh
./configure
make 
sudo make install
echo yourAccessKey:yourSecretKey > /etc/passwd-s3fs
chmod 600 /etc/passwd-s3fs
mkdir /home/testuser/data/<yourBucket>
echo s3fs#<yourBucket> /home/testclient/data/testsqsfundrecs fuse _netdev,rw,nosuid,nodev,allow_other 0 0 >> /etc/fstab 
mount -a
Troubleshooting:
  1. Make sure that access key and secret key have access to bucket which you are trying to access
  2. Permission and Ownership of testuser directories are appropriately set (for authentication and for syncing bucket data)

Comments

Post a comment

Popular posts from this blog

Practice Questions - AWS Solutions Architect - Associate Certification

These are some of the practice questions from the e-book which is available on Amazon : 1.        Name of S3bucket must be unique: a)        Within your AWS Account b)       Across all your AWS Accounts c)        Within a region d)       Globally across all AWS Accounts 2.        Which of following can be used as storage class for S3: (Select all that apply) a)        S3 - Standard Infrequent b)       Amazon Glacier c)        Storage Gateway d)       EBS e)       Dynamo DB 3.        S3 replicates all objects: a)        On multiple devices within availability zone b)       In multiple availability zones within a region c)        Across multiple regions to achieve higher durability and availability d)       On a single device 4.        You have successfully launched an ec2 instance in VPC and have also downloaded the Private key which you will use to access instance via SSH. What else do you need to do to access the instance? a)        Add
Read more

AWS: Increase Connection Timeout

Environment: Elastic Beanstalk - Java with Tomcat Recently, we faced a situation where java-tomcat server was taking more than 60 seconds to process the request. And we were getting Bad Gateway 5xx Errors. For this particular use case, we could not use asynchronous communication protocol such as Queues; And business users were more than happy to wait for requests to complete even if it takes more than a minute. For every request, Classic Load Balancer maintains 2 connections. One connection is from client to load balancer, and another connection from load balancer to ec2-instance. By default, idle timeout of classic load balancer is set to 60 seconds which means it can close connection if data is not sent within 60 seconds. In order to handle requests which takes more than 60 seconds to process, we can edit the idle timeout setting and increase it to any value upto 4000 seconds . However, If we want to handle requests which takes more than 60 seconds to process, then Load Balancer
Read more

AWS Parameter Store

A typical web application needs credentials to access different resources such as credentials to connect to database and tokens to communicate with other web services. It is common practice to pass these secret parameters to applications via system properties or environment variables. For example, if you are using Elastic Beanstalk for java web application then you can pass parameters (database url, username, password etc.) as properties.  Every infrastructure is different, but in general this practice is neither secure nor manageable. Some of the common problems are: parameters are not encrypted, parameters might be available in plain text on ec2-instance ebs scripts, parameters are hard to rotate if parameters are being shared by multiple applications. Moreover, if credentials are shared by different applications and multiple people are responsible for deployment then all those people will need access to credentials. Instead of passing secret parameters as environment variab
Read more