SFTP File System Backed by S3

These are instructions to set up SFTP file server backed by S3 and to restrict user access to their own bucket. There are pros and cons of storing your data in S3. For some use-cases it might make sense and for others it might not. Selecting appropriate storage is important for your file system to function properly. For example, one of the variable that matters is how frequently you will be updating the files e.t.c. Consider appropriate storage based on your needs.
  1. Launch EC2 Instance using Amazon Linux AMI and download the private key. (You can use any other OS, following instructions might vary depending on OS)
  2. The private key will be in ‘.pem’ format. If using windows, convert private key into ‘.ppk’ format using PuttyGen.
  3. Using PuttyGen, Generate Public key against the same private key and save on your file system. We will need it later.
  4. SSH into ec2-instance using putty. The username for Linux AMI is ‘ec2-user’.
  5. Create user which will be used to access SFTP server. 
    sudo adduser testuser
  6. Create appropriate directories for this new user and save public key in '.ssh' directory. You can distribute your private key to users who need access to SFTP server and they will use this key to authenticate. Here, I am using same key pair which I used while launching ec2-instance. A better approach might be to generate another set of key pair and distribute this new private key to users who will be using SFTP server.
    cd ../testuser
mkdir .ssh
chmod 700 .ssh
touch .ssh/authorized_keys
chmod 600 .ssh/authorized_keys
  7. Save public key (which we generated as part of Step 3) in this file. 
    nano .ssh/authorized_keys
  8. Switch back to the root account and open the sshd config file. 
    sudo nano /etc/ssh/sshd_config
    Find following statement and comment it.
    Subsystem sftp /usr/libexec/openssh/sftp-server
    And add following statement. 
    Subsystem sftp internal-sftp
    Also add the following block at the end of file. 
    Match User testuser
    ChrootDirectory %h
    ForceCommand internal-sftp
    AllowTcpForwarding no
  9. Restart the service. 
    sudo service sshd restart
  10. Create directory as place holder for bucket.
    sudo mkdir /home/testuser/data
sudo chown ec2-user:testuser /home/testuser/data

  11. Install s3fu plugin to mount bucket. Please see this github page of plugin for more information.
    sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse
./autogen.sh
./configure
make 
sudo make install
echo yourAccessKey:yourSecretKey > /etc/passwd-s3fs
chmod 600 /etc/passwd-s3fs
mkdir /home/testuser/data/<yourBucket>
echo s3fs#<yourBucket> /home/testclient/data/testsqsfundrecs fuse _netdev,rw,nosuid,nodev,allow_other 0 0 >> /etc/fstab 
mount -a
Troubleshooting:
  1. Make sure that access key and secret key have access to bucket which you are trying to access
  2. Permission and Ownership of testuser directories are appropriately set (for authentication and for syncing bucket data)

Comments

Post a comment

Popular posts from this blog

Practice Questions - AWS Solutions Architect - Associate Certification

These are some of the practice questions from the e-book which is available on Amazon : 1.        Name of S3bucket must be unique: a)        Within your AWS Account b)       Across all your AWS Accounts c)        Within a region d)       Globally across all AWS Accounts 2.        Which of following can be used as storage class for S3: (Select all that apply) a)        S3 - Standard Infrequent b)       Amazon Glacier c)        Storage Gateway d)       EBS e)       Dynamo DB 3.        S3 replicates all objects: a)        On multiple devices within availability zone b)       In multiple availability zones within a region c)        Across multiple regions to achieve higher durability and availability d)       On a single device 4.        You have successfully launched an ec2 instance in VPC and have also downloaded the Private key which you will use to access instance via SSH. What else do you need to do to access the instance? a)        Add
Read more

AWS Parameter Store

A typical web application needs credentials to access different resources such as credentials to connect to database and tokens to communicate with other web services. It is common practice to pass these secret parameters to applications via system properties or environment variables. For example, if you are using Elastic Beanstalk for java web application then you can pass parameters (database url, username, password etc.) as properties.  Every infrastructure is different, but in general this practice is neither secure nor manageable. Some of the common problems are: parameters are not encrypted, parameters might be available in plain text on ec2-instance ebs scripts, parameters are hard to rotate if parameters are being shared by multiple applications. Moreover, if credentials are shared by different applications and multiple people are responsible for deployment then all those people will need access to credentials. Instead of passing secret parameters as environment variab
Read more

Continuous Integration using AWS CodePipeline (GitHub to Elastic BeanStalk)

AWS CodePipeline can be used to continuously deploy code changes to Elastic Beanstalk in Staging Environment. Here is an example which makes use of amazon web services to deploy code changes for Grails Project. CodePipeline will get code from GitHub repository, build using CodeBuild and then deploy the war file to Elastic Beanstalk. AWS CodeBuild uses buildspec.yml to get build instructions. Add buildspec.yml at root folder of your project. Since Grails uses gradle to build the project, the content of file can be: version : 0.1 phases: install: commands: - echo Install started on ` date ` pre_build: commands: - echo Pre Build started on ` date ` build: commands: - echo Build started on ` date ` - . / gradlew assemble post_build: commands: - echo Build completed on ` date ` artifacts: files : - build / libs / * .war The root folder should also contain gradle wrapper (gradlew) . We won't have t
Read more