AWS: Increase Connection Timeout

Environment: Elastic Beanstalk - Java with Tomcat

Recently, we faced a situation where java-tomcat server was taking more than 60 seconds to process the request. And we were getting Bad Gateway 5xx Errors. For this particular use case, we could not use asynchronous communication protocol such as Queues; And business users were more than happy to wait for requests to complete even if it takes more than a minute.

For every request, Classic Load Balancer maintains 2 connections. One connection is from client to load balancer, and another connection from load balancer to ec2-instance. By default, idle timeout of classic load balancer is set to 60 seconds which means it can close connection if data is not sent within 60 seconds. In order to handle requests which takes more than 60 seconds to process, we can edit the idle timeout setting and increase it to any value upto 4000 seconds. However, If we want to handle requests which takes more than 60 seconds to process, then Load Balancer Idle time out is not the only setting that we need to modify. We also need to modify time out setting for Apache as Elastic Beanstalk uses Apache as Proxy Server.
  1. Log into ec2 instance and find httpd.conf file at following path: 
    nano /etc/httpd/conf/httpd.conf
    Change Timeout setting to x seconds where x > 60. If keep Alive is on, then make sure KeepAliveTimeOut is greater than Timeout. Restart Apache server: 
    sudo service httpd restart
  2. Log into ec2 management console and change idle timeout setting of classic load balancer to x seconds. Now test web server with requests which were failing because of timeout. These requests should be fine now.
  3. If you are happy with timeout settings, then download the httpd conf file which was modified earlier. And make this conf file part of war file in .ebextensions folder. This way, with every new deployment, elastic beanstalk will replace default httpd conf with modified file, found in war, when processing pre-hook commands. Project Structure can look similar to this: (where .ebextensions is at root of project)
    .ebextensions
    --httpd
    ----conf
    ------httpd.conf
    --files.config
References:
Stack Overflow
AWS Docs
AWS Forum

Comments

Post a comment

Popular posts from this blog

Practice Questions - AWS Solutions Architect - Associate Certification

These are some of the practice questions from the e-book which is available on Amazon:
1.Name of S3bucket must be unique: a)Within your AWS Account b)Across all your AWS Accounts c)Within a region d)Globally across all AWS Accounts
2.Which of following can be used as storage class for S3: (Select all that apply) a)S3 - Standard Infrequent b)Amazon Glacier c)Storage Gateway d)EBS e)Dynamo DB
3.S3 replicates all objects: a)On multiple devices within availability zone b)In multiple availability zones within a region c)Across multiple regions to achieve higher durability and availability d)On a single device
Read more

Continuous Integration using AWS CodePipeline (GitHub to Elastic BeanStalk)

AWS CodePipeline can be used to continuously deploy code changes to Elastic Beanstalk in Staging Environment. Here is an example which makes use of amazon web services to deploy code changes for Grails Project. CodePipeline will get code from GitHub repository, build using CodeBuild and then deploy the war file to Elastic Beanstalk. AWS CodeBuild uses buildspec.yml to get build instructions. Add buildspec.yml at root folder of your project. Since Grails uses gradle to build the project, the content of file can be: version: 0.1 phases: install: commands: - echo Install started on `date` pre_build: commands: - echo Pre Build started on `date` build: commands: - echo Build started on `date` - ./gradlew assemble post_build: commands: - echo Build completed on `date` artifacts: files: - build/libs/*.war The root folder should also contain gradle wrapper (gradlew). We won't have to install gradle on build server …
Read more

AWS Parameter Store

A typical web application needs credentials to access different resources such as credentials to connect to database and tokens to communicate with other web services.
It is common practice to pass these secret parameters to applications via system properties or environment variables. For example, if you are using Elastic Beanstalk for java web application then you can pass parameters (database url, username, password etc.) as properties.  Every infrastructure is different, but in general this practice is neither secure nor manageable. Some of the common problems are: parameters are not encrypted, parameters might be available in plain text on ec2-instance ebs scripts, parameters are hard to rotate if parameters are being shared by multiple applications. Moreover, if credentials are shared by different applications and multiple people are responsible for deployment then all those people will need access to credentials.
Instead of passing secret parameters as environment variables, se…
Read more