AWS: Increase Connection Timeout

Environment: Elastic Beanstalk - Java with Tomcat

Recently, we faced a situation where java-tomcat server was taking more than 60 seconds to process the request. And we were getting Bad Gateway 5xx Errors. For this particular use case, we could not use asynchronous communication protocol such as Queues; And business users were more than happy to wait for requests to complete even if it takes more than a minute.

For every request, Classic Load Balancer maintains 2 connections. One connection is from client to load balancer, and another connection from load balancer to ec2-instance. By default, idle timeout of classic load balancer is set to 60 seconds which means it can close connection if data is not sent within 60 seconds. In order to handle requests which takes more than 60 seconds to process, we can edit the idle timeout setting and increase it to any value upto 4000 seconds. However, If we want to handle requests which takes more than 60 seconds to process, then Load Balancer Idle time out is not the only setting that we need to modify. We also need to modify time out setting for Apache as Elastic Beanstalk uses Apache as Proxy Server.
  1. Log into ec2 instance and find httpd.conf file at following path: 
    nano /etc/httpd/conf/httpd.conf
    Change Timeout setting to x seconds where x > 60. If keep Alive is on, then make sure KeepAliveTimeOut is greater than Timeout. Restart Apache server: 
    sudo service httpd restart
  2. Log into ec2 management console and change idle timeout setting of classic load balancer to x seconds. Now test web server with requests which were failing because of timeout. These requests should be fine now.
  3. If you are happy with timeout settings, then download the httpd conf file which was modified earlier. And make this conf file part of war file in .ebextensions folder. This way, with every new deployment, elastic beanstalk will replace default httpd conf with modified file, found in war, when processing pre-hook commands. Project Structure can look similar to this: (where .ebextensions is at root of project)
    .ebextensions
    --httpd
    ----conf
    ------httpd.conf
    --files.config
References:
Stack Overflow
AWS Docs
AWS Forum

Comments

  1. alez14 December 2019 at 15:05

    Thanks a lot for sharing
    Having good health is what most people out there wants but can not achieve. some people takes buy ibogaine online AND buy weed online to get it.

    ReplyDelete

Post a Comment

Popular posts from this blog

Practice Questions - AWS Solutions Architect - Associate Certification

These are some of the practice questions from the e-book which is available on Amazon:
1.Name of S3bucket must be unique: a)Within your AWS Account b)Across all your AWS Accounts c)Within a region d)Globally across all AWS Accounts
2.Which of following can be used as storage class for S3: (Select all that apply) a)S3 - Standard Infrequent b)Amazon Glacier c)Storage Gateway d)EBS e)Dynamo DB
3.S3 replicates all objects: a)On multiple devices within availability zone b)In multiple availability zones within a region c)Across multiple regions to achieve higher durability and availability d)On a single device
Read more

Continuous Integration using AWS CodePipeline (GitHub to Elastic BeanStalk)

AWS CodePipeline can be used to continuously deploy code changes to Elastic Beanstalk in Staging Environment. Here is an example which makes use of amazon web services to deploy code changes for Grails Project. CodePipeline will get code from GitHub repository, build using CodeBuild and then deploy the war file to Elastic Beanstalk. AWS CodeBuild uses buildspec.yml to get build instructions. Add buildspec.yml at root folder of your project. Since Grails uses gradle to build the project, the content of file can be: version: 0.1 phases: install: commands: - echo Install started on `date` pre_build: commands: - echo Pre Build started on `date` build: commands: - echo Build started on `date` - ./gradlew assemble post_build: commands: - echo Build completed on `date` artifacts: files: - build/libs/*.war The root folder should also contain gradle wrapper (gradlew). We won't have to install gradle on build server …
Read more

AWS Cross-Account RDS Backups

It is hard to deny importance of cross-region copies of RDS snapshots. Recovery will be a very difficult process, if not entirely impossible, in case of a disaster as small as un-availability of AWS region. However, even cross-region copies of backups are not sufficient. What if AWS Account credentials gets compromised? Or some employee goes rogue and deletes snapshots? These scenarios are not unheard of [1]. In any case, it is good idea to have off-site backups. Backup is not a backup if it is not in a completely separate location. Over AWS, these off-site backups can be snapshots stored in entirely different AWS Account. Unfortunately, AWS does not have service which you can use to create and store backups in different account. But it is trivial to set it up yourself using scripts or Lambdas. Creating manual snapshots and saving these snapshots in completely different AWS account will ensure data recovery in majority of disaster scenarios. And this process can be automated easily u…
Read more