AWS: Increase Connection Timeout

Environment: Elastic Beanstalk - Java with Tomcat

Recently, we faced a situation where java-tomcat server was taking more than 60 seconds to process the request. And we were getting Bad Gateway 5xx Errors. For this particular use case, we could not use asynchronous communication protocol such as Queues; And business users were more than happy to wait for requests to complete even if it takes more than a minute.

For every request, Classic Load Balancer maintains 2 connections. One connection is from client to load balancer, and another connection from load balancer to ec2-instance. By default, idle timeout of classic load balancer is set to 60 seconds which means it can close connection if data is not sent within 60 seconds. In order to handle requests which takes more than 60 seconds to process, we can edit the idle timeout setting and increase it to any value upto 4000 seconds. However, If we want to handle requests which takes more than 60 seconds to process, then Load Balancer Idle time out is not the only setting that we need to modify. We also need to modify time out setting for Apache as Elastic Beanstalk uses Apache as Proxy Server.

1. Log into ec2 instance and find httpd.conf file at following path:
   

   nano /etc/httpd/conf/httpd.conf 

   Change Timeout setting to x seconds where x > 60. If keep Alive is on, then make sure KeepAliveTimeOut is greater than Timeout. Restart Apache server:

   sudo service httpd restart

2. Log into ec2 management console and change idle timeout setting of classic load balancer to x seconds. Now test web server with requests which were failing because of timeout. These requests should be fine now.
3. If you are happy with timeout settings, then download the httpd conf file which was modified earlier. And make this conf file part of war file in .ebextensions folder. This way, with every new deployment, elastic beanstalk will replace default httpd conf with modified file, found in war, when processing pre-hook commands. Project Structure can look similar to this: (where .ebextensions is at root of project)
   .ebextensions
   --httpd
   ----conf
   ------httpd.conf
   --files.config

References:
Stack Overflow
AWS Docs
AWS Forum