Agile FAQs
  About   Slides   Home  

 
Managed Chaos
Naresh Jain's Random Thoughts on Software Development and Adventure Sports
     
`
 
RSS Feed
Recent Thoughts
Tags
Recent Comments

Upstream Connection Time Out Error in Nginx

Currently at Industrial Logic we use Nginx as a reverse proxy to our Tomcat web server cluster.

Today, while running a particular report with large dataset, we started getting timeouts errors. When we looked at the Nginx error.log, we found the following error:

[error] 26649#0: *9155803 upstream timed out (110: Connection timed out) 
while reading response header from upstream, 
client: xxx.xxx.xxx.xxx, server: elearning.industriallogic.com, request: 
"GET our_url HTTP/1.1", upstream: "internal_server_url", 
host: "elearning.industriallogic.com", referrer: "requested_url"

After digging around for a while, I discovered that our web server is taking more than 60 secs to respond. Nginx has a directive called proxy_read_timeout which defaults to 60 secs. It determines how long nginx will wait to get the response to a request.

In nginx.conf file, setting proxy_read_timeout to 120 secs solved our problem.

server {
    listen       80;
    server_name  elearning.industriallogic.com;
    server_name_in_redirect off;
    port_in_redirect        off;
 
    location / {
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Real-Host  $host;
        proxy_read_timeout 120;
        ...
    }
    ...
}
  • Ovidiu

    ok, but taking 60 seconds is abnormally long. any other solution or did you just stick with this solution?

    • http://blogs.agilefaqs.com Naresh Jain

      This worked for us and there seem to be no apparent problems so far. Hence never bothered to find another other solution.

      • Ovidiu

        Thanks, may I ask which site is the one so I can access it and see how the performance feels like please?

        • http://blogs.agilefaqs.com Naresh Jain

          The proxy_read_timeout property basically tells nginx how long it should wait to hear back from the downstream servers before it times out. So I don’t see why there would be any performance issues. Only for a few requests that take longer, nginx is going to wait for the specified timeout; for rest of the requests it should respond in few milli-seconds.

          • Ovidiu

            Ah, I understand now. Will give it a try.
            My issue looked like this:

            I run a very small VPS with only one website, all perfectly working. Now I moved a second site onto the VPS which has been inactive for 2 years, updated everything, put it online and all of a sudden I was facing the same time-outs you mentioned here in this post. None of the two sites has much traffic… The problem was, all of a sudden, none of the two sites was working at all. Only getting these timeouts in nginx’s log file :-(Have implemented proxy_read_timeout 120; and will monitor it :-)thanks for posting this here.

  • Ovidiu

    doesn’t help me at all. al I get are these errors in my nginx logs:

    2012/07/28 14:18:31 [error] 24210#0: *1601 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 62.75.146.19, server: quilombobrasil.com, request: “GET /nginx_status HTTP/1.1″, upstream: “fastcgi://127.0.0.1:9000″, host: “62.75.146.19”

    I can’t even do a /etc/init.d/php-cgi restart – that command simply hangs.
    If I do a killall -9 php-cgi followed by a /etc/init.d/php-cgi restart the two websites work for a few hours than the same problem happens again :-( 

    any other pointers?

  • j

    you are my life saver, thanks

  • recarv

    Setting this two a long time for the entire HTTP stack is probably casting the net a bit too wide. Most responses should happen within a few seconds. I’d suggest watching the logs (as was done) and making separate location blocks for each so you handle poor performance by exception, otherwise this is a DDoS issue.


    Licensed under
Creative Commons License