Using an NGINX reverse proxy behind an ELB in AWS
If you want to use an NGINX reverse proxy behind an ELB (elastic load balancer) in AWS, you need a few extra tricks in order for it to work as expected. In my experience, none of this information seems to be documented all in the same place. This information is the sum of both research as well as trial and error, whether it be my own or others. It's my hope that this article will help others looking to accomplish something similar.
Forwarding real IP's to NGINX (and rate limiting)
This isn't technically required, but I would assume you'd want real IP's in your NGINX logs. Further, if you intend to use NGINX for rate limiting, you'll need this setup otherwise NGINX will only log (and hence use for rate limiting) the IP that it sees from your ELB, since the ELB is technically the client connecting to NGINX and not the real client.
In your main http block (probably the bulk of your nginx.conf file) add:
real_ip_header X-Forwarded-For;
In addition, you need to set set_real_ip_from to the IP of your ELB, or a CIDR range. If you're using Elastic Beanstalk or ECS and doing auto scaling, you probably want to pick a CIDR address range that covers possibilities for IP's that may be assigned to your ELB.
set_real_ip_from 172.31.0.0/16;
And in your location block make sure you have:
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
Now that that's covered, rate limiting should also work as intended per the NGINX documentation, except for one additional caveat. If rate limiting actually kicks in, you will start to see errors that look like this in your NGINX logs:
epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream
This is again because the ELB is actually the client in this case, and it's the ELB that is keeping connections open to your NGINX. This can be safely disabled (since you're behind an ELB) using this line:
proxy_ignore_client_abort on;
Now rate limiting should work properly and you'll see real IP's in your NGINX logs behind an ELB.
HTTPS to NGINX behind an ELB
If you're using NGINX itself to do HTTPS/TLS termination (or say, letsencrypt) then you can ignore this section and configure NGINX exactly like the other hundreds of guides and blog articles throughout the internet suggest you do. However, if you're using Amazon Certificate Manager to do HTTPS/TLS termination on your ELB, you need some additional tricks up your sleeve to get everything to work as expected.
ELB health checks only come through via HTTP and will not do HTTPS at all. If you configure a standard 301 redirect to HTTPS in NGINX behind an ELB, your health check will fail which will take your instance out of service and then be completely non-functional. Here's the workaround: in your main location block, use an if statement to do a rewrite. Make sure you have server_name set, or write out the whole hostname instead.
if ($http_x_forwarded_proto != 'https') {
return 301 https://$server_name$request_uri;
}
I know what you may be thinking - someone told you not to use if statements in NGINX, didn't they? Well, in this case it's perfectly acceptable and safe to do so. This is probably the best way to tackle this problem. I'm not saying this is the only way, but this is probably the best way.
After you do this, make a separate location block for your health check path that doesn't include an if statement like this. Also include a proxy_redirect off; statement alongside your proxy_pass. By separating your health check into a different location block without the check, your ELB health check will pass through without being redirected to HTTPS and failing.
I'm sure there are other things you will want to do to tune your particular setup, but they are more than likely things that are specific to your own infrastructure. This should get you well on your way to a functional NGINX reverse proxy behind an Amazon ELB. If you have any questions about this or your own setup, signup for an account here on Steemit and ask away.