Viewing Category: Nginx  [clear category selection]

GeoIP with NGINX for CFML Apps

I've been working with the GeoIP module for NGINX. The instructions on the wiki are well written and it's not difficult to compile with the module enabled. There are two instances of Tomcat hosting deployed versions of the Automaton App on my test machine. On the NGINX machine, the client IP (or the value from X-Forwarded-For; more on that in a bit) will be used as a key to fetch data from the GeoIP data files. Here are the configuration lines inside the location that handles proxying all CFML requests for this application:

proxy_set_header X-NGINX-Server-IP $server_addr; proxy_set_header X-NGINX-Server-Port $server_port; proxy_set_header X-NGINX-Server-Scheme $scheme; proxy_set_header X-NGINX-Client-IP $remote_addr; proxy_set_header X-NGINX-GeoIP-Country $geoip_city_country_code; proxy_set_header X-NGINX-GeoIP-Region-Code $geoip_region; proxy_set_header X-NGINX-GeoIP-Region-Name $geoip_region_name; proxy_set_header X-NGINX-GeoIP-City $geoip_city; proxy_set_header X-NGINX-GeoIP-Coords "$geoip_latitude $geoip_longitude"; proxy_set_header X-NGINX-GeoIP-DMA-Code $geoip_dma_code; proxy_set_header X-NGINX-TC-Protocol $scheme; proxy_set_header X-NGINX-TC-Remote-IP $remote_addr;

In the onRequestStart method of Application.cfc, it uses an instance of ConnectionInspector.cfc to grab all the connection data, including the geolocation HTTP headers. The application could do anything with the information: display different content for requests coming from different countries, make a guess about how far away the user is, set Google AdWords API parameters, and so forth.

During testing, however, it can be difficult to simulate connections from all over the world. There's an easy solution. The following are the configuration directives in the NGINX http block:

geoip_country /usr/share/GeoIP/GeoIP.dat; geoip_city /usr/share/GeoIP/GeoLiteCity.dat; geoip_proxy; geoip_proxy_recursive on;

The geoip_proxy directives allows a trusted forwarding proxy server to specify the client's address, rather than using the actual TCP connection. Setting it to allows my workstation, upon which is Charles Proxy and Firefox, to make requests sending a forged X-Forwarded-For header. In the following screenshot, you can see that I used the rewriting feature to peg the header value at

On the server machine, I can see the following written out to the console because I have debug logging enabled in my ColdBox.cfc for the Headers.cfc Interceptor:

DEBUG interceptors.Headers Header accept-encoding=gzip, deflate DEBUG interceptors.Headers Header accept-language=en-US,en;q=0.5 DEBUG interceptors.Headers Header accept=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 DEBUG interceptors.Headers Header cache-control=max-age=0 DEBUG interceptors.Headers Header DEBUG interceptors.Headers Header user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0 DEBUG interceptors.Headers Header x-forwarded-for= DEBUG interceptors.Headers Header x-nginx-client-ip= DEBUG interceptors.Headers Header x-nginx-geoip-city=Reykjavik DEBUG interceptors.Headers Header x-nginx-geoip-coords=64.1500 -21.9500 DEBUG interceptors.Headers Header x-nginx-geoip-country=IS DEBUG interceptors.Headers Header x-nginx-geoip-dma-code=0 DEBUG interceptors.Headers Header x-nginx-geoip-region-code=10 DEBUG interceptors.Headers Header x-nginx-geoip-region-name=Gullbringusysla DEBUG interceptors.Headers Header DEBUG interceptors.Headers Header x-nginx-server-ip= DEBUG interceptors.Headers Header x-nginx-server-port=80 DEBUG interceptors.Headers Header x-nginx-server-scheme=http DEBUG interceptors.Headers Header x-nginx-tc-protocol=http

In addition to the normal headers, we have a bunch containing the GeoIP data. On a side note, notice how the Tomcat RemoteIPValve striped out the X-NGINX-TC-Remote-IP header as part of its execution to set the servlet context client IP address. I looked for a configuration option to prevent that, but so far I don't have a good solution other than sending duplicate headers. Technically, the X-NGINX-TC-Protocol header doesn't need to be duplicated, but I did both for consistency. At any rate, the application now knows that the request came from Reykjavik, Gullbringusysla in Iceland and can display it in the footer of the site:

If you too are experimenting geolocation and need some test IP addresses, you'll find the list I composed useful:         Fuzhou, Fujian (CN)       Alcorcón, Madrid (ES)    Shenyang, Liaoning (CN)       Vadodara, Gujarat (IN)      Sunnyvale, California (US)     Toronto, Ontario (CA)      Most, Ustecky kraj (CZ)      Bandar, Qazvin (IR)       Reykjavík, Gullbringusysla (IS)       Halifax, Calderdale (GB)       Jindalee, New South Wales (AU)      Fuji, Shizuoka (JP)       Guangzhou, Guangdong (CN)       Zabrze, Slaskie (PL)      Chapecó, Santa Catarina (BR)     Moscow, Moscow City (RU)        Bogotá, Distrito Especial (CO)   Porirua, Wellington (NZ)        Hürriyet, Ordu (TR)

Blocking Access by IP in NGINX

There are several ways to restrict access to a resource served by NGINX based on the client IP address. If you want to block every user agent, except some private addresses for administration, just use the access directives:

server {   location /railo-context/admin {     rewrite ^ /private$uri;   }   location /private {     internal;     allow;     allow;     deny all;     rewrite ^/private(.*)$ /$1 break;     # proxy request to Railo server   } }

However, if you need to block a batch of addresses, say the IPs you've notice used by script kiddies, you could use the map module. This is handy if your binary doesn't have the geo module available, which will be discussed later.

map $remote_addr $denied {   default 0; 1; 1; 1; 1; } server {   location / {     rewrite ^ /public/index.cfm$uri last;   }   location /public {     internal;     if ($denied) {       return 444;     }     rewrite ^/public(.*)$ /$1 break;     # proxy request to Railo server   } }

The configuration above will use the map module to resolve the $denied variable when needed. If matched, the special NGINX return code 444 will close the TCP socket and prevent writing to the logs. You could return 403 to keep track of the incoming request and return a forbidden response.

Lastly, if you have the geo module available, you can write matching statements using CIDR address notation, like this (replacing the map block with a geo block):

geo $denied {   default 0; 1; # TimeWarner/RoadRunner cable modems 1; # VirginMedia cable modems }

Note how the $remote_addr first argument is optional. If you needed to match on a different value, say the value from a header sent from front-end load balancer, you would specify the value to match first:

geo $lb_client_ip $denied {   default 0; 1; # TimeWarner/RoadRunner cable modems 1; # VirginMedia cable modems }

The configuration above assumes that the $lb_client_ip variable is set somewhere inside the server block, like so:

set $lb_client_ip $remote_addr; if ($http_x_forwarded_client_ip ~ "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}") {   set $lb_client_ip $http_x_forwarded_client_ip; }

You could also use the real IP module to copy the header value into $remote_addr, if that's available to you.

NGINX on Windows: Sending Signals

Trying to send a reload signal to NGINX running in a command prompt started with elevated privileges doesn't work:

C:\nginx>nginx.exe -s reload nginx: [error] OpenEvent("Global\ngx_reload_6620") failed (5: Access is denied)

Even though the DOS box was started as the administrator, NGINX can't or won't respond to the signal. We could use Sysinternals PsExec to run as the same user that started the master process, which in this case was NT AUTHORITY\SYSTEM. The -s flag specifies that the command should be run as local system account.

C:\nginx>"C:\Program Files\SysinternalsSuite\PsExec.exe" -s C:\nginx\nginx.exe -s reload   PsExec v1.98 - Execute processes remotely Copyright (C) 2001-2010 Mark Russinovich Sysinternals -     nginx: [alert] could not open error log file: CreateFile() "logs/error.log" failed (3: The system cannot find the path specified) 2013/05/31 10:09:06 [emerg] 5804#7952: CreateFile() "C:\Windows\system32/conf/nginx.conf" failed (3: The system cannot find the path specified) C:\nginx\nginx.exe exited on BITCHIN_CAMARO with error code 1.

Oh, bummer. Our current working directory isn't passed along to the process via PsExec. We either need to set the working directory or pass the prefix path to NGINX. Here's an example setting the working directory:

C:\nginx>"C:\Program Files\SysinternalsSuite\PsExec.exe" -s -w C:\nginx C:\nginx\nginx.exe -s reload   PsExec v1.98 - Execute processes remotely Copyright (C) 2001-2010 Mark Russinovich Sysinternals -     C:\nginx\nginx.exe exited on BITCHIN_CAMARO with error code 0.

It worked. The configuration was reloaded without needing to restart the service. Cool. Here's the same result using the prefix argument:

C:\nginx>"C:\Program Files\SysinternalsSuite\PsExec.exe" -s C:\nginx\nginx.exe -p C:\nginx -s reload   PsExec v1.98 - Execute processes remotely Copyright (C) 2001-2010 Mark Russinovich Sysinternals -     C:\nginx\nginx.exe exited on BITCHIN_CAMARO with error code 0.

And remember, don't forget to run nginx.exe -t to test the configuration before attempting to send any signals.

Using Nginx to Modify System Monitoring Requests

In a previous post, Reducing Unnecessary Tomcat Sessions, I described using a Valve to group stateless requests into a single session. It functions by parsing the User-Agent HTTP header and client IP address. If the incoming request does not contain a User-Agent, as is the case with some server monitoring agents that simply open TCP 80 and throw some characters that appear similar to an HTTP request the connection, the valve will function as intended. This is true of Server Nanny, which we use to monitor Microsoft Windows server health.

Fortunately, the web application being monitored is behind an Nginx broker proxy. We can create a virtual URL for the agent to use, such as /ServerNannyCheck, and modify any requests to that location so that they pass through to Tomcat with a proper User-Agent header. The following is an abbreviated Nginx configuration file:

upstream appserver {     server max_fails=2 fail_timeout=10s;     server backup; }   server {     listen 80;     server_name;       set $ua $http_user_agent;       location ~ ^/ServerNannyCheck {         set $ua "ServerNanny (nginx)";         rewrite ^ /app/index.cfm/HealthReport last;     }       location ~ ^/app/ {         proxy_pass http://appserver;         proxy_redirect / /;         proxy_set_header Host $host;         proxy_set_header User-Agent $ua;     } }

The configuration defines appserver as representing a primary or failover application server. The variable named $ua is created to hold the value of original User-Agent header, say "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:13.0) Gecko/20100101 Firefox/13.0.1" in the case of a normal browser request. If the URI is /ServerNannyCheck, Nginx will reassign a static value of "ServerNanny (nginx)" to the variable. In the next line, Nginx rewrites the request to a hit a real resource on the application server. The last location block performs the proxy for all application requests, adding the determined value for the User-Agent. Tomcat now receives a proper header and does its Crawler Session Manager Valve magic.

A very similar technique can be used to pass the actual client IP address to the application server in the HTTP header of your choice. I like to use X-Forwarded-Client-IP to pass the value of $remote_addr. I send many other headers containing information that only the broker knows, and that are very useful to have at the actual application server.