Viewing Category: Tomcat  [clear category selection]

Using Nginx to Modify System Monitoring Requests

In a previous post, Reducing Unnecessary Tomcat Sessions, I described using a Valve to group stateless requests into a single session. It functions by parsing the User-Agent HTTP header and client IP address. If the incoming request does not contain a User-Agent, as is the case with some server monitoring agents that simply open TCP 80 and throw some characters that appear similar to an HTTP request the connection, the valve will function as intended. This is true of Server Nanny, which we use to monitor Microsoft Windows server health.

Fortunately, the web application being monitored is behind an Nginx broker proxy. We can create a virtual URL for the agent to use, such as /ServerNannyCheck, and modify any requests to that location so that they pass through to Tomcat with a proper User-Agent header. The following is an abbreviated Nginx configuration file:

upstream appserver {     server 10.0.0.10:8080 max_fails=2 fail_timeout=10s;     server 10.0.0.11:8080 backup; }   server {     listen 80;     server_name app.internal.network;       set $ua $http_user_agent;       location ~ ^/ServerNannyCheck {         set $ua "ServerNanny (nginx)";         rewrite ^ /app/index.cfm/HealthReport last;     }       location ~ ^/app/ {         proxy_pass http://appserver;         proxy_redirect / /;         proxy_set_header Host $host;         proxy_set_header User-Agent $ua;     } }

The configuration defines appserver as representing a primary or failover application server. The variable named $ua is created to hold the value of original User-Agent header, say "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:13.0) Gecko/20100101 Firefox/13.0.1" in the case of a normal browser request. If the URI is /ServerNannyCheck, Nginx will reassign a static value of "ServerNanny (nginx)" to the variable. In the next line, Nginx rewrites the request to a hit a real resource on the application server. The last location block performs the proxy for all application requests, adding the determined value for the User-Agent. Tomcat now receives a proper header and does its Crawler Session Manager Valve magic.

A very similar technique can be used to pass the actual client IP address to the application server in the HTTP header of your choice. I like to use X-Forwarded-Client-IP to pass the value of $remote_addr. I send many other headers containing information that only the broker knows, and that are very useful to have at the actual application server.

Reducing Unnecessary Tomcat Sessions

We have an internal application (Tomcat 7.0.x, Railo 3.3.x) that is configured with very long session timeout — 10 hours, in fact. This enables a user to login first thing in the morning and not have to worry about losing any session data, even after long periods of inactivity. However, we also have monitoring systems that check the application health every few minutes. Similarly, we have scheduled tasks that run throughout the day. In both of these cases, the HTTP user agents make a single connection, receive a response, and do not save any state between requests, i.e. cookies. This presents a bit of a problem; our application server will allocate a new session for each of these stateless connections and leave it in memory for 10 hours. Consider that one health check occurs every 3 minutes. The minimum number of sessions created before they start expiring is 200 (10 × 60 ÷ 3). That's not an insignificant amount of memory wasted. It would be ideal if we could expire those abandoned sessions sooner, or better yet, recycle a single session for use with these stateless user agents.

Tomcat has a solution. The Crawler Session Manager Valve will effectively gather requests made by the same user agent from the same IP address into a single session. Enabling the valve is as simple as adding the following into the conf/server.xml file:

<Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true">     <Valve className="org.apache.catalina.valves.CrawlerSessionManagerValve" crawlerUserAgents=".*ServerNanny.*|.*CFSCHEDULE.*" sessionInactiveInterval="600"/> </Host>

To verify that the valve is working as expected, make a few requests of the server:

for I in `jot - 1 100`; do curl -s -j -o /dev/null -A 'ServerNanny' 'http://tomcat:8080/app/'; done

Then open the Tomcat Manager and view your application's sessions. Even though there were 100 stateless requests thrown at the server, they should have been forced into a single session. One of the attributes within the session will be org.apache.catalina.valves.CrawlerSessionManagerValve. For example, here's what I see when view the session details:

Tomcat Session Details

In contrast, opening the same URL with Firefox, configured to ignore cookies, will result in a new session being created for each request. You can see the CFML application attribute created, and the serialized display of the session scope, which might be useful if you want to snoop on the data in this shared session.

Some things to keep mind

  • If access to your application server made only through a broker proxy, you'll want to use the Remote IP Valve to pass accurate client IP addresses to the Crawler Session Manager Valve. I'll discuss this in another post.
  • If the client does not send a User-Agent HTTP header, as is the case with Server Nanny, you'll need to insert a custom header into the request so it can be identified by the regex pattern in the valve configuration. I'll describe how I satisfied this requirement in another post.
  • You will probably want to have your application set username information in the session so they stand out as authenticated sessions in the Tomcat Manager list of sessions. You could even detect the crawlers and insert a fake username to make the shared session obvious.

Railo Multi-web on Tomcat

Both Sean Corfield and Jamie Krug have written about configuring Railo for multiple web sites/contexts long ago. However, having just done this setup myself, I thought I'd take notes and share them. I created a Google Document called (surprisingly enough) Railo Multi-web on Tomcat. I'm going to add to it as I have gather more information.

Painlessly Updating Open BlueDragon

I recently wrote a modest shell script to make updating an installation of Open BlueDragon less cumbersome. I reported an issue with a nightly build, and I wanted to be able to switch between build versions reliably and quickly. The script, available as openbluedragon-nightly-updater.sh, will download and archive the nightly build and then deploy to web application server configured. My development environment is Apache Tomcat on Mac OS X 10.5/10.6. I create a directory in $HOME called Servers into which I download tarball distributions of Tomcat. I then create a symbolic link to the current version as ~/Servers/Tomcat. I use the ROOT context for simplicity. I keep old nightly builds in my ~/Downloads directory. Therefore, the configuration in the update script looks like this:

NIGHTLY="OpenBlueDragon-Nightly-`date +%Y-%m-%d`.zip" SERVER="$HOME/Servers/Tomcat" DEPLOY="$SERVER/webapps" CONTEXT="ROOT" URL="http://www.openbluedragon.org/download/nightly/openbd.war" ARCHIVE="$HOME/Downloads/$NIGHTLY"

The script has two options: (-m) minimal or full update, and (-f) local file or remote download. With the minimal update, it just replaces the $DEPLOY/$CONTEXT/WEB-INF/lib/OpenBlueDragon.jar file. The full update replaces all of the Open BlueDragon JAR files, as well as the administration application and manual. It does not, however delete the existing Open BlueDragon configuration in $DEPLOY/$CONTEXT/WEB-INF/bluedragon. This is important to me because I don't want to lose the current settings for datasources, mail, debugging, and whatnot. I also want to keep all the symbolic links to my CFML applications in $DEPLOY/$CONTEXT in tact.

While working on a problem today, I wanted to verify that the issue wasn't introduced with last night's build. I just ran the following commands to revert to last week's build:

~/Servers/Tomcat/bin/catalina.sh stop ~/Workspace/admin/coldfusion/openbluedragon-nightly-updater.sh -m -f \     ~/Downloads/OpenBlueDragon-Nightly-2010-06-12.zip ~/Servers/Tomcat/bin/catalina.sh start

In less time than it takes to get another Diet Coke, the installation of Open BlueDragon was running an arbitrary version. It turned out that the issue I was troubleshooting had nothing to do with Open BlueDragon -- it was an error in my MXUnit test case. To get running again on the latest version, I ran the script without any arguments.

Configuring a Production Open BlueDragon Server

I've just finished building up a couple production servers to host web applications. The servers are Xen guests on an AMD Quad-Core Opteron x86_64 host. The VPS template is a minimal installation of CentOS, to which I added packages as needed. The release of Sun Java 1.6u12 came out just as I was writing this, so these instructions will need to get updated slightly when JPackage has a new RPM (more on that later). Both Matt Woodward and Dave Shuck recently wrote about configuring CFML engines with Tomcat. The installation I'll describe is somewhat similar.

  • CentOS 5.2
  • Tomcat 5.5.23 (tomcat5-5.5.23-0jpp.7.el5_2.1)
  • Apache 2.2 (httpd-2.2.3-11.el5_2.centos.4)
  • Sun Java 1.6u11 (java-1.6.0-sun-1.6.0.11-1jpp)
  • Sun JavaMail 1.4.1
  • Open BlueDragon 1.0.1

The installation of packages using yum is a snap, however there was an issue with the architecture detection. There is a simple workaround, to hard-code i386 as the basearch:

sed -i -r 's/\$basearch/i386/g' /etc/yum.repos.d/CentOS-Base.repo

The procedure is to install jpackage-utils, then download and repackage the Sun Java SE Development Kit 6 (jdk 1.6) using the JPackage Project non-free nosrc RPM. I install some, but not all of the, resulting RPMs:

yum --nogpgcheck localinstall java-1.6.0-sun-1.6.0.11-1jpp.i586.rpm java-1.6.0-sun-devel-* java-1.6.0-sun-fonts-*

The CentOS Wiki has a thorough article on installing Java on CentOS. I've considered using OpenJDK, but I don't know what sort of compatibility issues that would raise.

The Tomcat server starts up just fine with GNU's version of the Java runtime (libgcj and java-1.4.2-gcj-compat). However, using the GNU version of JavaMail (classpathx-mail) instead of Sun JavaMail, the following chunk of CFML will fail with a javax.mail.NoSuchProviderException exception from within the Open BlueDragon web application:

<cfscript> server = "localhost"; port = 25; username = ""; password = ""; mailSession = createObject("java", "javax.mail.Session").getDefaultInstance(createObject("java", "java.util.Properties").init()); transport = mailSession.getTransport("smtp"); transport.connect(server, JavaCast("int", port), username, password); transport.close(); </cfscript>

Open BlueDragon does include include the correct Jar, but the JVM that Tomcat configures loads the system version first. Rather that muck about with the classpaths, I downloaded the current version of JavaMail, extracted mail.jar, and created alternatives link:

unzip -j -d /tmp javamail-1_4_1.zip javamail-1.4.1/mail.jar mv /tmp/mail.jar /usr/share/java/javamail-1.4.1.jar alternatives --install /usr/share/java/javamail.jar javamail /usr/share/java/javamail-1.4.1.jar 5000 alternatives --auto javamail file /var/lib/tomcat5/common/lib/\[javamail\].jar

Tomcat installs a set of symlinks to /usr/share/tomcat5. Configuration files are placed in /etc/tomcat5. For this installation, I use a stripped-down version of server.xml that provides web application hosting on a per-user basis.

<Server port="8005" shutdown="SHUTDOWN"> <GlobalNamingResources /> <Service name="Catalina"> <Connector port="8080" address="127.0.0.1" protocol="HTTP/1.1" /> <Connector port="8009" address="127.0.0.1" protocol="AJP/1.3" /> <Engine name="Catalina" defaultHost="localhost"> <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true" debug="0" /> <Host name="localhost-username" appBase="/home/username/webapps" unpackWARs="false" autoDeploy="false" debug="1"> <Context path="" docBase="openbd" allowLinking="true" caseSensitive="true" swallowOutput="true" /> </Host> </Engine> </Service> </Server>

The standard Tomcat configuration has a single Host within an Engine named Catalina. I've added a second Host that is specific to a system user username, which allows each user on the system to manage their own deployed web applications and choose their own root Context. Installing Open BlueDragon as the default web application simplifies the Apache HTTP configuration.

The username user has an Apache HTTP configuration file in /etc/httpd/conf.d/username.conf with mod_rewrite rules to proxy all requests for CFML files to the Tomcat HTTP Connector. I had intended to use the AJP Connector with mod_proxy_ajp, but there is a problem with the the proxy request not specifying the proper hostname. There might be a solution to that issue, but I haven't found it yet. The plain mod_proxy_http module works properly in the following configuration:

<VirtualHost *:80> DocumentRoot /home/username/websites/sitename ... RewriteCond %{SCRIPT_FILENAME} \.cfm$ RewriteRule ^/(.*)$ http://localhost-username:8080/$1 [P] </VirtualHost>

The rest of the Apache HTTP configuration handles web requests for flat files, served from ~/websites/sitename. The CFML files can be placed in ~/webapps/openbd, however an easier deployment is to place everything in ~/websites/sitename (like you would with a typical ColdFusion server). Symbolic links can be added for directories containing CFML. Consider the following:

cd ~/webapps/openbd ln -s ../../websites/sitename/MachII MachII

It would probably be a good idea to set the Open BlueDragon root mapping appropriately. There are a few issues with file ownership and permissions that I didn't address above. I've added username to the /etc/sudoers file, granting that user limited access.

Tomcat Monitoring and Startup Via Cron

Over the last week, my virtual private server needed to be restarted a couple times. Once, I was there to see the restart, and manually run the script to bring Tomcat up. However, another time I wasn't. Since I run Tomcat from a plain user account, it doesn't start up with the server itself using the SysV-style init scripts from /etc/init.d. Many years ago, I created a cronjob to check on an Eggdrop IRC bot that sometimes died or went haywire. The same solution works fine for Tomcat. The following shell script (re)starts Tomcat, if needed. It searches for all the processes with the command name java. Any found processes are output with a user-defined format that includes just two fields, and no header. The next command in the pipeline filters out lines that do not start with the appropriate username -- the one that kicked off the cronjob. Any lines making it to the second grep are matched for the Tomcat class name. Lastly, wc counts up the number of lines, which should accurately specify the number of Tomcat instances started by this user. Currently, there aren't any other user accounts that would start an instance of Tomcat, but it's best to prepare for the possibility that a different user account will run Tomcat.

#!/bin/sh export CATALINA_HOME=$HOME/server/tomcat export JRE_HOME=$HOME/java/jre/default PROCS=`/bin/ps -C java -o euser=,args= | grep -E "^$USER" | grep -o -F org.apache.catalina.startup.Bootstrap | wc -l` case $PROCS in 0) echo "Tomcat does not appear to be running. Starting Tomcat..." $CATALINA_HOME/bin/catalina.sh start exit 1 ;; 1) exit 0 ;; *) echo "More than one Tomcat appears to be running. Restarting Tomcat..." $CATALINA_HOME/bin/catalina.sh stop && $CATALINA_HOME/bin/catalina.sh start exit 2 ;; esac

The crontab for the plain user account will run the script above every 5 minutes, which seems pretty reasonable.

0-59/5 * * * * $HOME/bin/check-tomcat.sh

While working on this VPS, I decided to update the JRE to Java 6 update 10 because I've heard that some operations, such as CFC instantiation, are faster. It seems faster, but I don't have any actual performance data to prove it.