Override Tomcat Session Cookie

Tomcat uses HTTP cookie to track browser sessions. By default Tomcat 5.5 generates session cookies without an expiration date (Expires=...), like:

Set-Cookie: JSESSIONID=A39F8F3623D20EF9E66D309E298E87E0; Path=/

using Cookie.setMaxAge(-1).

Without an expiration date, this cookie should be deleted by the browser when it is closed, which is what IE7 does. Thus, even the session has a lifetime, say 12 hours, at the Tomcat side, if the browser was restarted, the session would be lost.

Firefox keeps the session cookie when it restarts.

I use the following code to override this behavior:

// after users log in
// HttpServletResponse response
response.setHeader("Set-Cookie", "JSESSIONID=" + request.getSession().getId()
+ "; Expires=" + getCookieExpiresFormat().formatByAge(age)
+ "; Path=/");

It generates something like

Set-Cookie: JSESSIONID=A39F8F3623D20EF9E66D309E298E87E0; Expires=Thu, 22-Apr-2010 20:07:56 GMT; Path=/

Thus the session can be kept live for any time period even when browser restarted.

Classloading of GridSphere

JSR-168 portlet applications developed for GridSphere 3.1 has a $CATALINA_HOME/webapps/$PORTLET_WEBAPP/WEB-INF/lib/gridsphere-portletservlet-3.1.jar, which contains a single org.gridsphere.provider.portlet.jsr.PortletServlet class. Originally I thought this class is intended to override the class with the same name in $CATALINA_HOME/shared/lib, according to the rules of Tomcat class loading, by some "clever" Java programmer. Later I found out this class is exactly the same class as in $CATALINA_HOME/shared/lib. So though it replaces the one in $CATALINA_HOME/shared/lib, but it changes nothing. In fact, it does change one thing: the defining class loader.

So if removing $CATALINA_HOME/webapps/$PORTLET_WEBAPP/WEB-INF/lib/gridsphere-portletservlet-3.1.jar, running GridSphere will raise such an exception:

16825:ERROR:(PortletServlet.java:loadJSRPortletWebapp:102)
Unable to create jsr portlet instance: org.gridsphere.gsexamples.portlets.HelloWorld

java.lang.ClassNotFoundException: org.gridsphere.gsexamples.portlets.HelloWorld
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:164)
at org.gridsphere.provider.portlet.jsr.PortletServlet.loadJSRPortletWebapp(PortletServlet.java:87)
at org.gridsphere.provider.portlet.jsr.PortletServlet.service(PortletServlet.java:182)

So basically it complains it can not find the portlet class. Let me explain why.

First org.gridsphere.provider.portlet.jsr.PortletServlet is defined in the portlet web.xml as the generic servlet. Not surprisingly, the portlet application is just a web servlet application in GridSphere.

Second, on line 87 of PortletServlet, there is
Portlet portletInstance = (Portlet) Class.forName(portletClass).newInstance();

It says to use the defining class loader of the current class, i.e., PortletServlet, to load the portlet class. So if PortletServlet is inside $CATALINA_HOME/shared/lib, then there is no way to find the portlet class. PortletServlet has to be put inside $CATALINA_HOME/webapps/$PORTLET_WEBAPP/WEB-INF/lib/.

I am not very convinced that this is a very elegant solution towards portlet classloading. Somehow I have a feeling that it might be better if the portal and portlets stay in the same web application, and the webapp classloader acts as the parent of portlet classloaders. Thus, at least, classloading of portlets is not affected by Tomcat classloading rules.

Use Security Manager to Control Java Web Application Behavior

In a Java web application server like Tomcat, we have the Tomcat container and multiple web applications running in the same JVM. It is thus very important to cut the unexpected interactions between applications and between applications and Tomcat. Ideally it is expected the behavior of one web application should not affect the behavior of other applications as well as Tomcat, in a bad way.

One of such unexpected interactions is caused by modifying shared classes and objects, for instance, changing system properties, changing system classes' behavior.

Tomcat 5.5's class loaders are organised as:

      Bootstrap
  |
System
  |
Common
/      \
Catalina   Shared
     /   \
Webapp1  Webapp2 ...

Tomcat 6.0's class loaders are organised as:

      Bootstrap
   |
System
   |
Common
/     \
Webapp1   Webapp2 ...

In both cases, web applications and Tomcat share the classes managed by class loaders Bootstrap, System and Common.

Let's say one web application use the following code to tell Java runtime to use the XSLT implementation shipped with JDK.

System.setProperty("javax.xml.transform.TransformerFactory", "com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl");

Other web applications may choose to use other XSLT implementations in a similar way. If a certain web application's correct behavior is depending on a particular XLST implementation, then we may get a problem, because System.setProperty("javax.xml.transform.TransformerFactory", ...) will change the JVM-wide XSLT implementation.

Therefore it is advisable to use Java security manager to control the permissions granted to web application code.

Note $CATALINA_HOME/bin/startup.sh does not start up a Tomcat with the security manager. To enable the security manager, use $CATALINA_HOME/bin/startup.sh -security. It will append "-Djava.security.manager -Djava.security.policy=..." to the JVM arguments.

A simple Java program testing security manager and policy file

public class SecurityManagerTest {
public static void main(String[] args) throws Exception {
System.setProperty("greeting", "hello world!");
System.out.println(System.getProperty("greeting"));
}
}

Run this program, you will see "hello world!" printed out.

Prepare a policy file called lab.policy:

grant {
permission java.util.PropertyPermission "*", "read";
};

It says any code can only read system properties, but not write.

By the way, the policy file can be created using policytool.

Then run the program like this:

java -Djava.security.manager -Djava.security.policy=lab.policy SecurityManagerTest

You will see an excpetion pop up:

Exception in thread "main" java.security.AccessControlException: access denied (java.util.PropertyPermission greeting write)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:323)
at java.security.AccessController.checkPermission(AccessController.java:546)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:532)
at java.lang.System.setProperty(System.java:727)
at SecurityManagerTest.main(SecurityManagerTest.java:11)

See the security manager is working. It is important to give the right location for the policy file. Because when the security manager is enabled, by default no permission is granted. So any permission will be denied except those defined in the policy file. In case the policy file cannot be found, then the code is given no permission at all.

Tomcat Troubleshooting

The OMII-UK website is a Java web application running in Tomcat, which sits behind an Apache web server. Several methods are there for monitoring and troubleshooting it.

Monitoring

First, a simple web monintor is running on another machine, and checking the contents of several important web pages every 15 minutes. If anything goes wrong, I will get an email notification. Of course, this task can also be done by Nagios. The advantage of using my own web monitor instead of a complicated monitor system like Nagios, is that I have full control over what to inspect. For instance, I am able to detect that although Tomcat is still OK but the database connection is down.

Second, in the Java web applicaiton, which is based on the Spring Framework, Spring AOP (Aspect Oriented Programming) is leveraged to monitor the performance of all servlets. If it takes any servlet more than a pre-defined threshold to serve a request, I'll get an email notification. The threshold, currently set to 200ms, is defined in Spring XML configuration file.

Third, Tomcat is started with "-Dcom.sun.management.jmxremote", which enables local jconsole connection. JDK 1.5 is used. In JDK 1.6, jconsole support is default, so no need to set a JVM option to enable it.

What if something goes wrong ...

1. Check Apache web server only. http://hostname/server-status. You may need to modify httpd.conf to enable this.
   
2. Check Tomcat only. http://hostname:8080.
   
3. Check jconsole->Memory. See if JVM runs out of memory.
4. Check Tomcat logs, Apache logs and database log.
5. Last but not least important, use Chainsaw to check log4j.xml. Chainsaw is a GUI, which makes it easy to browse log4j's log. The log4j configuration may look like:

log4j.appender.FileLog = org.apache.log4j.RollingFileAppender
log4j.appender.FileLog.MaxFileSize = 10MB
log4j.appender.FileLog.MaxBackupIndex = 14
log4j.appender.FileLog.File=$CATALINA_HOME/logs/log4j.xml
log4j.appender.FileLog.layout = org.apache.log4j.xml.XMLLayout
log4j.rootCategory=WARN,FileLog

Let's say there is a memory leak or something inside the Java web application is wrong ...

There are various JVM options that allow us to debug or profile Java web application:

• Frequently used server VM options: -server -Xms512m -Xmx512m
  
• jconsole support: -Dcom.sun.management.jmxremote
  
• Dump heap on running out of memory: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/root/heapdump
• Enable debugging: -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=8000
• Enable YourKit profiling: -agentlib:yjpagent
  

YourKit is a very good Java profiling tool, and it has a free licence for open source projects. That is what I am using. To enable profiling manually, see here.

• On Windows, 32-bit, add \bin\win32 to the (SYSTEM) PATH.
  
• On Linux x86, 32-bit, add /bin/linux-x86-32 to the LD_LIBRARY_PATH.
• Check whether it is working: java -agentlib:yjpagent=help
• Profiling: java -agentlib:yjpagent
• Do not forget to force garbage collection before capture memory.

I also have a stress test tool that replays saved Apache/Tomcat access logs.

Pass Apache Authentication to Tomcat

It costs me several hours to figure this out. So I think it is worthy writing it down.

I have an Apache web server (2.2.3) sitting in front of Tomcat (5.5.27). In Apache configuration, I have:

<location>
AuthType Basic
AuthName "Secure Service"
AuthUserFile /etc/httpd/conf/user.db
require valid-user
</location>

If the authentication succeeds, the HTTP request is passed to Tomcat by mod_proxy_ajp:

ProxyPass /secure/ ajp://localhost:8009/secure/

In Tomcat server.xml, I disable the Tomcat authentication in the AJP connector (both tomcatAuthenticaiton and request.tomcatAuthentication work):

<Connector port="8009" enablelookups="false" redirectport="8443" protocol="AJP/1.3" address="127.0.0.1" tomcatauthentication="false">

If the authentication succeeds, Apache will create an HTTP head:

REMOTE_USER = omii

But in Tomcat, I do not see the REMOTE_USER header. Instead, I see

authorization = Basic b3ip9kd9dkekd9

It turns out that Tomcat puts the Apache authentication information in the form of a user principal, which can be accessed by the following code inside a JSP page:

java.security.Principal pr = request.getUserPrincipal();
if (pr != null) String r = pr.getName(); // r.equals("omii")