Wednesday, July 08, 2009

Tomcat Admin Web App and JMX

Tomcat has provided an Admin webapp, which sits inside $CATALINA_HOME/server/webapps in order to access classes contained in Tomcat jars, to make it easy to configure webapps, for instance, to add a DataSource to a webapp.

As seen from its soource code, the Admin webapp simply creates JMX MBeans (managed bean), and save them to a MBean server. Tomcat MBean server then rewrites Tomcat server.xml and webapps/webapp/META-INF/context.xml.

The JMX Remote API specification details how an LDAP server can be used to store and retrieve information about JMX connectors exposed by JMX agents. JNDI is used to talk to an LDAP server.

MBeans can be viewed in the MBeans tab of jconsole.

Further digging on MBeans in Tomcat
  • All key constructs in Tomcat, such as Server, Engine, Host and Context, are implemented as MBeans, see package org.apache.catalina.core.
  • Since Tomcat makes use of Apache Commons Modeler to deliver the Model MBean support, mbeans-descriptors.xml (read by Apache Commons Modeler) appears in many packages that contain MBeans.
  • Tomcat uses the MBean server implementation provided by JVM.
  • The Server and Context MBeans support operations to store their configurations, which are delegated to the StoreConfig MBean that implements the logic of rewriting various Tomcat configuration files, see package org.apache.catalina.storeconfig in container/modules/storeconfig.
  • org.apache.catalina.storeconfig implements a StoreConfigLifecycleListener that registers the StoreConfig MBean right after Tomcat is started. StoreConfigLifecycleListener is configured in Tomcat server.xml.
The information above is based on Tomcat 5.5.27.

Tuesday, July 07, 2009

Troubleshooting Remote Java Application

Previously, I was using SSH+VPN to reduce the task of troubleshooting a remote Java application to the task of troubleshooting a local Java application. This is a generic approach, but has its own disadvantages. One of them is that sometimes the proper debugging/profiling/monitoring tools can not be run in remote machine due to the restraints in the deployment environment. In that case, we have to run tools locally and perform a genuine remote troubleshooting.

Remote jconsole

To start a Java application that supports remote jconsole is easy: just add the following into Java command line arguments:

Then we can ask jconsole to simply connect to hostname:portNum, given that a firewall is not set up on the remote machine.

jconsole uses JMX which is built on RMI. The portNum we specify is the port number used by the RMI registry. The actural RMI connection will be opened using another port that can not be specified as a Java command line argument. When the firewall on the remote machine is disabled, and the local jconsole is successfully connected to the remote Java application, we can use "lsof -p pid | grep TCP" to check which port is used by RMI.

See this and that for a programmatic approach to get jconsole through the firewall. It basically starts up a customised RMI registry that open a RMI channel on a pre-defined port, which is implemented as a Java Agent. Here is an almost "official" tutorial to achieve that.

Remote jvisualvm

First of all, according to VisualVM's document, VisualVM can retrieve monitoring information on remote applications but it cannot profile remote applications. VisualVM requires jstatd running on the remote machine. Since jstatd is based on RMI, so VisualVM suffers from the same issue as jconsole when facing firewall.

Remote YourKit

Maybe it is easier to set up YourKit Java profiler to troubleshoot a remote Java application. And YourKit is claimed to be so lightweight that it can be used when the application is running in the production mode.

YourKit provides some means to integrate with a remote JEE/servlet container, such as Tomcat, JBoss, WebSphere, WebLogic ... For Tomcat, the integration creates a based on, which simply adds the following magic Java options:


"port=portNum" can be used to specified a number instead of the default 10001. To connect the remote Java application with YourKit agent turned on, simply ask to connect to serverName:portNum in the local YourKit UI. Since all profiling data go in the specified port number, it is easy to set up an SSH tunnel if that port is blocked by firewall.

More investigation needs to be done to understand the profiling overhead given that setting.