More on Java Troubleshooting

JDK has provided better and better troubleshooting facilities, among other tools. Obviously they are extremely important to develop serious Java applications, which usually have to offer specified performance under resource restraint.

These tools include

• jinfo pid : check Java command line options and system properties.
• jstack pid : print thread's stack the their status (e.g., blocking on some object). This is great for knowing what's going on inside the Java application at runtime. And there is no need to set up JVM specially for this purpse.
• jvisualvm : profile and monitor Java process. It seems JDK on Linux has jvisualvm by default. But you need to install jvisualvm separately on Windows.

as well as some other tools already mentioned before:

• jconsole
• jmap
• jhat

Monitoring/Profiling/Debugging Java Process Remotely

Usually there are two difficulties involved:

1. How to communicate between the remote (target) machine and the local (console) machine? This varies per monitoring/profiling/debugging method. E.g., the built-in JMX console (jconsole) uses remote RMI.
   
2. How to penetrate the firewall if existing?

The generic approach is:

1. Know how to do it locally;
2. Run a vncserver on the remote machine (In fact, vncserver is lightweight, and easy to install in case there is no vncserver installed on the remote machine);
3. Use ssh to set up a tunnel between the remote machine and the local machine, e.g., ssh -L5901:remote_machine:5901 userid@remote_machine (ssh is very user-friendly w.r.t. firewall);
   
4. Start a vnc verview locally, pointing to localhost:1, and follow the procedure established in step 1.

In this way, remote monitoring/profiling/debugging tasks are doable following a step-by-step procedure.

Of course, sometimes, there is no need to set up a vncserver as long as the communication between the remote machine and the local machine is done in TCP and the port number is known.

Check Java Thread CPU Usage

Today I came across a question: if an application occupies 100% CPU time, and its source code is extremely large so that reading its source code to find what is going on may not be an option; what should we do to find out the problem?

Let's assume the application is written in Java.

First, we can use jstack combined with jps to print out threads' stack trace, which gives us a good idea about which methods are being executed.

We can also use jconsole with some plugin to display threads' CPU usage. The jconsole plugin is based on JTop (/demo/management/JTop).



Tomcat Troubleshooting

The OMII-UK website is a Java web application running in Tomcat, which sits behind an Apache web server. Several methods are there for monitoring and troubleshooting it.

Monitoring

First, a simple web monintor is running on another machine, and checking the contents of several important web pages every 15 minutes. If anything goes wrong, I will get an email notification. Of course, this task can also be done by Nagios. The advantage of using my own web monitor instead of a complicated monitor system like Nagios, is that I have full control over what to inspect. For instance, I am able to detect that although Tomcat is still OK but the database connection is down.

Second, in the Java web applicaiton, which is based on the Spring Framework, Spring AOP (Aspect Oriented Programming) is leveraged to monitor the performance of all servlets. If it takes any servlet more than a pre-defined threshold to serve a request, I'll get an email notification. The threshold, currently set to 200ms, is defined in Spring XML configuration file.

Third, Tomcat is started with "-Dcom.sun.management.jmxremote", which enables local jconsole connection. JDK 1.5 is used. In JDK 1.6, jconsole support is default, so no need to set a JVM option to enable it.

What if something goes wrong ...

1. Check Apache web server only. http://hostname/server-status. You may need to modify httpd.conf to enable this.
   
2. Check Tomcat only. http://hostname:8080.
   
3. Check jconsole->Memory. See if JVM runs out of memory.
4. Check Tomcat logs, Apache logs and database log.
5. Last but not least important, use Chainsaw to check log4j.xml. Chainsaw is a GUI, which makes it easy to browse log4j's log. The log4j configuration may look like:

log4j.appender.FileLog = org.apache.log4j.RollingFileAppender
log4j.appender.FileLog.MaxFileSize = 10MB
log4j.appender.FileLog.MaxBackupIndex = 14
log4j.appender.FileLog.File=$CATALINA_HOME/logs/log4j.xml
log4j.appender.FileLog.layout = org.apache.log4j.xml.XMLLayout
log4j.rootCategory=WARN,FileLog

Let's say there is a memory leak or something inside the Java web application is wrong ...

There are various JVM options that allow us to debug or profile Java web application:

• Frequently used server VM options: -server -Xms512m -Xmx512m
  
• jconsole support: -Dcom.sun.management.jmxremote
  
• Dump heap on running out of memory: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/root/heapdump
• Enable debugging: -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=8000
• Enable YourKit profiling: -agentlib:yjpagent
  

YourKit is a very good Java profiling tool, and it has a free licence for open source projects. That is what I am using. To enable profiling manually, see here.

• On Windows, 32-bit, add \bin\win32 to the (SYSTEM) PATH.
  
• On Linux x86, 32-bit, add /bin/linux-x86-32 to the LD_LIBRARY_PATH.
• Check whether it is working: java -agentlib:yjpagent=help
• Profiling: java -agentlib:yjpagent
• Do not forget to force garbage collection before capture memory.

I also have a stress test tool that replays saved Apache/Tomcat access logs.