Two Useful Eclipse Plugins

FindBugs uses static analysis to look for bugs in Java code.

ANTLR IDE provides support for ANTLR parser generator.

1<<32 = ?

Let 1 be a 32 bit integer in Java, what should 1<<32 be?

0? Because the "1" bit will be shifted to the far left?

In fact, it is 1. See in the Java programming spec, "If the promoted type of the left-hand operand is int, only the five lowest-order bits of the right-hand operand are used as the shift distance. It is as if the right-hand operand were subjected to a bitwise logical AND operator & with the mask value 0x1f. The shift distance actually used is therefore always in the range 0 to 31, inclusive." So << can be considered as a cyclic left shift operation, the bits shifted on the left end will reappear on the right end.

Ways of Implementing DSL

DSL, domain-specific language, or simply our very own language intended to solve our own specific problem, may not be limited to a programming or scripting language, it can also be, for instance, a file format, or a network protocol. In any case, the language has a syntax which defines the legal vocabulary in the language, and a grammar which defines rules of composing words in the vocabulary to meaningful sentences and paragraphs. So in order to understand the language and to take the right action based on the language, what we need are a lexer that translates language stream into token stream, and a parser that maps tokens to the grammar rules.

There are many ways of implementing DSL.

First, if the language is really simple, we can just code the lexer and parser manually. I.e., manually separate character stream into tokens, and manually construct an LL recursive descent parser. Obviously this method won't work well if the language becomes complex.

Second, make the DSL an XML language. For instance, BPEL is a web service workflow language. Though BPEL is nothing more than an XML language, it surely can fulfill complex tasks. The advantages of making DSL as XML are: first we can use XML schema (XSD) to design the language, so we can also use XML schema validation to validate if something is valid according to the schema; second there are mature and popular tools for validating and parsing XML possibly in every progamming languages.

Third, we can go to the conventional way to design a complex language, i.e., to use a parser generator, such as Antlr. Antlr is implemented in Java, but it can be used to generate code in many other languages than Java, such as C++, Python, C# ... One of the reasons for Antlr is quite popular is that Antlr has a very sophisticated tooling for designing, visualising and even debugging grammar: AntlrWorks, which is quite impressive.

We can even combine the above two methods together when designing a DSL. For instance, XSLT, XSL Transformation, is an XML-based language for transforming XML documents into other documents. XSLT makes use of XPath, which can be implemented using Antlr.

Fourth, nowadays, modern dynamic languages like Ruby and Groovy can be used to implement many DSLs. Because such DSLs will live inside Groovy, they have to obey Groovy's grammar rules. Compared to languages implemented using Antlr, their flexibility is somehow restricted. See here for an introduction to designing DSL with Groovy.

Buzzwords in Job Descriptions

In these days, I am looking at job descriptions for senior Java developer position. Here is the list of buzzwords appearing inside. The list will grow when I come across more buzzwords.

JMeter: a Java framework for measuring server performance. Server types include Web, Web service, database (via JDBC), LDAP, JMS, mail (POP3 ...

Selenium: a Firefox extension that allows composing web tests inside Firefox, replaying tests and generating tests in many different programming languages such as Java, C#, Ruby, Groovy ... This is a good example for what extra functionalities Firefox extension can bring to the browser.

Memory Overhead of Java Objects

First, each Java object has two implicit references: one to its monitor (lock), the other to its method dispatch table. Each reference occupies 4 bytes, so that is 8 bytes overhead.

Second, byte alignment needs to be taken into consideration. On a 32-bit machine, object needs to be aligned at 4-byte boundary. On a 64-bit machine, object needs to be aligned at 8-byte boundary. Nowadays, 64-bit machines become popular. For instance, my laptop has a 64-bit Intel Core 2 Duo CPU T7500.

So on my laptop if I create an object that has only one byte field, then the actual size of the object will be 16 bytes. That is 93.75% overhead. If I create an object that has three int fields, then the actual size of the object will be 24 bytes. That is 50% overhead.

So be very careful when creating a huge number of small objects, because significant extra memory will be required for object overhead.

How to measure the size of an object

Write a simple program consisting of an infinite loop. Inside the loop, create the object whose size is to be measured. Then use "jmap -histo pid" to measure the size. pid is the process id of the Java program. Because the loop is infinite, it gives plenty of time for jmap to connect to the Java process.

Adjustable Timer Task

Java provides TimerTask for tasks that can be scheduled for one-time or repeated execution by a Timer. But it has some serious drawbacks.

First, TimerTask can only be scheduled either for one-time or for repeated execution of a roughly fixed period. It can not be scheduled with changing periods, for example, at the following moments since its start: 0, 5, 6, 12, 100, 101, 102 seconds ... If it is scheduled twice, for example, first scheduled after 1 seconds, then 3 seconds after the first execution, no matter in which thread the second schedule is requested, inside or outside the Timer thread, a java.lang.IllegalStateException: Task already scheduled or cancelled will be thrown.

Second, each TimerTask is served with an individual background thread. Thus it makes TimerTask not a very scalable solution if a lot of TimerTasks are needed but each of them is not very heavy weighted.

Since 1.5, Java provides the java.util.concurrent package, which includes an Executor Framework. The basic idea of Executor Framework is to separate the concerns of tasks and the mechanism to execute tasks. So programmers define tasks and then leave tasks to be executed by the Executor Framework, which can be configurable to use a single thread, or a thread pool to execute the tasks.

The following Java code illustrates how to use ScheduledExecutorService to implement adjustable timer task.

public class Test {

 static private ScheduledExecutorService scheduler =
     Executors.newSingleThreadScheduledExecutor();

 static private Runnable pig = new Runnable() {
     public void run() {
         System.out.println("This is pig");
         scheduler.schedule(this, 1,
            TimeUnit.SECONDS);
     };
 };

 static private Runnable bear = new Runnable() {
     Random random = new Random();
     public void run() {
         System.out.println("This is bear");
         scheduler.schedule(this,
            2 + random.nextInt(3),
            TimeUnit.SECONDS);
     };
 };

 /**
  * @param args
  */
 public static void main(String[] args) {
     scheduler.schedule(pig, 0, TimeUnit.SECONDS);
     scheduler.schedule(bear, 0, TimeUnit.SECONDS);
 }

}

Jar Service Provider

Jar File Spec provides a simple service provider mechanism: a provider configuration file is located inside META-INF/services/, with the service interface or abstract class name as the configuration file name, which contains the list of implementing class names. Java 1.6 has a ServiceLoader class for looking up service providers.

It will be handy to define a well known interface, and to use the provider configuration file to list all implementations. Thus at runtime, ServiceLoader can be used to iterate all available service providers.

Set up HTTP Cookie in Java

In my Java programming, I need to set up some pre-defined HTTP cookie so that I can tell the website what content I prefer on using HttpURLConnection, for example. This is what I do:

// Create my cookie
HttpCookie cookie = new HttpCookie("cookie_name", "cookie_value");
// So that the cookie applies to all pages
cookie.setPath("/");

CookieManager cookieManager = new CookieManager();
cookieManager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);

CookieStore cookieStore = cookieManager.getCookieStore();
cookieStore.add(uri, cookie);

// Set my cookie manager that contains my cookie to be used system-wide
CookieHandler.setDefault(cookieManager);

// From now on my cookie will be used for all connections to the web site denoted by uri.

Web Scraping in Java

There are at least three ways to do web scraping in Java.

First, "manually" use string matching and regular expression to extract information from downloaded HTML.

Second, use JTidy to transform HTML to XHTML, and then use XQuery (e. g., Saxon, ...) over XHTML to extract required information.

Third, which is what I prefer:

1. Create a TagSoup HTML parser, which provides an SAX interface;
2. Use XOM to build a DOM from HTML using the TagSoup SAX parser;
3. Use the built-in XPath query facility inside XOM (i.e., Jaxen) to parse the XOM DOM document.
   

A sample code skeleton looks like:

// Create a TagSoup SAX parser.
XMLReader parser = new org.ccil.cowan.tagsoup.Parser();

// Use the TagSoup parser to build an XOM document from HTML.
Document doc = new Builder(parser).build(new File("index.html"));

// Do some XPath query: find all "table" elements.
Nodes nodes = doc.query("//*[local-name()='table']");

Tomcat Admin Web App and JMX

Tomcat has provided an Admin webapp, which sits inside $CATALINA_HOME/server/webapps in order to access classes contained in Tomcat jars, to make it easy to configure webapps, for instance, to add a DataSource to a webapp.

As seen from its soource code, the Admin webapp simply creates JMX MBeans (managed bean), and save them to a MBean server. Tomcat MBean server then rewrites Tomcat server.xml and webapps/webapp/META-INF/context.xml.

The JMX Remote API specification details how an LDAP server can be used to store and retrieve information about JMX connectors exposed by JMX agents. JNDI is used to talk to an LDAP server.

MBeans can be viewed in the MBeans tab of jconsole.

Further digging on MBeans in Tomcat

• All key constructs in Tomcat, such as Server, Engine, Host and Context, are implemented as MBeans, see package org.apache.catalina.core.
  
• Since Tomcat makes use of Apache Commons Modeler to deliver the Model MBean support, mbeans-descriptors.xml (read by Apache Commons Modeler) appears in many packages that contain MBeans.
• Tomcat uses the MBean server implementation provided by JVM.
• The Server and Context MBeans support operations to store their configurations, which are delegated to the StoreConfig MBean that implements the logic of rewriting various Tomcat configuration files, see package org.apache.catalina.storeconfig in container/modules/storeconfig.
• org.apache.catalina.storeconfig implements a StoreConfigLifecycleListener that registers the StoreConfig MBean right after Tomcat is started. StoreConfigLifecycleListener is configured in Tomcat server.xml.

The information above is based on Tomcat 5.5.27.

Troubleshooting Remote Java Application

Previously, I was using SSH+VPN to reduce the task of troubleshooting a remote Java application to the task of troubleshooting a local Java application. This is a generic approach, but has its own disadvantages. One of them is that sometimes the proper debugging/profiling/monitoring tools can not be run in remote machine due to the restraints in the deployment environment. In that case, we have to run tools locally and perform a genuine remote troubleshooting.

Remote jconsole

To start a Java application that supports remote jconsole is easy: just add the following into Java command line arguments:

-Dcom.sun.management.jmxremote.port=portNum -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

Then we can ask jconsole to simply connect to hostname:portNum, given that a firewall is not set up on the remote machine.

jconsole uses JMX which is built on RMI. The portNum we specify is the port number used by the RMI registry. The actural RMI connection will be opened using another port that can not be specified as a Java command line argument. When the firewall on the remote machine is disabled, and the local jconsole is successfully connected to the remote Java application, we can use "lsof -p pid | grep TCP" to check which port is used by RMI.

See this and that for a programmatic approach to get jconsole through the firewall. It basically starts up a customised RMI registry that open a RMI channel on a pre-defined port, which is implemented as a Java Agent. Here is an almost "official" tutorial to achieve that.

Remote jvisualvm

First of all, according to VisualVM's document, VisualVM can retrieve monitoring information on remote applications but it cannot profile remote applications. VisualVM requires jstatd running on the remote machine. Since jstatd is based on RMI, so VisualVM suffers from the same issue as jconsole when facing firewall.

Remote YourKit

Maybe it is easier to set up YourKit Java profiler to troubleshoot a remote Java application. And YourKit is claimed to be so lightweight that it can be used when the application is running in the production mode.

YourKit provides some means to integrate with a remote JEE/servlet container, such as Tomcat, JBoss, WebSphere, WebLogic ... For Tomcat, the integration creates a startup_with_yjp.sh based on startup.sh, which simply adds the following magic Java options:

"-agentpath:YJP_HOME/bin/linux-x86-32/libyjpagent.so=disablestacktelemetry,
disableexceptiontelemetry,delay=10000,
port=16666,sessionname=Tomcat"

"port=portNum" can be used to specified a number instead of the default 10001. To connect the remote Java application with YourKit agent turned on, simply ask to connect to serverName:portNum in the local YourKit UI. Since all profiling data go in the specified port number, it is easy to set up an SSH tunnel if that port is blocked by firewall.

More investigation needs to be done to understand the profiling overhead given that setting.

More on Java Troubleshooting

JDK has provided better and better troubleshooting facilities, among other tools. Obviously they are extremely important to develop serious Java applications, which usually have to offer specified performance under resource restraint.

These tools include

• jinfo pid : check Java command line options and system properties.
• jstack pid : print thread's stack the their status (e.g., blocking on some object). This is great for knowing what's going on inside the Java application at runtime. And there is no need to set up JVM specially for this purpse.
• jvisualvm : profile and monitor Java process. It seems JDK on Linux has jvisualvm by default. But you need to install jvisualvm separately on Windows.

as well as some other tools already mentioned before:

• jconsole
• jmap
• jhat

Troubleshooting OutOfMemoryError

When a large running Java application throws out an OutOfMemoryError, it indicates either the existence of a memory leak bug, or simply the fact that the maximum heap size has been reached. Don't panic, it is straightforward to do troubleshooting.

Most importantly, DO NOT SHUTDOWN the problematic JVM. Keep the crime scene.

1. Use jmap to dump the heap. Under JDK 1.5, the dumped heap, in the format of hprof, is always put under the home directory and under the name "heap.bin". In addition, jmap can also be used to show the heap object histogram, a quick way to see the classes occupying the most space.
2. Use jhat to analyze the heap dumped by jmap. If the heap is dumped by a JDK 1.5 jmap, invoke jhat (that is only available in JDK6+) in this way: jhat -J-mxNm -stack false heap_dump_file. "-stack false" turns off tracking object allocation call stack because the allocation site information is not available in the heap dump. N should be larger than the maximum heap size used when running the problematic Java process. jhat takes quite some time to analyze the heap dump.
   
3. jhat starts up a web server after finishes analyzing the heap dump. Now we can use a browser to point to the jhat web server, and find out who use up all the memory.
4. At the very end of the front page, click the link "Show heap histogram". it takes quite some time to generate the histogram.
5. In the histogram, the classes whose instances occupy most of the heap can be easily identified. Obviously they are suspects of the crime.
6. Clicking one of the suspects brings us to the page showing the referers of the suspect. In this way, we can track down which part of our code hold references to objects that use up heap. Now by using our knowledge of the program logic, the problem cause can be finally located.

See, the method is staightforward. All we need is patience and a good understanding of the source code in order to find the problem cause.

Direction+

I am working on a web app, Direction+, running on Google App Engine for Java. Yes, it is another direction service that suggests a driving route based on your chosen source and destination. But it has some add-on features as its name suggests: Direction Plus.

First, you can personalise Direction+ by providing your car's information, in particular, the fuel consumption measurement of your car, for urban cycle, extra-urban cycle and combined cycle. Thus, when calculating the route, Direction+ is also able to calculate how much fuel you will use and how much it costs (based on the fuel price you set). The fuel consumption calculation is measured on each individual step of the route. For instance, driving on a motor way will be much more fuel-efficient than driving in the city center. Isn't it good to know the cost of the driving before hand? In addition, the calculation is done in your own browser and your personalisation is stored in your browser as a cookie. So there is no privacy concern at all.

Second, Direction+ is a mashup that combines Google Maps Service and BBC travel news service together. When a driving route is calculated, the travel news, e. g., accidents news, along the route are displayed as well on the map. Isn't it good to know the traffic situation before driving? Currently too many travel news are returned especially when the route is very long, efforts are being made to improve the algorithm of filtering the not-so-relevant news.

How quick is Direction+ updated with BBC travel news? Please see its status page. A complete update takes about 30 minutes. For instance, at the time of writing, the last update took 26 minutes to finish. It read 2297 news from BBC, among which only 50 happened after the previous update, and among the 50 news that newly happened, 12 have cached coordinate in the data store. So though the number of news is large, Direction+ manages to reduce the communication to a small amount.

One of the tricky things about GAE for java is that every servlet request must be served within 30 seconds. So the update of news is implemented in an incremental and on-the-fly way:

• Incremental: the complete update is divided into many small steps, and each step can be finished in a controlled time slot;
• On-the-fly: the news updated in each small step is available immediately, long before the complete update is done.

Direction+ is designed for UK users because I live in UK. Enjoy it and let me know what you think!

Monitoring/Profiling/Debugging Java Process Remotely

Usually there are two difficulties involved:

1. How to communicate between the remote (target) machine and the local (console) machine? This varies per monitoring/profiling/debugging method. E.g., the built-in JMX console (jconsole) uses remote RMI.
   
2. How to penetrate the firewall if existing?

The generic approach is:

1. Know how to do it locally;
2. Run a vncserver on the remote machine (In fact, vncserver is lightweight, and easy to install in case there is no vncserver installed on the remote machine);
3. Use ssh to set up a tunnel between the remote machine and the local machine, e.g., ssh -L5901:remote_machine:5901 userid@remote_machine (ssh is very user-friendly w.r.t. firewall);
   
4. Start a vnc verview locally, pointing to localhost:1, and follow the procedure established in step 1.

In this way, remote monitoring/profiling/debugging tasks are doable following a step-by-step procedure.

Of course, sometimes, there is no need to set up a vncserver as long as the communication between the remote machine and the local machine is done in TCP and the port number is known.

Logging in Java

Yes, this is a very basic issue in Java programming, but sometimes it can be really confusing to find where the logs are and how to declaratively configure what to log.

Log4j is widely used and is configured using log4j.properties. But some programmers choose to use Apache commons logging instead of using log4j API directly. Commons logging is a thin-wrapper for other pluggable logging tools, such log4j and Sun logging facility. The configuration guide of commons logging describes a five step procedure to find the underlying logging mechanism, among which, step 3 says:

"If the Log4J logging system is available in the application class path, use the corresponding wrapper class (Log4JLogger)."

In other words, usually even commons logging API is used in the program, the actual logging service is provided by log4j. Thus log4j.properties is used to configure how logging should be done.

Another logging configuration file is logging.properties, which is used by java.util.logging.

Best Java Decompiler

Recently I am working on a UK e-Science project which uses BPEL to orchestrate web services and submits computational jobs to a condor pool. The source code of some of those web services is missing (great!), so I have to decompile the class files in order to make necessary changes.

In my pursuit of the "best" java decompiler, I first tried JD. It is free. But in my use case, it generates some Java code that can not be compiled. Then I tried DJ. It is good, but it only allows for 10 free trials. Each invocation of DJ is considered to be one trial. After that you have to purchase it.

I hadn't been very happy until I found Jad. Jad did my job. And it turns out that Jad is behind many Java decompiler GUIs such as DJ, Cavaj, and JadClipse, an Eclipse plugin. I am quite happy with Jad even without a GUI to drive it.

It is interesting to see JD and Jad were implemented in C++. Won't it be better for Java programmers to have a Java decompiler written in Java?

Subversive Problem

I am using Subversive - the Eclipse subversion plugin - 0.7.7 plus its subversion connector 2.1.0.

Today an exception jumped out of nowhere, making all my subversion based projects unable to synchronise with the server.

java.lang.NoSuchMethodError: org.eclipse.team.svn.core.connector.SVNChangeStatus.(Ljava/lang/String;Ljava/lang/String;IJJJLjava/lang/String;IIIIZZLjava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;JZLjava/lang/String;Ljava/lang/String;Ljava/lang/String;JLorg/eclipse/team/svn/core/connector/SVNLock;JJILjava/lang/String;ZZLorg/eclipse/team/svn/core/connector/SVNConflictDescriptor;)V
at org.tigris.subversion.javahl.ConversionUtility.convert(ConversionUtility.java:287)
at org.tigris.subversion.javahl.ConversionUtility$4.doStatus(ConversionUtility.java:146)
at org.tigris.subversion.javahl.SVNClient.status(Native Method)
at org.polarion.team.svn.connector.javahl.JavaHLConnector.status(JavaHLConnector.java:406)
at org.eclipse.team.svn.core.extension.factory.ThreadNameModifier.status(ThreadNameModifier.java:606)
at org.eclipse.team.svn.core.utility.SVNUtility.status(SVNUtility.java:330)
at org.eclipse.team.svn.core.utility.SVNUtility.getSVNInfoForNotConnected(SVNUtility.java:803)
at org.eclipse.team.svn.core.SVNTeamProvider.uploadRepositoryResource(SVNTeamProvider.java:241)
at org.eclipse.team.svn.core.SVNTeamProvider.connectToProject(SVNTeamProvider.java:172)
at org.eclipse.team.svn.core.SVNTeamProvider.getRepositoryResource(SVNTeamProvider.java:71)
at org.eclipse.team.svn.core.svnstorage.SVNRemoteStorage.loadLocalResourcesSubTreeSVNImpl(SVNRemoteStorage.java:628)
at org.eclipse.team.svn.core.svnstorage.SVNRemoteStorage.loadLocalResourcesSubTree(SVNRemoteStorage.java:521)
at org.eclipse.team.svn.core.svnstorage.SVNRemoteStorage.getRegisteredChildren(SVNRemoteStorage.java:273)
at org.eclipse.team.svn.core.synchronize.AbstractSVNSubscriber.resourcesStateChangedImpl(AbstractSVNSubscriber.java:212)
at org.eclipse.team.svn.core.synchronize.AbstractSVNSubscriber.resourcesStateChanged(AbstractSVNSubscriber.java:169)
at org.eclipse.team.svn.core.svnstorage.SVNRemoteStorage$3.runImpl(SVNRemoteStorage.java:152)
at org.eclipse.team.svn.core.operation.AbstractActionOperation.run(AbstractActionOperation.java:77)
at org.eclipse.team.svn.core.operation.LoggedOperation.run(LoggedOperation.java:38)
at org.eclipse.team.svn.core.utility.ProgressMonitorUtility.doTask(ProgressMonitorUtility.java:104)
at org.eclipse.team.svn.core.utility.ProgressMonitorUtility.doTaskExternal(ProgressMonitorUtility.java:90)
at org.eclipse.team.svn.core.utility.ProgressMonitorUtility$1$1.run(ProgressMonitorUtility.java:60)
at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:1800)
at org.eclipse.team.svn.core.utility.ProgressMonitorUtility$1.run(ProgressMonitorUtility.java:58)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)

The only possible reason that I can think of is I accidentally copied some ".svn" directories into some non-subversion-based project.

On eclipse.technology.subversive Newsgroup, there is a discussion on the same exception, but happen on Subversive 0.7.8.

I am very lucky since after I changed the subversion connector in use from Native JavaHL to SVN Kit, the problem seems gone! Otherwise, all my development work has to stop!

Override Tomcat Session Cookie

Tomcat uses HTTP cookie to track browser sessions. By default Tomcat 5.5 generates session cookies without an expiration date (Expires=...), like:

Set-Cookie: JSESSIONID=A39F8F3623D20EF9E66D309E298E87E0; Path=/

using Cookie.setMaxAge(-1).

Without an expiration date, this cookie should be deleted by the browser when it is closed, which is what IE7 does. Thus, even the session has a lifetime, say 12 hours, at the Tomcat side, if the browser was restarted, the session would be lost.

Firefox keeps the session cookie when it restarts.

I use the following code to override this behavior:

// after users log in
// HttpServletResponse response
response.setHeader("Set-Cookie", "JSESSIONID=" + request.getSession().getId()
+ "; Expires=" + getCookieExpiresFormat().formatByAge(age)
+ "; Path=/");

It generates something like

Set-Cookie: JSESSIONID=A39F8F3623D20EF9E66D309E298E87E0; Expires=Thu, 22-Apr-2010 20:07:56 GMT; Path=/

Thus the session can be kept live for any time period even when browser restarted.

Check Java Thread CPU Usage

Today I came across a question: if an application occupies 100% CPU time, and its source code is extremely large so that reading its source code to find what is going on may not be an option; what should we do to find out the problem?

Let's assume the application is written in Java.

First, we can use jstack combined with jps to print out threads' stack trace, which gives us a good idea about which methods are being executed.

We can also use jconsole with some plugin to display threads' CPU usage. The jconsole plugin is based on JTop (/demo/management/JTop).