Weijian's Technical Notes

Two Useful Eclipse Plugins

2010-03-31T21:29:00.003+01:00

FindBugs uses static analysis to look for bugs in Java code.

ANTLR IDE provides support for ANTLR parser generator.

1<<32 = ?

2010-02-06T20:09:00.002Z

Let 1 be a 32 bit integer in Java, what should 1<<32 be?

0? Because the "1" bit will be shifted to the far left?

In fact, it is 1. See in the Java programming spec, "If the promoted type of the left-hand operand is int, only the five lowest-order bits of the right-hand operand are used as the shift distance. It is as if the right-hand operand were subjected to a bitwise logical AND operator & with the mask value 0x1f. The shift distance actually used is therefore always in the range 0 to 31, inclusive." So << can be considered as a cyclic left shift operation, the bits shifted on the left end will reappear on the right end.

Ways of Implementing DSL

2010-01-18T21:38:00.005Z

DSL, domain-specific language, or simply our very own language intended to solve our own specific problem, may not be limited to a programming or scripting language, it can also be, for instance, a file format, or a network protocol. In any case, the language has a syntax which defines the legal vocabulary in the language, and a grammar which defines rules of composing words in the vocabulary to meaningful sentences and paragraphs. So in order to understand the language and to take the right action based on the language, what we need are a lexer that translates language stream into token stream, and a parser that maps tokens to the grammar rules.

There are many ways of implementing DSL.

First, if the language is really simple, we can just code the lexer and parser manually. I.e., manually separate character stream into tokens, and manually construct an LL recursive descent parser. Obviously this method won't work well if the language becomes complex.

Second, make the DSL an XML language. For instance, BPEL is a web service workflow language. Though BPEL is nothing more than an XML language, it surely can fulfill complex tasks. The advantages of making DSL as XML are: first we can use XML schema (XSD) to design the language, so we can also use XML schema validation to validate if something is valid according to the schema; second there are mature and popular tools for validating and parsing XML possibly in every progamming languages.

Third, we can go to the conventional way to design a complex language, i.e., to use a parser generator, such as Antlr. Antlr is implemented in Java, but it can be used to generate code in many other languages than Java, such as C++, Python, C# ... One of the reasons for Antlr is quite popular is that Antlr has a very sophisticated tooling for designing, visualising and even debugging grammar: AntlrWorks, which is quite impressive.

We can even combine the above two methods together when designing a DSL. For instance, XSLT, XSL Transformation, is an XML-based language for transforming XML documents into other documents. XSLT makes use of XPath, which can be implemented using Antlr.

Fourth, nowadays, modern dynamic languages like Ruby and Groovy can be used to implement many DSLs. Because such DSLs will live inside Groovy, they have to obey Groovy's grammar rules. Compared to languages implemented using Antlr, their flexibility is somehow restricted. See here for an introduction to designing DSL with Groovy.

Protocol Verification Using Tcpdump and Wireshark

2010-01-10T23:12:00.003Z

It is a must to use Tcpdump (to capture) and/or Wireshark (to analyse) to verify protocol implementation when doing TCP/IP network programming.

First, it can verify if the message sent by me is as intended, thus it can expose any bug in my protocol implementation. Today a bug has been found in this sense: I use Java DataOutputStream.write(int) to send a short, so only the lowest byte of the short has been sent.

Second, it can detect any inconsistency between the protocol specification, i.e., what I think the message should look like, and the actual messages that are sent by the other party, i.e., out of my control. Today, several inconsistencies have been found in this way. For instance, the length of a message (a TCP packet) is different from that defined in the spec. Clearly the spec in my hand is out of dated.

Without Tcpdump and Wireshark, I would be like walking in the dark when implementing my protocol.

Capture and Analyse Network Packets

2009-12-29T23:31:00.004Z

tcpdump is the standard packet capturing facility available on most Linux systems, which is based on command line. Wireshark, formerly called Ethereal, is another popular packet capturing facility, free and GUI-based. Both tcpdump and Wireshark are based on pcap, so it is possible to combine them in capturing and analysing network packets, to take advantages of both.

For instance, I use the following tcpdump command to capture the traffic to and from www.google.com using http protocol:

sudo tcpdump -i wlan0 -w td.dat -nnvvXSs 1514 host www.google.com

Note:
sudo: It may require root privilege to capture packets.
-i wlan0: By default tcpdump captures packets on the eth0 interface. Since I am using wireless, I need to specify the wireless interface wlan0. When using VPN, the interface should be ppp0 instead usually.
-w td.dat: write all captured packets to the file td.dat.
-nn: no hostname and port resolving.
-vv: very verbose.
-X: print in both hex and ascii.
-S: absolute sequence.
-s 1514: tcpdump takes the first 68 bytes of data from a packet by default. Here the first 1514 bytes are taken.
host www.google.com: this is the expression which says capturing packets whose dst host or src host is www.google.com.

See this tcpdump tutorial for more info about tcpdump usage.

Now Wireshark can be used to analyse the captured packets by tcpdump. Here Wireshark's GUI is exploited.

Use Wireshark to open td.dat, and apply the preset http filter. The http traffic can be easily browsed.

Buzzwords in Job Descriptions

2009-12-12T13:49:00.002Z

In these days, I am looking at job descriptions for senior Java developer position. Here is the list of buzzwords appearing inside. The list will grow when I come across more buzzwords.

JMeter: a Java framework for measuring server performance. Server types include Web, Web service, database (via JDBC), LDAP, JMS, mail (POP3 ...

Selenium: a Firefox extension that allows composing web tests inside Firefox, replaying tests and generating tests in many different programming languages such as Java, C#, Ruby, Groovy ... This is a good example for what extra functionalities Firefox extension can bring to the browser.

Memory Overhead of Java Objects

2009-12-05T22:07:00.003Z

First, each Java object has two implicit references: one to its monitor (lock), the other to its method dispatch table. Each reference occupies 4 bytes, so that is 8 bytes overhead.

Second, byte alignment needs to be taken into consideration. On a 32-bit machine, object needs to be aligned at 4-byte boundary. On a 64-bit machine, object needs to be aligned at 8-byte boundary. Nowadays, 64-bit machines become popular. For instance, my laptop has a 64-bit Intel Core 2 Duo CPU T7500.

So on my laptop if I create an object that has only one byte field, then the actual size of the object will be 16 bytes. That is 93.75% overhead. If I create an object that has three int fields, then the actual size of the object will be 24 bytes. That is 50% overhead.

So be very careful when creating a huge number of small objects, because significant extra memory will be required for object overhead.

How to measure the size of an object

Write a simple program consisting of an infinite loop. Inside the loop, create the object whose size is to be measured. Then use "jmap -histo pid" to measure the size. pid is the process id of the Java program. Because the loop is infinite, it gives plenty of time for jmap to connect to the Java process.

Adjustable Timer Task

2009-10-23T15:16:00.004+01:00

Java provides TimerTask for tasks that can be scheduled for one-time or repeated execution by a Timer. But it has some serious drawbacks.

First, TimerTask can only be scheduled either for one-time or for repeated execution of a roughly fixed period. It can not be scheduled with changing periods, for example, at the following moments since its start: 0, 5, 6, 12, 100, 101, 102 seconds ... If it is scheduled twice, for example, first scheduled after 1 seconds, then 3 seconds after the first execution, no matter in which thread the second schedule is requested, inside or outside the Timer thread, a java.lang.IllegalStateException: Task already scheduled or cancelled will be thrown.

Second, each TimerTask is served with an individual background thread. Thus it makes TimerTask not a very scalable solution if a lot of TimerTasks are needed but each of them is not very heavy weighted.

Since 1.5, Java provides the java.util.concurrent package, which includes an Executor Framework. The basic idea of Executor Framework is to separate the concerns of tasks and the mechanism to execute tasks. So programmers define tasks and then leave tasks to be executed by the Executor Framework, which can be configurable to use a single thread, or a thread pool to execute the tasks.

The following Java code illustrates how to use ScheduledExecutorService to implement adjustable timer task.


public class Test {

 static private ScheduledExecutorService scheduler =
     Executors.newSingleThreadScheduledExecutor();

 static private Runnable pig = new Runnable() {
     public void run() {
         System.out.println("This is pig");
         scheduler.schedule(this, 1,
            TimeUnit.SECONDS);
     };
 };

 static private Runnable bear = new Runnable() {
     Random random = new Random();
     public void run() {
         System.out.println("This is bear");
         scheduler.schedule(this,
            2 + random.nextInt(3),
            TimeUnit.SECONDS);
     };
 };

 /**
  * @param args
  */
 public static void main(String[] args) {
     scheduler.schedule(pig, 0, TimeUnit.SECONDS);
     scheduler.schedule(bear, 0, TimeUnit.SECONDS);
 }

}

Globus Toolkit's Future

2009-10-21T14:42:00.003+01:00

Today Ian Foster sent out an email about Globus' future plans, which contains many interesting points.

First, a GRAM5 (for job submission) will come out as a replacement for both GRAM2 (old fashion) and GRAM4 (web service based). It confirms my feeling that web service based job submission such as GridSAM does not pick up enough users. The old fashion job submission just can not be retired at this moment.

Second, the Java web service core will be re-implemented to leverage state-of-the-art technologies. Apache CXF is selected as the web service development kit. See here for a very informative comparison between several popular latest the web service development kits including Apache CXF, Apache Axis2 and Metro (the JAX-WS RI). Apache CXF is favored over Apache Axis2 because Axis2 uses proprietary deployment model and lacks the support for IoC containers such as Spring Framework.

Third, globus.org will provide RFT (reliable file transfer) online service, which provides reliable, high-performance, end-to-end, fire-and-forget data transfer. globus.org has moved into the era of cloud computing!

Fourth, MDS (monitoring and discovery service) will be refactored to separate monitoring and service/resource discovery. Monitoring will be left to mature dedicated monitor softwares. MDS itself will focus on acting as a service registry. I certainly agree this decision since it favors the idea of separating concerns. The recently published GLUE2 spec still mixes up monitoring data and service metadata for discovery. It is interesting to see if the future development of GLUE will reflect the decision made on MDS.

Jar Service Provider

2009-10-15T14:51:00.002+01:00

Jar File Spec provides a simple service provider mechanism: a provider configuration file is located inside META-INF/services/, with the service interface or abstract class name as the configuration file name, which contains the list of implementing class names. Java 1.6 has a ServiceLoader class for looking up service providers.

It will be handy to define a well known interface, and to use the provider configuration file to list all implementations. Thus at runtime, ServiceLoader can be used to iterate all available service providers.

Set up HTTP Cookie in Java

2009-09-14T20:54:00.003+01:00

In my Java programming, I need to set up some pre-defined HTTP cookie so that I can tell the website what content I prefer on using HttpURLConnection, for example. This is what I do:

// Create my cookie
HttpCookie cookie = new HttpCookie("cookie_name", "cookie_value");
// So that the cookie applies to all pages
cookie.setPath("/");

CookieManager cookieManager = new CookieManager();
cookieManager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);

CookieStore cookieStore = cookieManager.getCookieStore();
cookieStore.add(uri, cookie);

// Set my cookie manager that contains my cookie to be used system-wide
CookieHandler.setDefault(cookieManager);

// From now on my cookie will be used for all connections to the web site denoted by uri.

Web Scraping in Java

2009-08-26T14:56:00.003+01:00

There are at least three ways to do web scraping in Java.

First, "manually" use string matching and regular expression to extract information from downloaded HTML.

Second, use JTidy to transform HTML to XHTML, and then use XQuery (e. g., Saxon, ...) over XHTML to extract required information.

Third, which is what I prefer:

Create a TagSoup HTML parser, which provides an SAX interface;
Use XOM to build a DOM from HTML using the TagSoup SAX parser;
Use the built-in XPath query facility inside XOM (i.e., Jaxen) to parse the XOM DOM document.

A sample code skeleton looks like:

// Create a TagSoup SAX parser.
XMLReader parser = new org.ccil.cowan.tagsoup.Parser();

// Use the TagSoup parser to build an XOM document from HTML.
Document doc = new Builder(parser).build(new File("index.html"));

// Do some XPath query: find all "table" elements.
Nodes nodes = doc.query("//*[local-name()='table']");

Tomcat Admin Web App and JMX

2009-07-08T14:42:00.009+01:00

Tomcat has provided an Admin webapp, which sits inside $CATALINA_HOME/server/webapps in order to access classes contained in Tomcat jars, to make it easy to configure webapps, for instance, to add a DataSource to a webapp.

As seen from its soource code, the Admin webapp simply creates JMX MBeans (managed bean), and save them to a MBean server. Tomcat MBean server then rewrites Tomcat server.xml and webapps/webapp/META-INF/context.xml.

The JMX Remote API specification details how an LDAP server can be used to store and retrieve information about JMX connectors exposed by JMX agents. JNDI is used to talk to an LDAP server.

MBeans can be viewed in the MBeans tab of jconsole.

Further digging on MBeans in Tomcat

All key constructs in Tomcat, such as Server, Engine, Host and Context, are implemented as MBeans, see package org.apache.catalina.core.
Since Tomcat makes use of Apache Commons Modeler to deliver the Model MBean support, mbeans-descriptors.xml (read by Apache Commons Modeler) appears in many packages that contain MBeans.
Tomcat uses the MBean server implementation provided by JVM.
The Server and Context MBeans support operations to store their configurations, which are delegated to the StoreConfig MBean that implements the logic of rewriting various Tomcat configuration files, see package org.apache.catalina.storeconfig in container/modules/storeconfig.
org.apache.catalina.storeconfig implements a StoreConfigLifecycleListener that registers the StoreConfig MBean right after Tomcat is started. StoreConfigLifecycleListener is configured in Tomcat server.xml.

The information above is based on Tomcat 5.5.27.

Troubleshooting Remote Java Application

2009-07-07T14:22:00.011+01:00

Previously, I was using SSH+VPN to reduce the task of troubleshooting a remote Java application to the task of troubleshooting a local Java application. This is a generic approach, but has its own disadvantages. One of them is that sometimes the proper debugging/profiling/monitoring tools can not be run in remote machine due to the restraints in the deployment environment. In that case, we have to run tools locally and perform a genuine remote troubleshooting.

Remote jconsole

To start a Java application that supports remote jconsole is easy: just add the following into Java command line arguments:

-Dcom.sun.management.jmxremote.port=portNum -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

Then we can ask jconsole to simply connect to hostname:portNum, given that a firewall is not set up on the remote machine.

jconsole uses JMX which is built on RMI. The portNum we specify is the port number used by the RMI registry. The actural RMI connection will be opened using another port that can not be specified as a Java command line argument. When the firewall on the remote machine is disabled, and the local jconsole is successfully connected to the remote Java application, we can use "lsof -p pid | grep TCP" to check which port is used by RMI.

See this and that for a programmatic approach to get jconsole through the firewall. It basically starts up a customised RMI registry that open a RMI channel on a pre-defined port, which is implemented as a Java Agent. Here is an almost "official" tutorial to achieve that.

Remote jvisualvm

First of all, according to VisualVM's document, VisualVM can retrieve monitoring information on remote applications but it cannot profile remote applications. VisualVM requires jstatd running on the remote machine. Since jstatd is based on RMI, so VisualVM suffers from the same issue as jconsole when facing firewall.

Remote YourKit

Maybe it is easier to set up YourKit Java profiler to troubleshoot a remote Java application. And YourKit is claimed to be so lightweight that it can be used when the application is running in the production mode.

YourKit provides some means to integrate with a remote JEE/servlet container, such as Tomcat, JBoss, WebSphere, WebLogic ... For Tomcat, the integration creates a startup_with_yjp.sh based on startup.sh, which simply adds the following magic Java options:

"-agentpath:YJP_HOME/bin/linux-x86-32/libyjpagent.so=disablestacktelemetry,
disableexceptiontelemetry,delay=10000,
port=16666,sessionname=Tomcat"

"port=portNum" can be used to specified a number instead of the default 10001. To connect the remote Java application with YourKit agent turned on, simply ask to connect to serverName:portNum in the local YourKit UI. Since all profiling data go in the specified port number, it is easy to set up an SSH tunnel if that port is blocked by firewall.

More investigation needs to be done to understand the profiling overhead given that setting.

More on Java Troubleshooting

2009-06-29T11:08:00.002+01:00

JDK has provided better and better troubleshooting facilities, among other tools. Obviously they are extremely important to develop serious Java applications, which usually have to offer specified performance under resource restraint.

These tools include

jinfo pid : check Java command line options and system properties.
jstack pid : print thread's stack the their status (e.g., blocking on some object). This is great for knowing what's going on inside the Java application at runtime. And there is no need to set up JVM specially for this purpse.
jvisualvm : profile and monitor Java process. It seems JDK on Linux has jvisualvm by default. But you need to install jvisualvm separately on Windows.

as well as some other tools already mentioned before:

jconsole
jmap
jhat

Clean up Harddisk before Dispose of Computer

2009-06-25T15:16:00.005+01:00

After 3 years' working as a heater, my Sony VAIO laptop is finally retiring. There is one critical task that must be done before it can be disposed of: cleaning up its harddisk. Since I graduated from a computer lab that researched and developed one of the very first harddisks in China, I long know not only deleted files but also overwritten files can be recovered. Nevertheless, this blog as well as its comments still give me a lot of information. There are some useful points:

On Linux, shred can be used to clean up disk. By default it overwrites the harddisk for 25 times. Call it like this: shred -vz /dev/hda (-z: finally overwrite with all zero to disguise the shred process.)
Or, use DBAN, which is used by many governments. DBAN stands for "Darik's Boot And Nuke". The software is a boot CD/DVD image that is used to boot the computer and then do the cleaning up.
Here is the seminal paper describing the theory behind securely deleting harddisk data.

It will be useful to know how long it takes for shred or DBAN to process a harddisk of a certain size.

On my Sony VAIO VGN-SZ3XWP/C, shred takes about 3.5 hours to 4 hours to run a random pass of a 39 GiB disk partition! A random pass overwrites the disk with random bits. Not sure how random it is. The 1st and the 13th (possibly the last one, the 25th pass) passes are random passes. The other passes use different short fixed pattern to overwrite. Each costs about half an hour.

Further updates for running "shred -vz /dev/sda" on a 94GiB harddisk. The 1st, 13th and 25th passes are using random data generated from /dev/urandom. The other passes write fixed but different bit patterns. It takes 49h33m26s to finish.

IDE for C++

2009-06-22T20:59:00.002+01:00

I may work on a C++ project in future. So I am looking for an IDE for C++. I know A good IDE can double the productivity of programming.

Unlike Java, C++ is not cross platform and has many variants though it does have a standard. Thus which IDE to choose is dependent on which C++ to use.

On Windows, Microsoft Visual Studio is the choice for Visual C++ :)

I would like to choose Eclipse CDT for GNU C++. CDT is only IDE. To build and debug C++ code, CDT requires external toolchain, e.g., GNU toolchain. CDT on Liunx will automatically pick up GNU toolchain, which usually is there. But to make CDT on Windows work, we need to install MinGW (Minimalist GNU for Windows). Here is a good tutorial on installing MinGW. Once MinGW is properly installed and added into PATH, CDT can pick up MinGW as a toolchain automatically.

Troubleshooting OutOfMemoryError

2009-06-19T14:11:00.002+01:00

When a large running Java application throws out an OutOfMemoryError, it indicates either the existence of a memory leak bug, or simply the fact that the maximum heap size has been reached. Don't panic, it is straightforward to do troubleshooting.

Most importantly, DO NOT SHUTDOWN the problematic JVM. Keep the crime scene.

Use jmap to dump the heap. Under JDK 1.5, the dumped heap, in the format of hprof, is always put under the home directory and under the name "heap.bin". In addition, jmap can also be used to show the heap object histogram, a quick way to see the classes occupying the most space.
Use jhat to analyze the heap dumped by jmap. If the heap is dumped by a JDK 1.5 jmap, invoke jhat (that is only available in JDK6+) in this way: jhat -J-mxNm -stack false heap_dump_file. "-stack false" turns off tracking object allocation call stack because the allocation site information is not available in the heap dump. N should be larger than the maximum heap size used when running the problematic Java process. jhat takes quite some time to analyze the heap dump.
jhat starts up a web server after finishes analyzing the heap dump. Now we can use a browser to point to the jhat web server, and find out who use up all the memory.
At the very end of the front page, click the link "Show heap histogram". it takes quite some time to generate the histogram.
In the histogram, the classes whose instances occupy most of the heap can be easily identified. Obviously they are suspects of the crime.
Clicking one of the suspects brings us to the page showing the referers of the suspect. In this way, we can track down which part of our code hold references to objects that use up heap. Now by using our knowledge of the program logic, the problem cause can be finally located.

See, the method is staightforward. All we need is patience and a good understanding of the source code in order to find the problem cause.

Direction+

2009-06-12T11:38:00.003+01:00

I am working on a web app, Direction+, running on Google App Engine for Java. Yes, it is another direction service that suggests a driving route based on your chosen source and destination. But it has some add-on features as its name suggests: Direction Plus.

First, you can personalise Direction+ by providing your car's information, in particular, the fuel consumption measurement of your car, for urban cycle, extra-urban cycle and combined cycle. Thus, when calculating the route, Direction+ is also able to calculate how much fuel you will use and how much it costs (based on the fuel price you set). The fuel consumption calculation is measured on each individual step of the route. For instance, driving on a motor way will be much more fuel-efficient than driving in the city center. Isn't it good to know the cost of the driving before hand? In addition, the calculation is done in your own browser and your personalisation is stored in your browser as a cookie. So there is no privacy concern at all.

Second, Direction+ is a mashup that combines Google Maps Service and BBC travel news service together. When a driving route is calculated, the travel news, e. g., accidents news, along the route are displayed as well on the map. Isn't it good to know the traffic situation before driving? Currently too many travel news are returned especially when the route is very long, efforts are being made to improve the algorithm of filtering the not-so-relevant news.

How quick is Direction+ updated with BBC travel news? Please see its status page. A complete update takes about 30 minutes. For instance, at the time of writing, the last update took 26 minutes to finish. It read 2297 news from BBC, among which only 50 happened after the previous update, and among the 50 news that newly happened, 12 have cached coordinate in the data store. So though the number of news is large, Direction+ manages to reduce the communication to a small amount.

One of the tricky things about GAE for java is that every servlet request must be served within 30 seconds. So the update of news is implemented in an incremental and on-the-fly way:

Incremental: the complete update is divided into many small steps, and each step can be finished in a controlled time slot;
On-the-fly: the news updated in each small step is available immediately, long before the complete update is done.

Direction+ is designed for UK users because I live in UK. Enjoy it and let me know what you think!

Monitoring/Profiling/Debugging Java Process Remotely

2009-06-08T11:43:00.003+01:00

Usually there are two difficulties involved:

How to communicate between the remote (target) machine and the local (console) machine? This varies per monitoring/profiling/debugging method. E.g., the built-in JMX console (jconsole) uses remote RMI.
How to penetrate the firewall if existing?

The generic approach is:

Know how to do it locally;
Run a vncserver on the remote machine (In fact, vncserver is lightweight, and easy to install in case there is no vncserver installed on the remote machine);
Use ssh to set up a tunnel between the remote machine and the local machine, e.g., ssh -L5901:remote_machine:5901 userid@remote_machine (ssh is very user-friendly w.r.t. firewall);
Start a vnc verview locally, pointing to localhost:1, and follow the procedure established in step 1.

In this way, remote monitoring/profiling/debugging tasks are doable following a step-by-step procedure.

Of course, sometimes, there is no need to set up a vncserver as long as the communication between the remote machine and the local machine is done in TCP and the port number is known.

Logging in Java

2009-05-29T11:53:00.003+01:00

Yes, this is a very basic issue in Java programming, but sometimes it can be really confusing to find where the logs are and how to declaratively configure what to log.

Log4j is widely used and is configured using log4j.properties. But some programmers choose to use Apache commons logging instead of using log4j API directly. Commons logging is a thin-wrapper for other pluggable logging tools, such log4j and Sun logging facility. The configuration guide of commons logging describes a five step procedure to find the underlying logging mechanism, among which, step 3 says:

"If the Log4J logging system is available in the application class path, use the corresponding wrapper class (Log4JLogger)."

In other words, usually even commons logging API is used in the program, the actual logging service is provided by log4j. Thus log4j.properties is used to configure how logging should be done.

Another logging configuration file is logging.properties, which is used by java.util.logging.

Best Java Decompiler

2009-05-28T23:02:00.002+01:00

Recently I am working on a UK e-Science project which uses BPEL to orchestrate web services and submits computational jobs to a condor pool. The source code of some of those web services is missing (great!), so I have to decompile the class files in order to make necessary changes.

In my pursuit of the "best" java decompiler, I first tried JD. It is free. But in my use case, it generates some Java code that can not be compiled. Then I tried DJ. It is good, but it only allows for 10 free trials. Each invocation of DJ is considered to be one trial. After that you have to purchase it.

I hadn't been very happy until I found Jad. Jad did my job. And it turns out that Jad is behind many Java decompiler GUIs such as DJ, Cavaj, and JadClipse, an Eclipse plugin. I am quite happy with Jad even without a GUI to drive it.

It is interesting to see JD and Jad were implemented in C++. Won't it be better for Java programmers to have a Java decompiler written in Java?

Subversive Problem

2009-05-04T15:06:00.005+01:00

I am using Subversive - the Eclipse subversion plugin - 0.7.7 plus its subversion connector 2.1.0.

Today an exception jumped out of nowhere, making all my subversion based projects unable to synchronise with the server.

java.lang.NoSuchMethodError: org.eclipse.team.svn.core.connector.SVNChangeStatus.(Ljava/lang/String;Ljava/lang/String;IJJJLjava/lang/String;IIIIZZLjava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;JZLjava/lang/String;Ljava/lang/String;Ljava/lang/String;JLorg/eclipse/team/svn/core/connector/SVNLock;JJILjava/lang/String;ZZLorg/eclipse/team/svn/core/connector/SVNConflictDescriptor;)V
at org.tigris.subversion.javahl.ConversionUtility.convert(ConversionUtility.java:287)
at org.tigris.subversion.javahl.ConversionUtility$4.doStatus(ConversionUtility.java:146)
at org.tigris.subversion.javahl.SVNClient.status(Native Method)
at org.polarion.team.svn.connector.javahl.JavaHLConnector.status(JavaHLConnector.java:406)
at org.eclipse.team.svn.core.extension.factory.ThreadNameModifier.status(ThreadNameModifier.java:606)
at org.eclipse.team.svn.core.utility.SVNUtility.status(SVNUtility.java:330)
at org.eclipse.team.svn.core.utility.SVNUtility.getSVNInfoForNotConnected(SVNUtility.java:803)
at org.eclipse.team.svn.core.SVNTeamProvider.uploadRepositoryResource(SVNTeamProvider.java:241)
at org.eclipse.team.svn.core.SVNTeamProvider.connectToProject(SVNTeamProvider.java:172)
at org.eclipse.team.svn.core.SVNTeamProvider.getRepositoryResource(SVNTeamProvider.java:71)
at org.eclipse.team.svn.core.svnstorage.SVNRemoteStorage.loadLocalResourcesSubTreeSVNImpl(SVNRemoteStorage.java:628)
at org.eclipse.team.svn.core.svnstorage.SVNRemoteStorage.loadLocalResourcesSubTree(SVNRemoteStorage.java:521)
at org.eclipse.team.svn.core.svnstorage.SVNRemoteStorage.getRegisteredChildren(SVNRemoteStorage.java:273)
at org.eclipse.team.svn.core.synchronize.AbstractSVNSubscriber.resourcesStateChangedImpl(AbstractSVNSubscriber.java:212)
at org.eclipse.team.svn.core.synchronize.AbstractSVNSubscriber.resourcesStateChanged(AbstractSVNSubscriber.java:169)
at org.eclipse.team.svn.core.svnstorage.SVNRemoteStorage$3.runImpl(SVNRemoteStorage.java:152)
at org.eclipse.team.svn.core.operation.AbstractActionOperation.run(AbstractActionOperation.java:77)
at org.eclipse.team.svn.core.operation.LoggedOperation.run(LoggedOperation.java:38)
at org.eclipse.team.svn.core.utility.ProgressMonitorUtility.doTask(ProgressMonitorUtility.java:104)
at org.eclipse.team.svn.core.utility.ProgressMonitorUtility.doTaskExternal(ProgressMonitorUtility.java:90)
at org.eclipse.team.svn.core.utility.ProgressMonitorUtility$1$1.run(ProgressMonitorUtility.java:60)
at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:1800)
at org.eclipse.team.svn.core.utility.ProgressMonitorUtility$1.run(ProgressMonitorUtility.java:58)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)

The only possible reason that I can think of is I accidentally copied some ".svn" directories into some non-subversion-based project.

On eclipse.technology.subversive Newsgroup, there is a discussion on the same exception, but happen on Subversive 0.7.8.

I am very lucky since after I changed the subversion connector in use from Native JavaHL to SVN Kit, the problem seems gone! Otherwise, all my development work has to stop!

MVC Practice in Browser Application

2009-04-27T16:00:00.008+01:00

I am working on a browser application, called Direction+. I am trying to enforce the MVC pattern in order to achieve better code manageability. Basically, it is all about separating concerns.

View

All presentation goes into HTML code. Better no HTML code generated in JavaScript. Furthermore, style is separated into CSS. For the concern of performance, CSS could be embedded into HTML to save a pair of request and response.

Model

The states should be kept in JavaScript structs, objects or variables. HTML markups SHOULD NOT be used to store any state other than user input. Using HTML markup for non-user-input state, for instance, a hidden field in form, is a necessary communication mechanism between browser and web server, but not necessary in a browser application.

By keeping all states in JavaScript, it can be assured to only have a single copy of state in a centralised place and the HTML view is purely a presentation of the state.

Controller

Business logic is implemented in JavaScript. A minimal set of code, which are basically event handlers, are embedded into HTML, as entry points into the business logic.

JavaScript should be written in an object-oriented style.

Override Tomcat Session Cookie

2009-04-22T21:27:00.002+01:00

Tomcat uses HTTP cookie to track browser sessions. By default Tomcat 5.5 generates session cookies without an expiration date (Expires=...), like:

Set-Cookie: JSESSIONID=A39F8F3623D20EF9E66D309E298E87E0; Path=/

using Cookie.setMaxAge(-1).

Without an expiration date, this cookie should be deleted by the browser when it is closed, which is what IE7 does. Thus, even the session has a lifetime, say 12 hours, at the Tomcat side, if the browser was restarted, the session would be lost.

Firefox keeps the session cookie when it restarts.

I use the following code to override this behavior:

// after users log in
// HttpServletResponse response
response.setHeader("Set-Cookie", "JSESSIONID=" + request.getSession().getId()
+ "; Expires=" + getCookieExpiresFormat().formatByAge(age)
+ "; Path=/");

It generates something like

Set-Cookie: JSESSIONID=A39F8F3623D20EF9E66D309E298E87E0; Expires=Thu, 22-Apr-2010 20:07:56 GMT; Path=/

Thus the session can be kept live for any time period even when browser restarted.

JavaScript and Multi-Threading

2009-04-21T15:13:00.004+01:00

Several internet sources, such as this, suggest that JavaScript in browser, runs in a single thread that is also responsible for updating browser UI; and JavaScript code is triggered by events. If this is true, it is not a surprise to see why AJAX has to be asynchronous, otherwise network IO will block the browser UI, i.e., make the UI irresponsible.

There were/are several activities to make JavaScript multi-threaded. For instance, the popular JavaScript toolkit Dojo has some support for multi-threading. And there is even a paper talking about multi-threading in JavaScript.

JavaScript now really becomes a serious programming language. Not only there are a lot of libraries/frameworks available so that programmers do not need to write everything from scratch, but also there are a very good programming support, such as profiling (YUI), unit testing (YUI), logging (YUI), debugging (Venkman for Firefox and for Microsoft Script Editor for IE).

Browser Application

2009-04-20T11:33:00.003+01:00

I am working on a browser application, which I already made some good progress and will put online after it is refactored in object oriented JavaScript.

What I called browser application should have the following three characteristics.

A browser application's runtime environment consists of a web browser and internet only. So if you can surf the net, you can run it. Of course nothing prevents it from being deployed on a web server for people to access. Note. in that case, the web server is not required to have any extra support except static HTTP hosting.
A browser application uses AJAX to provide functionalities. With so many powerful AJAX APIs around on the internet, a browser application can have some amazing functionalities.
A browser application can easily be a rich internet application given that now JavaScript, as well as other client side technologies, have already become so rich in terms of presentation capability.

In summary, a browser application is a browser (only) based rich internet application using AJAX.

Check Java Thread CPU Usage

2009-04-06T16:32:00.002+01:00

Today I came across a question: if an application occupies 100% CPU time, and its source code is extremely large so that reading its source code to find what is going on may not be an option; what should we do to find out the problem?

Let's assume the application is written in Java.

First, we can use jstack combined with jps to print out threads' stack trace, which gives us a good idea about which methods are being executed.

We can also use jconsole with some plugin to display threads' CPU usage. The jconsole plugin is based on JTop (

/demo/management/JTop).

Add to Version Control before Commit

2009-04-06T13:15:00.002+01:00

I am using Eclipse 3.4.2 Ganymede plus Subversive 0.7.7. If I try to commit some newly added files, i.e., they are not under subversion control yet, Eclipse simply becomes irresponsive.

New files MUST be added into version control first, before they can be committed.

Invalid Virtual Machine

2009-03-09T15:29:00.004Z

Today I tried to add a VMware VM created by a VMware workstation to my VMware Server 2 on Windows, but failed because it was considered to be invalid. VMware server told me "either the vmx file is corrupted or the VM was created using a newer version of VMware product." Seems both were not the case.

However, it turns out the vmx file could really be "corrupted". In vmx, the first line is

.encoding = "windows-1252"

After change it to be

.encoding = "UTF-8"

The VM is no longer considered to be invalid.

Then after I changed the "numvcpus" from 2 to 1, the VM is able to run on my VMware server, which unfortunately only has one single core CPU.

But interestingly, the original VM can run on a VMware server 2 on Linux.

I also need to change VM's network to be NAT instead of Bridged, and restart VM, to enable networking.

The guideline is that the virtual hardware for a particular Guest OS, such as the number of CPUs and the network type, has to be compatible with the hardware spec of the VMware server installation. For instance, you can not run a 2-CPU VM on a VMware server supporting only one CPU; and if there is no DHCP on the bridged network, then not surprisingly the VM can not acquire an IP if using the bridged network.

JMeter

2009-03-04T16:01:00.003Z

Apache JMeter is a GUI that allows to compose a performance test without writing a single line of code! Basically what needs to do, in order to implement a performance test, is to properly insert test components such as "thread group, "loop control", "http requests" and "performance data graph", into the test plan. Of course the parameters of these test components should be specified, for instance, how many iterations of the http request are wanted. Then simply start it and a curve as well as data is presented!

With the help of JMeter, I am able to tell in a very simple test, when 27 urlrewrite rules are enabled, the OMII-UK website has a throughput of 3030.915 per minute, i.e., 19.796 ms per request; when rules disable, the throughput is 3005.259 per minute, i.e., 19.965 ms per request. In other words, in this setting, where urlrewrite 2.6 is used, urlrewrite rules pose a 0.9% slowdown. Though the credibility of this particular test result needs to be established.

A Portlet Class Loading Scheme

2009-03-04T10:42:00.005Z

According JVM Spec 5.3 Creation and Loading, there are two situations for a class D to load or to initiate loading another class C: either there are references to C in D's constant pool, or D creates C using reflection. In either case, if D was defined by a user-defined class loader, then that same user-defined class loader initiates loading of C.

Let's assume the portal framework is a web application, and it has the following directory structure:

WEB-INF/
    classes/
    lib/
    portlets/
            portlet1/
                    classes/
                    lib/
            portlet2/
                    classes/
                    lib/

The web application class loader W will create a portlet class loader P for each portlet, that has W as its parent class loader and is responsible to load classes from portletx/classes and portletx/lib. W will use P to instantiate a portlet entry point using reflection. From the portlet entry point, all required classes will be then loaded by P, but still following the delegation model. It means W will try to load class first, only after that fails, P will load.

This scheme does not violate Servlet Spec 2.4, which states "The Web application class loader must load classes from the WEB-INF/ classes directory first, and then from library JARs in the WEB-INF/lib directory. Also, any requests from the client to access the resources in WEB-INF/ directory must be returned with a SC_NOT_FOUND(404) response." It does not prevent loading class from elsewhere.

The advantages of such a scheme are:

It takes the view that the portal framework is a web application; its portlets are just its components instead of web applications themselves.
It respects the class loading scheme of the web application server. The portal framework jars can be kept inside the web application, and the portlets share those jars naturally.
It also facilitates deploying and undeploying portlets on the fly, because the control is purely inside the portal web application.

Embed OSGi to Monolithic Application as a Dynamic Plugin Mechanism

2009-02-27T11:43:00.005Z

Eclipse Equinox is used as OSGi framework.

Run Equinox as a standalone application

Read Equinox tutorial.

C:\eclipse\plugins>java -jar org.eclipse.osgi_3.3.0.v20070530.jar -console

osgi> install file:d:\documents\temp\plugins\org.silentsquare.osgi.test.bundle.h
elloworld_1.0.0.jar
Bundle id is 1

osgi> ss

Framework is launched.

id      State       Bundle
0       ACTIVE      org.eclipse.osgi_3.3.0.v20070530
1       INSTALLED   org.silentsquare.osgi.test.bundle.helloworld_1.0.0

osgi> start 1
Hello World!

osgi> ss

Framework is launched.

id      State       Bundle
0       ACTIVE      org.eclipse.osgi_3.3.0.v20070530
1       ACTIVE      org.silentsquare.osgi.test.bundle.helloworld_1.0.0

osgi> close

Goodbye World!

Startup Equinox programmatically

See this blog article: "Starting Equinox from a Java application".

With org.eclipse.osgi_3.x.x.jar on classpath, which is just an ordinary jar file plus extra OSGi information, the code looks like

public class App {
   public static void main(String[] args) throws Exception {
       String[] equinoxArgs = {"-console"};
 BundleContext context = EclipseStarter.startup(equinoxArgs,null);
 Bundle bundle = context.installBundle("http://www.eclipsezone.com/files/jsig/bundles/HelloWorld.jar");
       bundle.start();
 for (Bundle b : context.getBundles()) {
     System.out.println(b);
 }
   }
}

The next step would be to build Grimoires XMLView over OSGi and implement each pair of translators as OSGi bundle. But classloading could be tricky: translators need to use classes in Grimoires. How?

What OSGi Can Do For Me?

2009-02-27T10:43:00.003Z

Two interesting usages of OSGi could be explored in my development context.

One is to use OSGi to componentize an existing complex application. Consider the OMII-UK website. As a web application, it has 3 logical functional units: the old repository that holds projects and releases and is almost obsolete, the new web front that also reads information from database about projects and releases, and the wiki based on JSPWiki. It will be nice to break them into 3 OGSi bundles to take advantage of the dynamics of OSGi bundles.

The other is to use OSGi to implement a dynamic extension framework for an originally monolithic application. In Grimoires, there is an XMLView interface that allows publishing any application domain specific service descriptions as long as they are in XML. Inside the implementation of XMLView, there is a pair of translators for each domain-specific description, translating to and from Grimoires' UDDI+WSDL+Metadata service description model. It will be good that we build XMLView over OGSi framework, and then each pair of translators will become an OSGi bundle. So if we want to support a new type of domain-specific service description, we add a corresponding bundle without bringing down Grimoires. If later this support is no longer needed, we remove the bundle without bringing down Grimoires.

These can be done because OSGi is claimed to be humble: it does not take over the whole JVM, and one Java application can even host multiple OSGi frameworks.

According to OSGi,

If you are developing software in Java then OSGi technology should be a logical next step because it solves many problems that you might not even be aware can be solved. The advantages of OSGi technology are so numerous that if you are using Java, then OSGi should be in your tool chest.

This seems reasonable because componentization looks like a natural approach towards software complexity.

Google Protocol Buffer

2009-02-12T15:08:00.003Z

Google Protocol Buffer can be used to serialise and deserialise structured data, either to/from a file, or to/from network. Protocol Buffer acts as a (network) protocol parser generator. It can generate a parser that understands a specific protocol format. Of course this can be implemented using a general purpose parser generator such as ANTLR.

Several existing solutions are available to address the same problem.

The problematic Java Serialization.
The DIY approach. It is surely painful in particular for complex data structure.
Serialise data to XML. In Java, this is supported by JAXB. The advantages of using XML as transmission format are that, XML is language independent, is platform independent, and is human readable. The disadvantage is the associated performance cost.

According to the Protocol Buffer's tutorial for Java, it has several selling points:

It has bindings for Java, C++, and Python. Thus the data can be transmitted between Java, C++, and Python applications.
Its performance is expected to be good, because not only it is not based on XML, but also it can boost the performance by not using Java reflection.
As long as following certain rules, when the protocol is updated, the new code is compatible with the old code.

java.lang.NullPointerException

2009-02-03T11:26:00.003Z

In one of my previous blogs, I mentioned a map of downloads based on Google Maps API. To create such a map, first I collect IP addresses from where the downloads were initiated, then I use GeoLite City to translate the IP address to a geographic location on the Google map. GeoLite City is a free library with Java API, but of less precision than its commercial counterpart.

It was working fine for many month until recently it threw out the infamous java.lang.NullPointerException. NullPointerException might be the No.1 runtime exception. It is certainly for me.

A little investigation explains why NullPointerException happened. When I passed an IP address to the GeoLite City library, I naively assumed that GeoLite must be able to translate it into a geographic location, thus I did not check whether the Location object returned by GeoLite is null. So I was pulished. In fact, IP address allocation is dynamic, and an outdated GeoLite City library may not contain all the IP address information.

It was not rare I made such a naive mistake. The fundamental principle to avoid NullPointerException is that, when given an object reference, no matter it is from my own code or from outside my code, thinking defensively, reason the possibility that it may be null. To play safe, always check nullity.

An object reference could be given as a returned value of a method invocation, as method arguments when inside a method, or as a field member of a non-null object.

Parallel Array Expected in Java SE7

2008-12-17T15:01:00.004Z

Two Year in Review articles on JavaWorld, "Java in 2008" and "What to expect in Java SE7", are worth reading.

Among the new features expected in SE7, which is due on early 2010, the most surprising one to me is the parallel processing support. Though it is also the one I am mostly looking forward to since I get my PhD working on the topic of parallel computing and cluster computing in Java.

The parallel processing support not only provides a fork and join computing paradigm that might be tuned into a MapReduce mode, but also supports parallel array. Parallel array is a long existing feature in parallel computing languages such as High Performance Fortran. Inspired by HPF, HPJava supports parallel array as well.

The idea of parallel array is quite simple. It represents a large-sized data array, and the array is partitioned to many parts, with each part allocated to a single process. In such a way, each process can work on its own partition of the large array in a parallel way with regard to others, thus achieving speedup.

However, it is still under consideration whether the parallel array will be included as part of the JDK in Java SE 7 or whether it will be released as an external library.

A parallel array code example looks like:

// Instantiate the ForkJoinPool with a concurrency level
ForkJoinPool fj = new ForkJoinPool(16);
Donut[] data = ...
ParallelArray donuts = new ParallelArray(fj, data);

// Filter
Ops.Predicate hasSprinkles = new Ops.Predicate() {
 public boolean op(Donut donut) {
   return donut.hasSprinkles();
 }
};

// Map from Donut -> Integer
Ops.Predicate daysOld = new Ops.ObjectToInt() {
 public int op(Donut donut) {
   return donut.age();
 }
};

SummaryStatistics summary =
 orders.withFilter(hasSprinkles)
       .withMapping(daysOld)
       .summary();

System.out.println("with sprinkles: " + summary.size());
System.out.println("avg age: " + summary.average());

Use Eclipse RAP to Build Rich Internet Application

2008-12-12T11:55:00.005Z

It is a pity that I had not heard of Eclipse RAP (Rich Ajax Platform) when last year I developed Grimoires Browser, an Eclipse RCP based client for the Grimoires Service Registry.

RAP has joined the family of Eclipse UI toolkits, as existing ones such as SWT for Windows, SWT for Liunx, and SWT for Mac. In some sense, RAP can be considered as SWT for Web.

RAP is ued to develop Rich Internet Application (RIA). To develop an RAP-based web application, programmers are still working in Java, in SWT, and in JFace. And RAP makes sure the UI is built on top of HTML, JavaScript and Ajax. Remind you something? Yes, GWT. There are some very good introductory articles [1, 2] about RAP on IBM Developer Works.

It is claimed that in a minimal effort, we can transform an existing RCP application to RAP; or we can even build an application which is largely independent of RAP or RCP. So it can run as both a web application and a desktop application.

No doubt, RAP is the way to go to make Grimoires Browser a web application. It will then totally relieve users the pain to install some software on their machine in order to access Grimoires.

My Safe

2008-12-11T20:44:00.004Z

My Safe is an Eclipse RCP based software I developed last year. As suggested by its name, it is a secure file manager.

It is easy to use. You must input the correct password in order to enter My Safe. Inside My Safe, you can see all directories and files as they originally are. But these directories and files are encrypted in the file system. And they are decrypted on demand in My Safe's interface. The password you use to enter My Safe is the key for encryption and decryption. AES is used as the cipher.

I want to argue that such software as My Safe is necessary even on our personal desktops or laptops. So that other people may possess our computer but can not possess our data.

My Sony laptop does have a secure drive called My Safe as well, but it is not perfect. It is a proprietary technology. It means the data is tightly bound to the Sony laptop. So if the software is broken, there is no way for me to recover the encrypted data. And I can not decrypted the encrypted data on another computer.

Therefore I developed my very own My Safe. And it turns out to be very useful!

Yahoo User Interface

2008-12-11T20:16:00.003Z

Programming in JavaScript was very painful. You have to make sure your code is compatible with all major browsers such as IE and Firefox. Now thanks to well-developed JavaScript libraries, JavaScript programming is much more pleasant.

JavaScript libraries, such as Yahoo User Interface (YUI), jQuery, Dynamic Drive, and Dojo, not only deal with cross browser compatibility issues, but also make common tasks easy.

I have used several components of YUI in developing the OMII-UK website:

CSS Reset (neutralizes browser CSS styles)
CSS Base (applies consistent style foundation for common elements)
CSS Fonts (foundation for typography and font-sizing)
CSS Grids (page layout)
The YAHOO Global Object (base requirement for all YUI components)
DOM Collection (convenience methods for DOM interactions)
Event Utility (event normalization and custom events)
Button
TabView

BTW, browsershots.org allows you to see how your website looks like in tens of browsers.

Classloading of GridSphere

2008-12-11T16:11:00.005Z

JSR-168 portlet applications developed for GridSphere 3.1 has a $CATALINA_HOME/webapps/$PORTLET_WEBAPP/WEB-INF/lib/gridsphere-portletservlet-3.1.jar, which contains a single org.gridsphere.provider.portlet.jsr.PortletServlet class. Originally I thought this class is intended to override the class with the same name in $CATALINA_HOME/shared/lib, according to the rules of Tomcat class loading, by some "clever" Java programmer. Later I found out this class is exactly the same class as in $CATALINA_HOME/shared/lib. So though it replaces the one in $CATALINA_HOME/shared/lib, but it changes nothing. In fact, it does change one thing: the defining class loader.

So if removing $CATALINA_HOME/webapps/$PORTLET_WEBAPP/WEB-INF/lib/gridsphere-portletservlet-3.1.jar, running GridSphere will raise such an exception:

16825:ERROR:(PortletServlet.java:loadJSRPortletWebapp:102)
Unable to create jsr portlet instance: org.gridsphere.gsexamples.portlets.HelloWorld

java.lang.ClassNotFoundException: org.gridsphere.gsexamples.portlets.HelloWorld
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:164)
at org.gridsphere.provider.portlet.jsr.PortletServlet.loadJSRPortletWebapp(PortletServlet.java:87)
at org.gridsphere.provider.portlet.jsr.PortletServlet.service(PortletServlet.java:182)

So basically it complains it can not find the portlet class. Let me explain why.

First org.gridsphere.provider.portlet.jsr.PortletServlet is defined in the portlet web.xml as the generic servlet. Not surprisingly, the portlet application is just a web servlet application in GridSphere.

Second, on line 87 of PortletServlet, there is
Portlet portletInstance = (Portlet) Class.forName(portletClass).newInstance();

It says to use the defining class loader of the current class, i.e., PortletServlet, to load the portlet class. So if PortletServlet is inside $CATALINA_HOME/shared/lib, then there is no way to find the portlet class. PortletServlet has to be put inside $CATALINA_HOME/webapps/$PORTLET_WEBAPP/WEB-INF/lib/.

I am not very convinced that this is a very elegant solution towards portlet classloading. Somehow I have a feeling that it might be better if the portal and portlets stay in the same web application, and the webapp classloader acts as the parent of portlet classloaders. Thus, at least, classloading of portlets is not affected by Tomcat classloading rules.

ABC of P2P

2008-12-10T20:46:00.006Z

On 2007, the top two largest P2P networks (protocols) are BitTorrent and Gnutella. BitTorrent was increasing steadily.

The majority of Gnutella users are using LimeWire. Implemented in Java, LimeWire has a basic version which is open source. It supports BitTorrent as well.

Vuze, originally named Azureus, is a BitTorrent client. Also implemented in Java, Vuze (Azureus) is the No. 2 most popular download in SourceForge.

It is worth mentioning TorrentRelay. It is a purely web-based BitTorrent client. It does not require any software installed on your PC. The BitTorrent downloading is done at the TorrentRelay website. But as a free user, you can not download any file whose size is larger than 800MB.

Implement Principles of Modern OS inside Java Application

2008-12-10T15:27:00.002Z

There are some architectural similarities between a modern OS and an OSGi-based Java application.

For instance, there is a kernel in OS, which provides services to multitasking processes. A shell is there after logging in.

In an OSGi-based Java application, the OSGi acts as the kernel providing fundamental services to bundles. A shell is there after starting up OSGi. Of course multitasking is available in the form of multithreading.

In OS, for the safety and security reason, processes are running in their own memory space.

In OSGi, bundles run in their own code (class) space.

In OS, SELinux can be used to enforce security policies.

In Java, SecurityManager can be used to enforce security policies.

It seems to me that as a trend, principles of modern OS are being implemented in complicated, component-based Java software.

Access UK NGS

2008-12-02T14:18:00.008Z

UK NGS provides national wide grid service. The following is what I have done in order to access NGS facilities.

Apply for a UK e-Science certificate.
Apply for an NGS account.
Install Globus Toolkit 4.2.1. export $GLOBUS_LOCATION=/usr/local/globus-4.2.1; configure --prefix=$GLOBUS_LOCATION; make; make install Because my linux box is running RHEL5, I chose to install from source to make use of openssl 0.9.8. Before I installed GT, I also installed a perl-XML-parser which had been missing. The compilation of GT from source took a couple of hours. So be patient.
Install my UK e-Science certificate: see the section below the heading "Installing your e-Science Certificate and Private Key".
Install UK e-Science CA certificate and signing policy. On the NGS CA certificate page, download CA certificate files 367b75c3.0 and 98ef0ee5.0, and signing policy files 367b75c3.signing_policy and 98ef0ee5.signing_policy. Save them into $HOME/.globus/certificates.
Set up running environment. source $GLOBUS_LOCATION/etc/globus-user-env.sh
Create a proxy certificate. grid-proxy-init -verify -debug This should not display any error message. To display the current proxy certificate: grid-proxy-info. To destroy the current proxy certificate: grid-proxy-destroy
Use gsissh to connect to NGS nodes. gsissh -p 2222 ngs.rl.ac.uk It uses the proxy certificate, so does not prompt for username and password.
Upload the proxy certificate to the MyProxy server. myproxy-init -s myproxy.grid-support.ac.uk -l wjfang Use "-l" to specify a unique username used in the myproxy server. During the execution, you will be prompted to enter a MyProxy pass phrase. Together with the username, the pass phrase allows to access the proxy certificate stored in the MyProxy server.
Now I can log into NGS Application Repository even on a different machine using my MyProxy username and pass phrase.

OSGi: a Dynamic Component System for Java

2008-11-24T14:32:00.004Z

I know OSGi since I finally adopted Eclipse as my IDE for Java four years ago. Recently, two things re-ignite my strong interests in OSGi.

One is GridSphere. GridSphere is a portal framework implementing JSR-168, running on Tomcat. I am evaluating some portlets running in GridSphere. I found out deploying both GridSphere and portlets in GridSphere need to restart the JVM. It is clear to me this is not a desirable feature that an enterprise software should have. The selling point of the portal framework is that the portlets (i.e., the components) can be developed asynchronously and distributed, while can work in harmony in a portal. Let's say we have a portal that contains 25 portlets. Each portlet each year releases two major updates and two security patches. If we need to restart the portal every time when we update a portlet, then we need to restart the portal 100 times every year. This is not a good news to the administrator. Clearly we should have better support for components in architecting server-side software. At least, we should allow individual component be deployed and undeployed dynamically without affecting the remaining part of the system. By the way, 10 years ago in an interview with Huawei, I was asked the question how to patch a software when it is still running.

The other is SpringSource dm server, which uses OSGi. It seems to me OSGi is the answer to the architectural challenges in any complex, server-side, enterprise, or component-based software.

A further look at the OSGi website reveals that OSGi is behind many web application servers and J2EE servers: IBM Websphere, SpringSource Application Server, Oracle (formerly BEA) Weblogic, Sun's GlassFish, and Redhat's JBoss. And not surprisingly, the top two showcases for adopting OGSi are Eclipse and Spring.

Adopting OSGi is claimed to have many benefits, including dynamic update of bundle, security, support for dependencies and versioning, simple API and small footprint. To achieve those, OSGi has the module and service concepts deep in design.

In OSGi, the sharing between modules, like importing jars from other modules and exporting own jars for other modules to use, must be declared explicitly. By default, nothing is shared.

There is a service registry in OSGi. A service can be implemented using any POJO. While the OSGi Alliance publishes the Compendium specifications, which define a large number of standard services, from a Log Service to a Measurement and State specification.

In terms of the implementation, the two most popular ones are Apache Felix and Eclipse Equinox. My next step would be doing some hands on exercise with either of them.

Tweak the Delegation Model of Java Class Loading

2008-11-21T14:24:00.009Z

The Java class loader architecture [1, 2] affords a Java developer a tremendous amount of flexibility in the way that an application is assembled and extended. Basically, class loaders form a hierarchy where the root is bootstrap, who is the parent of sun.misc.Launcher$ExtClassLoader, who is the parent of sun.misc.Launcher$AppClassLoader. When a class loader is asked to load a class, it will first delegate the request to its parent. This is why it is called the delegation model.

But, occalsionally, we do need to tweek this delegation model in some way that is more suitable to our requirement. A good example is in the implementation of Java Servlet Specification. For instance, in Tomcat 6, the web application class loader attempts to load its classes before delegate the request to its parent, the common class loader.

The delegation model is implemented in java.lang.ClassLoader's protected method loadClass(String name, boolean resolve):

protected synchronized Class loadClass(String name,
boolean resolve) throws ClassNotFoundException {
// First, check if the class has already been loaded
Class c = findLoadedClass(name);
if (c == null) {
   try {
   if (parent != null) {
       c = parent.loadClass(name, false);
   } else {
       c = findBootstrapClass0(name);
   }
   } catch (ClassNotFoundException e) {
       // If still not found, then invoke findClass
       // in order to find the class.
       c = findClass(name);
   }
}
if (resolve) {
   resolveClass(c);
}
return c;
}

The following program shows how to override the protected method loadClass to tweak the delegation model. In the program, the classes inside the package "somewhere" will be loaded from somewhere else even the classes with the same name exist in the application classpath. For loading all the other classes, the delegation model is still respected.

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.lang.reflect.Method;

import somewhere.Greet;

public class ClassLoaderTest {
public static class MyClassLoader extends ClassLoader {
@Override
protected synchronized Class loadClass(
String name, boolean resolve)
throws ClassNotFoundException {
  Class c = findLoadedClass(name);
  if (c == null) {
    if (name != null && name.startsWith("somewhere."))
      c = findClass(name);
    else
      c = this.getParent().loadClass(name);
  }
  if (resolve) {
    resolveClass(c);
  }
  return c;
}

@Override
protected Class findClass(String name)
throws ClassNotFoundException {
byte[] buf = loadFromSomewhere(name);
return defineClass(name, buf, 0, buf.length);
}

private byte[] loadFromSomewhere(String name)
throws ClassNotFoundException {
String className = name.substring("somewhere.".length());
FileInputStream fis = null;
try {
fis = new FileInputStream("./class/somewhere/" + className
  + ".class");
} catch (FileNotFoundException e) {
throw new ClassNotFoundException(e.getMessage());
}
byte[] buffer = new byte[8192];
int n = 0;
try {
int c = fis.read(buffer);
n = c;
while (c != -1) {
 c = fis.read(buffer, n, buffer.length - n);
 n += c;
}
fis.close();
n++;
} catch (IOException e) {
throw new ClassNotFoundException(e.getMessage());
}
byte[] rv = new byte[n];
System.arraycopy(buffer, 0, rv, 0, n);
return rv;
}
}

/**
* @param args
* @throws ClassNotFoundException
*/
public static void main(String[] args) throws Exception {
MyClassLoader myClassLoader = new MyClassLoader();
Class clazz = myClassLoader.loadClass("somewhere.Greet");
Object object = clazz.newInstance();
Method method = clazz.getMethod("hello");
// Greet from somewhere, print out "hello world"
method.invoke(object);

Greet greet = new Greet();
// local Greet, print out "good morning"
greet.hello();
// java.lang.ClassCastException:
// somewhere.Greet cannot be cast to somewhere.Greet
greet = (Greet) object;
}

}

In fact, A loaded class in a JVM is identified by its fully qualified name and its defining class loader. Consequently, each class loader in the JVM can be said to define its own namespace.

Therefore, leveraging the Java class loader architecture, components inside a Java process are able to have their own, separated code (class) space. This partly forms the foundation for that individual component can be plugged and played as well as hot swapped without affecting the other components in the same process.

Let Java SSL Trust All Certificates without Violating Security Manager

2008-11-19T17:01:00.009Z

Java SSL by default does not trust self-signed certificate. Wikibooks:Programming reveals a way to allow connection to secure HTTP server using self-signed certificate. The magic looks like:

// Create a trust manager that does not validate certificate chains
TrustManager[] trustAllCerts = new TrustManager[]{
new X509TrustManager() {
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return null;
}

public void checkClientTrusted(
java.security.cert.X509Certificate[] certs, String authType) {
// do nothing
}

public void checkServerTrusted(
java.security.cert.X509Certificate[] certs, String authType) {
// do nothing
}
}
};

// Install the all-trusting trust manager
SSLContext sc = null;
try {
sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new java.security.SecureRandom());
} catch(GeneralSecurityException gse) {
throw new IllegalStateException(gse.getMessage());
}
HttpsURLConnection.setDefaultSSLSocketFactory(
sc.getSocketFactory());

However, HttpsURLConnection.setDefaultSSLSocketFactory(...) will throw a SecurityException (a RuntimeException) if a security manager exists and its checkSetFactory method does not allow a socket factory to be specified. The thrown SecurityException looks like

Exception in thread "main" java.security.AccessControlException: access denied (java.lang.RuntimePermission setFactory)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:323)
at java.security.AccessController.checkPermission(AccessController.java:546)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:532)
at java.lang.SecurityManager.checkSetFactory(SecurityManager.java:1612)
at javax.net.ssl.HttpsURLConnection.setDefaultSSLSocketFactory(HttpsURLConnection.java:308)
at SecurityManagerTest.main(SecurityManagerTest.java:50)

A workaround to avoid such a SecurityException is as below:

URL url = new URL("https://engage.ac.uk");
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
conn.setSSLSocketFactory(sc.getSocketFactory());
conn.getInputStream();

The trick is to use the instance method setSSLSocketFactory instead of the static method setDefaultSSLSocketFactory. The former does not throw a SecurityException.

Note: need to use conn.getInputStream() instead of url.openStream(), otherwise the customised SocketFactory won't be used.

Of course to allow to connect the secure web site, the following permission should be added in the Java security policy file:

permission java.net.SocketPermission "engage.ac.uk:443", "connect";

Use Security Manager to Control Java Web Application Behavior

2008-11-19T15:19:00.009Z

In a Java web application server like Tomcat, we have the Tomcat container and multiple web applications running in the same JVM. It is thus very important to cut the unexpected interactions between applications and between applications and Tomcat. Ideally it is expected the behavior of one web application should not affect the behavior of other applications as well as Tomcat, in a bad way.

One of such unexpected interactions is caused by modifying shared classes and objects, for instance, changing system properties, changing system classes' behavior.

Tomcat 5.5's class loaders are organised as:

      Bootstrap
  |
System
  |
Common
/      \
Catalina   Shared
     /   \
Webapp1  Webapp2 ...

Tomcat 6.0's class loaders are organised as:

      Bootstrap
   |
System
   |
Common
/     \
Webapp1   Webapp2 ...

In both cases, web applications and Tomcat share the classes managed by class loaders Bootstrap, System and Common.

Let's say one web application use the following code to tell Java runtime to use the XSLT implementation shipped with JDK.

System.setProperty("javax.xml.transform.TransformerFactory", "com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl");

Other web applications may choose to use other XSLT implementations in a similar way. If a certain web application's correct behavior is depending on a particular XLST implementation, then we may get a problem, because System.setProperty("javax.xml.transform.TransformerFactory", ...) will change the JVM-wide XSLT implementation.

Therefore it is advisable to use Java security manager to control the permissions granted to web application code.

Note $CATALINA_HOME/bin/startup.sh does not start up a Tomcat with the security manager. To enable the security manager, use $CATALINA_HOME/bin/startup.sh -security. It will append "-Djava.security.manager -Djava.security.policy=..." to the JVM arguments.

A simple Java program testing security manager and policy file

public class SecurityManagerTest {
public static void main(String[] args) throws Exception {
System.setProperty("greeting", "hello world!");
System.out.println(System.getProperty("greeting"));
}
}

Run this program, you will see "hello world!" printed out.

Prepare a policy file called lab.policy:

grant {
permission java.util.PropertyPermission "*", "read";
};

It says any code can only read system properties, but not write.

By the way, the policy file can be created using policytool.

Then run the program like this:

java -Djava.security.manager -Djava.security.policy=lab.policy SecurityManagerTest

You will see an excpetion pop up:

Exception in thread "main" java.security.AccessControlException: access denied (java.util.PropertyPermission greeting write)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:323)
at java.security.AccessController.checkPermission(AccessController.java:546)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:532)
at java.lang.System.setProperty(System.java:727)
at SecurityManagerTest.main(SecurityManagerTest.java:11)

See the security manager is working. It is important to give the right location for the policy file. Because when the security manager is enabled, by default no permission is granted. So any permission will be denied except those defined in the policy file. In case the policy file cannot be found, then the code is given no permission at all.

Update RHEL5

2008-11-14T16:34:00.002Z

See here (within intranet, may require authentication) for how to register RHEL4 for up2date and RHEL5 for yum.

After that, then can use yum update for a complete update.

To install a certain package: yum install package_name

To see what a package does: yum provides package_name

To find packages containing some keyword: yum list | grep keyword

To check already installed packages that contains some keyword: rpm -qa | grep keyword

Make a Java-enabled Virtual Machine Using Ubuntu JeOS

2008-11-11T22:19:00.007Z

As suggested in the best practice to build a virtual appliance by VMware, Ubuntu JeOS is used as the operating system. In around 100MB, it provides a just enough OS. After installed, its VMware virtual disk file size is about 380MB.

By default, Ubuntu does not provide a root password. The root privilege is carried out using the "sudo" command. To enable root login: sudo passwd root.

Also, see how to prepare a virtual appliance from the Ubuntu community.

Adding a new virtual disk

To add a new virtual disk, using the "Add Hardware" command in VMware web interface.
Need to restart the virtual machine to detect the new virtual disk.
Let's say the newly added hard drive is /dev/sdb. Use the following command to partition it: fdisk /dev/sdb. To create a single new partition, the entire size of /dev/sdb: n ENTER p ENTER 1 ENTER (default) ENTER (default) ENTER w ENTER.
To format it: mkfs.ext3 /dev/sdb.
To mount it on an existing location, say /data: mount /dev/sdb /data.

See how to format a new hard disk on ubuntu forums.

Making a Java appliance

The next step will be to make it a Java appliance, with JDK, ant as well as Tomcat installed, running some Java application like Grimoires.

Ubuntu JeOS does not pre-install Open SSH server. Use the following command to install Open SSH: apt-get install openssh-server.

Then install

JDK 1.6.0_10
Apache Ant 1.7.1
Apache Tomcat 5.5.27
Grimoires 2.0.0

The used disk space shown by "df" inside virtual machine is about 675MB.

Setting up NAT

Assume the virtual machine has been configured to support NAT. Add the following line to /etc/vmware/vmnet8/nat/nat.conf, under the [incomingtcp] section:

6660 = 172.16.59.132:8080

where 172.16.59.132 is the IP address of the virtual machine.

According to the VMware server user guide, clicking "Refresh Network List" in the Virtual Infrastructure web interface should bring up the modified network configuration. But it does not work in my case. I have to restart VMware to enforce the new NAT configuration: service vmware restart. (Better shut down vm before restart vmware server!)

After restart VMware, and make sure the port 6660 is opened in the firewall of the host machine, access http://hostname_of_the_host:6660/grimoires from another machine.

It seems that vmware server runs a separate daemon to handle the address translation when vm is set to use NAT. So this blocks iptables' NAT configuration.

Cloning the Virtual Machine

This is something a little bit tricky.

I used the conventional way to clone it: copy the vmdk and vmx files; then add the clone virtual machine to the inventory ("Add Virtual Machine to Inventory"); when asked whether you copied it or moved it, answer copied it.

The clone was able to start up. But there was no eth0. The routing table was empty (shown by route). There was an eth1 with the correct MAC address but without IP information (shown by ifconfig). Not surprisingly, TCP/IP was not working.

My judgement was that there was no reason for the virtual network card to go wrong, so it should be due to some configuration issue.

Hinted by a post about adding a second network card to Ubuntu, I used dmesg | grep eth to check the kernel messages on booting. Then I found an interesting message: "udev: rename eth0 to eth1". After a little bit googling and investigation, I solved the problem!

udev is the device manager for the Linux 2.6 kernel series. In /etc/udev/rules.d/70-persistent-net.rules, there were two lines:
SUBSYSTEM=="net", ...... ATTR{address}=="00:0c:29:07:6f:37", ...... NAME="eth0"
SUBSYSTEM=="net", ...... ATTR{address}=="00:0c:29:cf:c8:64", ...... NAME="eth1"

The MAC address in the first line is the one of the original vm, while the MAC address in the second line is the correct MAC address of the clone. Clearly, the first line has no effect, because there is no such a device in the system. And the second line adds the network card as eth1.

Looking at /etc/network/interfaces, it only defines lo and eth0. eth0 is defined like this:
auto eth0
iface eth0 inet dhcp

There is no wonder why only eth1 was visible but did not support TCP/IP. After commenting out the first line and changing eth1 to eth0 in the second line in /etc/udev/rules.d/70-persistent-net.rules, and restarting the system, TCP/IP is working!

Summary

This is a convenient way to prepare a small footprint Virtual Machine for running Java. It can be used to deliver Java software, or as a test or evaluation environment for Java product.

OpenSSL

2008-11-06T22:00:00.003Z

Recently I used the openssl utility to generate key pairs and certificates. For example,

To generate a private key: openssl genrsa -out ca.key 2048
To create a self-signed certificate: openssl req -new -key ca.key -x509 -days 365 -out ca.crt
To create a certificate signing request: openssl req -new -key temp.key -out temp.csr
To create a certificate from a certificate signing request: openssl x509 -req -in temp.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out temp.crt
To display a certificate: openssl x509 -text -in temp.crt
To display the content of a pkcs12 formatted certificate (the displayed private key and certificate are in PEM format, which can be used in the above commands): openssl pkcs12 -in old_uk_escience.p12 -out old.txt
To convert from pkcs12 format to PEM format: openssl pkcs12 -in cred.p12 -out cert.pem -nodes -clcerts -nokeys, openssl pkcs12 -in cred.p12 -out key.pem -nodes -nocerts
To create pkcs12 format certificate using PEM format private key and certificate: openssl pkcs12 -in temp.crt -inkey temp.key -out temp.p12 -export

Self-Decrypting HTML Page

2008-11-06T15:04:00.005Z

This is something worthy doing. Let's call it self-decrypting HTML page.

It is a publicly accessible, encrypted HTML page, that can be hosted in any website, but can only be decrypted by authorized persons who know the key. JavaScript can be used to perform encryption and decryption locally at the browser. Thus what is transmitted on the net and stays at the public website is the encrypted information, which guarantees the confidentiality of the information. Some symmetric key algorithm is preferred.

A self-decrypting HTML page generator is needed to convert the to-be-protected information to a self-decrypting HTML page.

It is noted that a United States Patent 7003800 has been filed on "Self-decrypting web site pages". It seems it talks about a method pertinent to a web site.

A self-decrypting email utility performs the similar task, which uses RC4 as the cipher. Here is the JavaScript code for RC4.

Compared with another web-based confidential information system, i.e., some information is hosted on a HTTPS server and is protected by some authentication method, self-decrypting HTML page does not require an authentication method, thus no need for user management. All needed to access the confidential information is the URL of the self-decrypting HTML page and the key.

Will Google App Engine be a good platform for this? Will it be possible to use Google AdSense to generate revenue from this application?

Handle Chinese in Java

2008-11-04T14:40:00.003Z

(This is one of my old Google Notes.)

In Java, Reader and Writer are used to handle character (char) stream, and InputStream and OutputStream are used to handle byte streams. InputStreamReader and OutuptStreamWriter are bridges between byte streams and character streams.

Characters can have many different coding schemes, such as ASCII, GB2312, UTF-8 (Unicode Transformation Format), when they are represented in bytes. While characters in Java, char or String, are Unicode only.

A Charset is a named mapping between sequences of sixteen-bit Unicode, which is the character representation in Java, and sequences of bytes. A Charset knows how to convert a byte sequence to a (Unicode) char sequence, and vice versa, following the standard it implements.

Not surprisingly, both InputStreamReader and OutuptStreamWriter can be configured to use a specified Charset, but there is no concept of Charset in Reader and Writer.

The following Java program demonstrates how to handle Chinese in Java.

import java.io.BufferedWriter;
import java.io.OutputStreamWriter;

public class ChineseTest {
public static void main(String[] args) throws Exception {
String chinese = "\u4eca\u65e5\u83dc\u6839\u8c2d"; // 1
System.out.println(chinese); //2
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(System.out, "GB2312")); //3
bw.write(chinese); //4
bw.close(); //5
}
}

Code comments:

This is the Unicode for 今日菜根谭, generated by "native2ascii -encoding GB2312 c.txt", where the content of c.txt is 今日菜根谭 encoded in GB2312. The utility native2ascii converts a file with native-encoded characters (characters which are non-Latin 1 and non-Unicode) to one with Unicode-encoded characters.
Can't print 今日菜根谭, by using the platform's default Charset.
Create a Writer using the GB2312 Charset.
Now we can print out 今日菜根谭.
Flush the output. Required.

Virtual DOM

2008-11-04T14:09:00.001Z

(This is one of my old Google Notes. Virtual DOM is a small piece of software I implemented.)

Complying with the W3C DOM interface, Virtual DOM is capable of representing in memory a large XML data which can not be represented using other DOM implementations such as Xerces. Virtual DOM is aiming at allowing off-the-shelf XPath engines such as Jaxen to process XML DOM representation that is too large to be held in memory otherwise. To support such a goal, Virtual DOM must be told how to load DOM element, and then it uses SoftReference to cache the loaded element. To put it in a simple way, the Virtual DOM elements can be garbage collected when Java heap space becomes scarce, and later be reloaded on demand.

The problem of Xerces DOM implementation

Each node has references to its parent, children, and siblings. So as long as there is a single reference to any node of the DOM tree, any parts of the tree can not be garbage collected.

Implementation of Virtual DOM

Virtual DOM adopts a forest mode, where there is a virtual document and a virtual root element. The virtual root element has a number of child elements, which is implemented as CollectableElement extended from CollectableNode, each representing a different DOM document.

Each Virtual DOM node, called CollectableNode, has a reference to DocumentCache, which in turn has a SoftReference to the cached owning document. DocumentCache also has enough information to load and reload the owning document whenever necessary.

Each Virtual DOM node has information specifying how to locate this node from the root node of the owning document.

Each Virtual DOM node also has a SoftReference to its corresponding concrete DOM node for convenience purpose.

The CollectableElement that is the direct child of the virtual root element has an index indicating which child it is in terms of the virtual root element. Thus it is able to retrieve its next sibling.

All exported DOM references should be of Collectable* instead of the default DOM ones to prevent DOM node references from being held externally.

Test

With 64 MB heap, using a revised Jaxen 1.1 (one of Jaxen 1.1's methods unnecessarily retains object references.), Virtual DOM is able to deal with at least 250,000 instances of a certain XML document. In a comparison, Xerces DOM runs out of memory at 39,000 instances.

(It is called VirtualDOM in my Eclipse projects.)

JSPWiki ACL Filter

2008-11-04T12:00:00.002Z

I am working on the OMII-UK website. JSPWiki has been adopted as the web content management system as well as a wiki in the website. JSPWiki has been customised with an OMII-UK template, and authentication and authorization modules. There are many users, so In-page ACLs are adopted to protect some page from unauthorised editing. For instance, the following ACLs say only members of staff group can view and edit the page containing this ACL:

[{ALLOW edit StaffGroup}]

Because members of staff group can edit this page, members of staff group can also edit the ACL, which is nothing more than a JSPWiki markup. This causes some potential security flaw: any member of staff group can edit ACL, e.g., by mistake, and thus violate the intended access control of this page. Ideally, ACL, though residing in a page, should be treated differently from the other page source.

Thus an ACL Filter is introduced to only allow users with AllPermission to create/edit/delete in-page ACL. For instance, in the above example, even any member of staff group can edit the page, but only users with AllPermission can change the ACL to something other than the above ACL.

Essentially, the ACL Filter is trying to separate two concerns, content and access control over content, which are originally mixed up in the wiki markup. With the ACL Filter, content and access control over content are treated differently: any one authorised by ACLs can edit content, but only some certain super users can edit access control.

This work has been contributed back the JSPWiki community. See here.

Spring JUnit Test and Rollback DB Transaction

2008-11-04T11:00:00.002Z

In Spring, doing unit test with JUnit4 can be as simple as this:

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration
public class HibernateDaoTest {
@Autowired
protected RepositoryHibernateDao repositoryHibernateDao;

@Test
@Transactional
public void submitProject() {
Project newp = new Project();
newp.setStatus("PENDING");
int pid = repositoryHibernateDao.save(newp);
assertTrue(pid > 0);
Project p = repositoryHibernateDao.getProjectDetail(pid);
assertEquals(p.getName(), name);
assertEquals(p.getSubmitUser(), owner);
repositoryHibernateDao.delete(p);
}
}

With the annotation @Transactional, when the test is finished, the database is supposed to roll back to the state before the test. But this does not happen in my Spring/Hibernate/MySQL setting.

It turns out to be that in order to support transaction, the MySQL table must be an InnoDB table.

Usually the tables are MyISAM ones, which are non-transactional. The MyISAM table provides high-speed storage and retrieval, as well as fulltext searching capabilities. It is supported in all MySQL configurations, and is the default storage engine unless you have configured MySQL to use a different one by default.

On the other hand, the InnoDB and BDB storage engines provide transaction-safe tables. InnoDB is included by default in all MySQL 5.0 binary distributions. Here describes how to
Convert a MyISAM table to innoDB. Basically, what needs to be done is: ALTER TABLE ... ENGINE=INNODB

Map of Downloads

2008-11-03T14:31:00.000Z

I have used Google Maps API to display on the map where the downloads of OMII-UK software are from. See the downloads for Grimoires 2.0.0. By clicking on the balloon, more geographic information about the download will pop up. For instance, I know there is a guy in Changsha, Hunan, China has downloaded Grimoires 2.0.0. All the other downloads are from UK, France and Germany.

Hibernate Performance

2008-11-03T14:27:00.002Z

(This is one of my old Google Notes.)

I have used YourKit to benchmark the performance of Hibernate some time ago.

In my database, I have two tables: Project and Release. Project is associated with release in a one-to-many relationship. In my setting, a SQL statement using JDBC costs from several milliseconds to ~30 milliseconds; serving a JSP costs from 1 second to 3 seconds in the first run, then from tens of milliseconds to ~250 milliseconds in the later run.

In Hibernate,

Getting a single project (as well as its releases) costs 234ms in the first run, and 14ms in average (excluding the first run);
Getting all projects costs 547ms in the first run, and 96ms in average;
Getting some partial information of all projects, costs 282ms in the first run, and 15ms in average.

The reason for that the first run is much slower, is because Hibernate does bytecode instrumentation to generate proxy object on the first touch of ORM, which is quite expensive. However, the late runs approximate JDBC's performance.

Some Hibernate tips:

Chapter 19 of the Hibernate reference describes how to improve its performance.
Use set instead of list. List uses the id as index, thus creates a large data structure with lots of empty cells.
Associated objects are initialized in a lazy way. They need to be touched to bring into memory.

Dependency Resolution

2008-11-02T14:25:00.002Z

(This is one of my old Google Notes.)

In the software architecture, one component (caller) is relying on some services provided by another component (callee). Before the caller can use the service of the caller, the caller must hold an instance of the callee. There are several ways to do it.

First, the caller can create an instance of the callee. Sometimes, only one instance of the callee is preferred to exist in the whole system, because it is unique or it is expensive to create, e.g., the database manager. In that case, the singleton pattern can be used. But there are some shortcomings with the singleton pattern. First, because the callee has to implement the singleton pattern and the callers have to invoke the singleton pattern in a proper way, the code of solving dependency is scattering around the whole system. Second, it is inflexible to replace a callee component with one of the same interface.

Second, the caller can go to a "service locator", then use, for instance, a JNDI directory to locate callees. Though the service locator maintains a central place of managing all service providers (callees), the caller still needs to know how to talk to the service locator.

Third, this is state-of-the-art: dependency injection, which is implemented in many frameworks, such as Spring and Google Guice. Compared to the above two approaches, the advantages of dependency injection are obvious. Both caller and callee can be POJO (Plain Old Java Object), with no special control implemented. And there is a single control point. In Spring, that is the Spring XML configuration file. Thus the dependency is solved in a declarative way! Nowadays we always prefer a declarative way to a programmatic way to do things.

In a service-oriented architecture, service workflow is a multi-component system. Any component service can be a caller, a callee, or both. There is some similarity between dependency injection and how a workflow is composed: both have a central control point for solving dependency in a declarative way. In workflow, that is the workflow description file, e.g., written using BPEL. On the other hand, in service-oriented architecture, the service registry shares a similar idea as the service locator pattern.

Tomcat Troubleshooting

2008-10-31T09:56:00.007Z

The OMII-UK website is a Java web application running in Tomcat, which sits behind an Apache web server. Several methods are there for monitoring and troubleshooting it.

Monitoring

First, a simple web monintor is running on another machine, and checking the contents of several important web pages every 15 minutes. If anything goes wrong, I will get an email notification. Of course, this task can also be done by Nagios. The advantage of using my own web monitor instead of a complicated monitor system like Nagios, is that I have full control over what to inspect. For instance, I am able to detect that although Tomcat is still OK but the database connection is down.

Second, in the Java web applicaiton, which is based on the Spring Framework, Spring AOP (Aspect Oriented Programming) is leveraged to monitor the performance of all servlets. If it takes any servlet more than a pre-defined threshold to serve a request, I'll get an email notification. The threshold, currently set to 200ms, is defined in Spring XML configuration file.

Third, Tomcat is started with "-Dcom.sun.management.jmxremote", which enables local jconsole connection. JDK 1.5 is used. In JDK 1.6, jconsole support is default, so no need to set a JVM option to enable it.

What if something goes wrong ...

Check Apache web server only. http://hostname/server-status. You may need to modify httpd.conf to enable this.
Check Tomcat only. http://hostname:8080.
Check jconsole->Memory. See if JVM runs out of memory.
Check Tomcat logs, Apache logs and database log.
Last but not least important, use Chainsaw to check log4j.xml. Chainsaw is a GUI, which makes it easy to browse log4j's log. The log4j configuration may look like:

log4j.appender.FileLog = org.apache.log4j.RollingFileAppender
log4j.appender.FileLog.MaxFileSize = 10MB
log4j.appender.FileLog.MaxBackupIndex = 14
log4j.appender.FileLog.File=$CATALINA_HOME/logs/log4j.xml
log4j.appender.FileLog.layout = org.apache.log4j.xml.XMLLayout
log4j.rootCategory=WARN,FileLog

Let's say there is a memory leak or something inside the Java web application is wrong ...

There are various JVM options that allow us to debug or profile Java web application:

Frequently used server VM options: -server -Xms512m -Xmx512m
jconsole support: -Dcom.sun.management.jmxremote
Dump heap on running out of memory: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/root/heapdump
Enable debugging: -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=8000
Enable YourKit profiling: -agentlib:yjpagent

YourKit is a very good Java profiling tool, and it has a free licence for open source projects. That is what I am using. To enable profiling manually, see here.

On Windows, 32-bit, add \bin\win32 to the (SYSTEM) PATH.
On Linux x86, 32-bit, add /bin/linux-x86-32 to the LD_LIBRARY_PATH.
Check whether it is working: java -agentlib:yjpagent=help
Profiling: java -agentlib:yjpagent
Do not forget to force garbage collection before capture memory.

I also have a stress test tool that replays saved Apache/Tomcat access logs.

Pass Apache Authentication to Tomcat

2008-10-30T10:57:00.007Z

It costs me several hours to figure this out. So I think it is worthy writing it down.

I have an Apache web server (2.2.3) sitting in front of Tomcat (5.5.27). In Apache configuration, I have:

<location>
AuthType Basic
AuthName "Secure Service"
AuthUserFile /etc/httpd/conf/user.db
require valid-user
</location>

If the authentication succeeds, the HTTP request is passed to Tomcat by mod_proxy_ajp:

ProxyPass /secure/ ajp://localhost:8009/secure/

In Tomcat server.xml, I disable the Tomcat authentication in the AJP connector (both tomcatAuthenticaiton and request.tomcatAuthentication work):

<Connector port="8009" enablelookups="false" redirectport="8443" protocol="AJP/1.3" address="127.0.0.1" tomcatauthentication="false">

If the authentication succeeds, Apache will create an HTTP head:

REMOTE_USER = omii

But in Tomcat, I do not see the REMOTE_USER header. Instead, I see

authorization = Basic b3ip9kd9dkekd9

It turns out that Tomcat puts the Apache authentication information in the form of a user principal, which can be accessed by the following code inside a JSP page:

java.security.Principal pr = request.getUserPrincipal();
if (pr != null) String r = pr.getName(); // r.equals("omii")

Shibboleth: Federated Trust

2008-10-30T09:16:00.006Z

Having been developed for around 3 years, Shibboleth provides web single sign on and attribute exchange, which together build up a trust federation.

Like many ideas in computer science, Shibboleth is also about separation of concerns. Two pairs of concerns have been identified and separated in architecting Shibboleth: Service Provider (SP) has been separated from Identity Provider (IdP), and identity has been separated from attributes.

By separating IdP from SP, an SP, i.e., a Shibboleth protected web application, is free of maintaining a user database and perform authentication, which are delegated to some appropriate IdP. Though SP may still needs to enforce local authorization decisions. the WAYF (Where Are You From) service is leveraged as a means to locate a suitable IdP that is able to authenticate a user who want to access SP.

By separating attributes from identity, flexibility is achieved on how to represent an authenticated user. Let's say I try to access an SP and I use University of Southampton's IdP to authenticate myself. As a result of the authentication, I will get two attributes: one is eduPersonPrincipalName, i.e, my name; the other is eduPersonRole, in my case, that is research staff. My IdP does not need to give out my eduPersonPrincipalName to the SP. Instead giving out the eduPersonRole attribute might sufficiently entitle me to the service. In this way, my privacy is somehow protected even when I am granted the access.

Shibboleth is built over SAML. If you look at Shibboleth technical specs, SAML specs add up to several hundred pages, while the Shibboleth architecture specification has only 19 pages. SAML defines core (XML schema for SAML assertions and protocol message elements), protocol (what is transmitted), binding (how the protocol messages are transmitted) and profiles ( a concrete manifestation of a defined use case using a particular combination of assertions, protocols, and bindings).

Shibboleth promotes a trust federation. It is the responsibility of the federation that decides which SP and IdP can join the federation. Thus the trust between a user and an SP is maintained by the federation. A peace of mind for users.

OpenID is another technology for web single sign on. Different from Shibboleth, there is no such a federation in the OpenID architecture. Thus it is down to users who decide whether they should trust a service provider and use their OpenIDs on the service provider's website. I am not happy with it. I also do not think it is a very good idea to use a single OpenID to access websites with different information confidentiality levels, such as online banking, webmail and a simple web-based game site. Clearly online banking has the highest level in terms of information confidentiality, the game site has the lowest, and webmail sits between them.

In UK, UK Federation manages a Shibboleth federation for eduation can research. Here is the current membership, i.e., who provides SP and IdP. And here is the list of all available services that support UK Federation managed Shibboleth authentication

Edsger W. Dijkstra in his 1974 paper "On the role of scientific thought" explains why separation of concerns is so important:

Let me try to explain to you, what to my taste is characteristic for all intelligent thinking. It is, that one is willing to study in depth an aspect of one's subject matter in isolation for the sake of its own consistency, all the time knowing that one is occupying oneself only with one of the aspects. We know that a program must be correct and we can study it from that viewpoint only; we also know that it should be efficient and we can study its efficiency on another day, so to speak. In another mood we may ask ourselves whether, and if so: why, the program is desirable. But nothing is gained --on the contrary!-- by tackling these various aspects simultaneously. It is what I sometimes have called "the separation of concerns", which, even if not perfectly possible, is yet the only available technique for effective ordering of one's thoughts, that I know of. This is what I mean by "focusing one's attention upon some aspect": it does not mean ignoring the other aspects, it is just doing justice to the fact that from this aspect's point of view, the other is irrelevant. It is being one- and multiple-track minded simultaneously.

Use Virtual Machine to Evaluate Software

2008-10-29T21:21:00.002Z

Here at OMII-UK we need to evaluate software generated in the commissioned software project from time to time. For instance, now it is my duty to evaluate SPAM-GP, some JSR-168 portlets for security stuff. I am using VMware Server to do this job. I think it is great! Basically, I build a virtual machine, in which I install, configure and test the software to be evaluated. Benefits by doing so are obvious:

It is a VM, where the software is running. So it has all advantages as we dedicate a physical machine to the software. This is a working instance, and we can keep it as long as we wish. It is also easy for someone else who wants to see a demo or to play with it.
Compared to a physical machine, the VM only costs around 3GB disk space. Sounds quite large, but it is little in my 200GB harddisk.

VMware Server supports snapshot. A snapshot is a persistent state of the whole VM. It can be used to save the current state of the VM, which we are happy with. Then we can carry on manipulating the VM and come back to the snapshot whenever we want. But unfortunately, in VMware Server, only one snapshot can be taken for each VM. A newly taken snapshot will simply replace the previous one.

My machine has 1GB memory. So sometimes it is slow to run VMware server, particularly when two VMs are running at the same time.

WTP Europa Release

2007-07-22T11:13:00.000+01:00

Eclipse Europa is the annual release of Eclipse projects of 2007. Here are some of my experience about it. Since I am using WTP mostly, my experience is mainly concerned with WTP.

Interesting, my WTP 2.0 can not start up with -Xmx1024m. But it is able to start up with a smaller maximum heap size, for instance, -Xmx768m. While my other Java application, such as Tomcat, has no problem with -Xmx1024m. I am using JDK 1.6.0_01. I posted a message at the eclipse.webtools newsgroup. I will follow this issue.

It is desirable that WS explorer has an extension point to support application domain service description document (non WSDL). WS explorer can be used as a UDDI browser. In UDDI data model, the service entity can be associated with a tModel, which is simply a link to some non-registered document that further describes the service, for instance, the WSDL of the service. WS explorer supports to retrieve the WSDL, visually display it, and even invoke its operations.

I am working on the Grimoires project, which is an extended UDDI service registry supporting metadata annotation and service discovery by interface signature. In our practice, we very often see that users design their own service description language, instead of WSDL, to describe their services. For instance, the myGrid project, an influential service-oriented bioinformatics experiment environment, uses its own Feta schema. GLUE is a popular schema to describe services and resources in grid computing.

explorer has an extension point to support application domain service description document (non- explorer to open it; if it points to some non-This is understandable. First, it is cheap to design an XML schema for their XML-based service description language. Second, it improves usability with a DSL (domain specific language) in place. According to UDDI' data model, the URL of domain specific service description documents can be put into a tModel, then associated with the service entity. Thus it is desirable that WSWSDL). That is, if the tModel points to WSDL, then WS explorer uses its embedded WSDLWSDL document, the WS explorer uses some extension registered corresponding explorer to visually handle it. Somehow UDDI's tModel can be considered as an extension point of UDDI data model, because it allows any document with a URL to be associated with a service.

The benefit of such an extension point in WS explorer is to improve usability by supporting domain-specific service description documents, which further enhances the expressiveness of service description: whether the description document is WSDL or non-WSDL, standard based or domain specific, current supported or emerging, all can be handled by WS explorer.

Another desirable feature is about being able to print visualized XSD/WSDL supports visualizing . WTPXSD/WSDL. The visualized result is very informative and is a good source to be put into documentation. But this is not supported a couple of years ago. It is very good that now WTP 2.0 supports printing visualized XSD! (Although I am not sure exactly since which version WTP has implemented this.) But the printed visualized XSD loses its icons standing for XSD attribute and element. And it still does not allow to print visualized WSDL. I am not sure whether there is some technical difficulty for printing contents of arbitrary editors.

"An open source project would be nothing without a supporting community"

2007-06-14T20:11:00.000+01:00

"An open source project would be nothing without a supporting community."
- from Eclipse' FAQ

Selling Data Through Web Service

2007-05-30T21:07:00.000+01:00

StrikeIron, "data as a service", provides a marketplace where people can put web services or purchase hits on web services that make available data which value is dependent on time, for instance, foreign currency exchange rates.

The key of business model is to sell valuable real time streamed data. And WS is adopted to simplify the data service implementation. The idea is cool and simple. And seems it works. I mean StrikeIron does have quite a few customers.

By googling "web serivces marketplace", I am aware Amazon has filed a patent of web services marketplace on 2005.

If someone is able to build up valuable data by doing data mining on Internet with the help of tools like google, and use StrikeIron to host a WS to provide the data, then (s)he can make money!

AJAX Enabled GrimoiresScript Interpreter

2007-05-29T10:19:00.000+01:00

I implemented a toy language to handle Grimoires registry, which now I renamed as GrimoiresScript. Now I created a web-based GrimoiresScript interpreter, using AJAX to communicate between HTML page that takes in user input, and server-side JSP that invokes GrimoiresScript interpretation engine. The implementation follows the example in w3schools' AJAX tutorial.

AJAX is nothing but, or mostly a JavaScript RMI, which enables HTML pages to invoke remote methods.

Web Services Explorer in Eclipse

2007-05-25T23:09:00.000+01:00

Web services explorer in eclipse can be used to browse UDDI registries, to parse services' WSDLs, and even to invoke service operations by sending them SOAP messages. It is implemented as a web application running on eclipse' embeded application server. So I am thinking to turn it into a standalone web application to act as Grimoires' web-based interface. As Chris Brealey pointed out, the WTP team also has this idea, but it is not easy to do this and the WTP team simply has no resource to be allocated on this. I played web services explorer today. It relies on quite a lot eclipse plug-ins. Even it could be turned into a standalone web application, it would be of a large footprint.

Monitor Memory Usage in Java

2007-05-24T21:20:00.001+01:00

There is no "sizeof" in Java. Originally there was no way to check or estimate how much memory has been used in Java. But since Java 1.5, we can use the Monitoring and Management facility to monitor memory usage. The following code


MemoryMXBean mbean = ManagementFactory.getMemoryMXBean();         
System.out.println(mbean.getHeapMemoryUsage().toString();
System.out.println(mbean.getNonHeapMemoryUsage().toString();

will print out


init = 0(0K) used = 319360(311K) committed = 2031616(1984K) max = 66650112(65088K)
init = 8585216(8384K) used = 2379416(2323K) committed = 8880128(8672K) max = 100663296(98304K)

MemoryMXBean is also able to send out notification when some memory usage threshold is met.

Call Google AJAX Search Service in Java

2007-05-24T10:11:00.000+01:00

Google has both a SOAP style service and an AJAX style for search. But it seems google focuses on the AJAX style rather than the SOAP style. For instance, the AJAX style has more functionalities than the SOAP style.

At the client side, the AJAX style is implemented as Javascripts embeded in a HTML page, which makes it impossible to be called from JDK script engine, at least not in a straightforward way.

Davanum Srinivas has published an interesting blog on how to call google ajax search service in Java. He used an reverse engineering approach to capture the url that the javascript sent to the google server, by some TCP/IP monitor. By manipulate the captured url, it is able to call google search in Java. The url looks like:

"http://www.google.com/uds/GwebSearch?"callback=GwebSearch.RawCompletion&context=0&
lstkp=0&
rsz=small&
hl=en&
sig=8656f49c146c5220e273d16b4b6978b2&
q=Axis2&
key=xxxxxxxxxxxxxxxxxx&
v=1.0"

When rsz is set to small, four records are returned; when large, eight are returned.

The search result contains result objects in JSON format, which can be parsed using some JSON parser.

It is easy to call google AJAX search service in Java. The challenge is when we have the capability to google repeatedly, untiredly, in brute force, or in a massive way, anything interesting can we achieve then?

Bridge GLUE LDAP schema and GLUE XML schema

2007-05-23T14:08:00.000+01:00

GLUE is an information model to describe Grid resources. It has both an LDAP schema and an XML schema as its concrete data models. Usually GLUE data is in LDAP DIT in Grids. I need to read the LDAP data, convert to XML data according to GLUE schema, and then publish into Grimoires registry. What I am doing is:

Use JAXB to generate Java bean classes according to GLUE XML schema;
Parse GLUE LDAP schema with the help of an ANTLR-based LDAP schema parser, and generate Java code that converts LDAP entries to Java beans using the classes generated in step 1;
Use JNDI to read GLUE LDAP DIT, call the code generated in step 2 to create beans that are further serialized to XML (in GLUE schema) by JAXB;
Publish GLUE XML to Grimoires through the XMLView interface, which takes the configuration of XSLT between GLUE schema and Grimoires schema.

A Toy Language to Handle Grimoires Registry

2007-05-23T13:44:00.000+01:00

I created a toy language to handle Grimoires registry using ANTLR. It also has a simple GUI to type and run the script. So I can type something like:

publish business (name="BusinessAA", desc="a business") -> $bk.

publish service (name="ServiceAA", desc="a service",bkey=$bk, wsdl="http://a/b/c.wsdl", address="http://a/b/") -> $sk.

attach metadata (type="http://www.grimoires.org/QoS", value="very good") to service (skey=$sk).

If run this script, it will first publish a business with name and description, and save the business key to a variable $bk, which will later be used in publishing service. It then will publish a service and attach a metadata to the published service.

Google launches TiSP — Free wireless broadband

2007-04-02T16:15:00.000+01:00

Google launches free wireless broadband which makes use of local plumbing and sewage systems.

The reason it is free is "We believe that all users deserve free, fast and sanitary online access. To offset the cost of providing the TiSP service, we use information gathered by discreet DNA sequencing of your personal bodily output to display online ads that are contextually relevant to your culinary preferences, current health status and likelihood of developing particular medical conditions going forward. "

Sun's Blackbox

2007-03-16T10:51:00.000Z

曾经历的桂林山水

2007-03-15T16:21:00.000Z

[摘自www.popyard.org] 国家机密文件：最新流行官场口诀

2007-03-14T14:42:00.000Z

现代企业职位新解：

总是在裁人，简称总裁；老是板着脸，所以称老板；总想监视人，

所以叫总监；经常没道理，就叫经理；让领导秘密舒服，称为秘书。

四清四不清：

开啥会不清楚，开会坐哪清楚；

谁送礼不清楚，谁没送清楚；

谁干得好不好不清楚，该提拔谁清楚；

和谁睡不清楚，睡觉干什么清楚。

当代八乱：

大棚把季节搞乱了，关系把程序搞乱了，

级别把能力搞乱了，法官把法律搞乱了，

公安把治安搞乱了，银子把官场搞乱了，

事故把交通搞乱了，小姐把辈份搞乱了。

最引人注目的哥哥──伟哥；

最年轻的奶奶──二奶；

最难设防的偷窃──偷情；

最热闹的走廊──发廊；

最畅销的书──女秘书。

办公室守则：

苦干实干，做给天看；东混西混，一帆风顺；

任劳任怨，永难如愿；会捧会献，杰出贡献；

尽职尽责，必遭指责；推托栽赃，邀功领赏！

中央机关出上联：上级压下级，一级压一级级级加码马到成功；

地方政府对下联：下层蒙上层，一层蒙一层层层掺水水到渠成。

横批：和谐社会。

官场日志：

清晨起床，打拳；上午开会，打盹；

中午吃饭，打嗝；下午上班，打哈；

傍晚加班，打牌；晚上娱乐，打炮；

半夜回家，打架。

百姓语录：

狠抓就是开会，管理就是收费，

重视就是标语，落实就是动嘴，

验收就是宴会，检查就是喝醉，

研究就是扯皮，政绩就是神吹，

汇报就是掺水。

官场之最；

最难找的地方──有关部门；

最难捉摸的官话──研究研究；

最神秘的机构──组织上；

最大的官──一把手；

最难管的东西──一张嘴；

最谦虚的时候──在上级面前；

最冠冕堂皇的语言──工作需要；

最易接受的行贿──您讲得真好！

最关心的信息──自己这次能否升迁；

最傻的高兴──你的问题组织上也考虑了；

最无奈的选择──因为年龄！

组织就是......

在你遇到难事时对你说：我们无能为力！

在你遭遇用人不公时对你说：你要正确对待！

在你合法权益受侵时对你说：你要顾全大局！

在你受到诬陷时对你说：你要相信组织！

组织部长：谁关心我我就关心谁；

纪委书记：谁不关心我我就关心谁；

宣传部长：谁关心我, 我就关心他的正面，谁不关心我, 我就关心他的反面;；

市委书记：谁关心我我就让组织部长关心他，谁不关心我我就让纪委书记关心他。

忙碌的公仆在包厢里，重要的工作在宴会里，

干部的任免在交易里，工程的发包在暗箱里，

该抓的工作在口号里，须办的急事在会议里，

妥善的计划在抽屉里，应煞的歪风在通知里，

扶贫的干部在奥迪里，宝贵的人才在悼词里，

优质的商品在广告里，辉煌的数字在总结里。

公务员素描：

满腔热血投身社会，摸爬滚打终日疲惫；

低三下四谋取地位，常年奔波天天喝醉；

收入可怜啥都嫌贵，交往叩头处处破费；

有用本事已经作废，不学无术擅长开会；

口是心非阳奉阴违，溜须拍马寻找机会；

青春年华如此狼狈，苟且偷生窝囊一辈！

做人的难处：

有钱吧，说你准变坏；没钱吧，说你真失败；

有成就吧，说你会投机；没成就吧，说你没出息；

有情人吧，说你真坏；没有情人吧，说你变态！

中年烦恼：

职务不高，工资不高，血压血脂血糖高；

政治不突出，业务不突出，腰椎盘突出；

大会不发言，小会不发言，前列腺发炎；

炒菜糊，烧饭糊，麻将不胡。

批评上级，官位难保；批评同级，关系难搞；

批评下级，选票减少；批评自己，自寻烦恼；

批评老婆，她就乱跑；批评老公，他就乱搞；

唉，麻烦透了！

领导用过的叫文物，百姓用过的叫废物；

领导强词夺理叫坚持真理，百姓据理力争叫无理取闹；

领导握百姓的手叫关怀，百姓握领导的手叫巴结；

领导做蠢事叫轶事，百姓做错事叫傻子；

领导情人叫小蜜，百姓情人叫破鞋。

CCTV新闻联播的惊人发现：

开会没有不隆重的，闭幕没有不胜利的，讲话没有不重要的，决议没有不通过的，

鼓掌没有不热烈的，人心没有不鼓舞的，领导没有不重视的，进展没有不顺利的，

问题没有不解决的，完成没有不超额的，成就没有不巨大的，竣工没有不提前的，

接见没有不亲切的，中日没有不友好的，中美没有不合作的，交涉没有不严正的，

会谈没有不圆满的。

男女有别定律：

男人爱上女人会做诗，女人爱上男人会做梦；

女人对男人往往朝思暮想，男人对女人往往朝秦暮楚；

女人看家本领是撒娇，男人拿手好戏是撒谎。

女人的幸福在于：他真的爱我，

女人吻男人算是一种幸福，

男人吻女人算是一种口福；

当女人走投无路时她会和一个男人结婚，

当男人走投无路时一个女人会和他离婚；

做情人的时候女人会让男人心疼，

做妻子的时候女人会让男人头疼。

穷人富人论：

欠个人的钱是穷人，欠国家的钱是富人；

喝酒看度数的是穷人，喝酒看牌子的是富人；

写书的是穷人，盗版的是富人；

吃家禽的是穷人，吃野兽的是富人；

耕种土地的是穷人，买卖土地的是富人；

女人给别人睡的是穷人，睡别人女人的是富人。

贫穷与富裕的新解：

贫穷时养猪，富裕后养狗；

贫穷时种稻，富裕后种草；

贫穷时想娶老婆，富裕后想找情人；

贫穷时老婆兼秘书，富裕后秘书兼老婆。

水调歌头：重上酒楼

应酬复应酬，天天忙不休。

社交公关人情事，尽在酒里头。

泱泱五千年，底蕴多丰厚，帝王墨客江湖汉，江山美人酒。

改革大潮涌，迂腐全荡走，惟有美女权钱欲，兴盛遍神州。

工厂变歌厅，书店改酒楼，失业大军数百万，桑拿歌与酒。

上面常来人，视察加旅游，小心陪侍别出错，工作有成就。

世事多变幻，官场巧运筹，相互照应方便多，公酒交私友。

人生苦且短，工作多烦忧，哥们姐们再进一杯酒，泡完脚再走。

这年头，到处都是错别字：

植树造零；白收起家；

勤捞致富；择油录取；

得财兼币；检查宴收；

大力支吃；为民储害；

提钱释放；攻官小姐。

今日菜根谭

2007-03-14T14:23:00.000Z

I created a news feeder (RSS), quote of the day from Chinese wisdom: Vegetable Root (菜根谭）, which can be added into Google personal homepage.

Use Numbers

2006-12-15T22:46:00.000Z

To argue, try to use statements with numbers, like

"1000 sites * 20 services * 200 VOs * 10 roles/groups = 40M metadata"
"100k CPUs * 10 jobs per day * 10 queries per job = 10M queries per day, 115 queries per second."
Publishing one service adds 138 RDF statements to the triple store, which consumes up about 25 KB of memory. Then each triple uses 181 bytes. Each metadata has at least 3 triples: (entity, hasMetadata, metadata), (metadata, hasMetadataType, type), (metadata, hasMetadataValue, value). Thus 4M metadata uses 181 * 3 * 4M = 2.1GB.

And pay attention to detail.

似曾相识：雨中的苏州

2006-12-06T23:34:00.000Z

Create A User Community to Improve Our Open Source Software

2006-12-05T22:48:00.000Z

Why?

Create impact. Open source software's value relies on its usage.
Get feedback, which could be good ideas for the future work.

How?

Create a well organized website, easy downloads, a bugzilla system, etc. Twiki and Sourceforge are very helpful in this sense.
Well answer questions by email.
Create a user list, which includes the project name, url, contacts, etc.

BTW, check out OpenCola, which applies the open source idea in making cola.

My New Laptop

2006-12-04T22:29:00.000Z

Today I got a new laptop, a SONY VAIO, which has

Intel® Core™ 2 Duo T7200 Processor 2.0GHz
2GB, 100GB HDD & DVD/RW Super Multi DL Drive
13.3" WXGA TFT Display, Bluetooth & FREE T-Mobile WAN Card
... ...
And a fingerprint reader! So that I am able to log in Windows by using my fingerprint. Cool!

学什么专业？挣多少钱？——美国各级毕业生2006薪水调查

2006-12-03T18:38:00.000Z

摘自八阕。

<CopiedContent>

八阕】美国大学与雇主协会，每年按4个季度分别发布新毕业的大学生、硕士和博士研究生的薪水调查报告，这报告共涉及80个学士专业，40个硕士专业，23个博士专业。冬、春、夏三季发布的数据是各季度的数据，秋季版发布的是去年8月31日到今年9月1日为止的整个学年数据。以下列出的是该协会最新公布的美国博士和硕士毕业生2006年秋季薪水调查报告。

美国博士毕业生2006年薪水最高的几种专业：

1、商业管理(Business Administration/Management)：$81,438

2、电气/电子与通信工程(Electrical/Electronics & Communications Engineering)： $81,297

3、冶金工程(含陶瓷科学与工程)(Metallurgical Engineering )：$80,667

4、计算机科学(Computer Sciences)：$76,630

5、化学工程(Chemical Engineering)：$75,659

6、计算机工程(Computer Engineering)：$74,750

7、工业/制造工程(Industrial/Manufacturing Engineering)：$73,292

8、物理(Physics)：$72,357

如果这些工程师没有升迁到高层管理，做到退休一般年薪不会超过$150,000

美国博士毕业生2006年薪水最低的几种专业：

1、心理学(Psychology)：$49,374

2、人文科学(Humanities)：$48,938

3、社会科学(Social Sciences)：$48,487

4、生物/生命科学(Biological Sciences/Life Sciences)：$43,916

5、英语语言与文学(English Language & Literature/Letters )：$41,405

美国硕士毕业生2006年薪水最高的10种专业：

01、计算机科学(Computer Sciences)：$71,165

02、石油工程(Petroleum Engineering)：$68,833

03、电气/电子与通信工程(Electrical/Electronics & Communications Engineering)：$66,687

04、计算机工程(Computer Engineering)：$66,545

05、地质及相关科学(Geological & Related Sciences)：$64,111

06、材料科学(Materials Sciences)：$63,500

07、航天/航空/宇航工程(Aerospace/Aeronautical/Astronautical Engineering)：$62,811

08、工业/制造工程(Industrial/Manufacturing Engineering )：$61,273

09、机械工程(Mechanical Engineering)：$61,234

10、化学工程(Chemical Engineering)：$59,008

如果这些工程师没有升迁到高层管理，做到退休一般年薪不会超过$120,000

2004年美国律师的年薪平均为：$94,930

最低年薪：$64,620，最高年薪： $143,620

在公司和企业管理层谋职(Management of companies and enterprises)：$126,250

在联邦政府谋职(Federal Government)：$108,090

在法律机构谋职(Legal services)：$99,580

在地方政府谋职(Local government)：$73,410

在州政府谋职(State government)：$70,280

2004年美国各科医生的年薪：

麻醉科(Anesthesiology)：$306,964

外科、普通外科(Surgery, general)：$255,438

妇产科(Obstetrics/Gynecology：$233,061

内科(Internal medicine)：$155,530

小儿/青少年科(Pediatrics/Adolescent medicine)：$152,690

精神科(Psychiatry)：$163,144

家庭全科(Family Practice)：$150,267

联邦人口普查局2006年11月还发表了一项调查数据“2004年美国学历与平均年薪”：

高中以下学历平均年薪：$19,169

高中毕业平均年薪：　　$28,645

大学毕业平均年薪：　　$51,554

硕士以上学位平均年薪：$78,093

从以上各项的调查报告看来，华人根深蒂固的教育观念“学好数理化，走遍天下都不怕”，似乎真的放之四海而皆准了。并且，华人父母努力说服自己的子女当律师和医生也是有根有据的。当然，学历越高收入越多的说法更是真理。

</CopiedContent>

我的评论：

选择专业前先要做好各个专业的benchmark, 什么专业工资最高。看来benchmark真是无时无地不重要。
计算机科学博士起薪只比计算机科学硕士高7.7%。虽然理论上博士比硕士有更好的成长空间，究竟读不读博士还是要慎重考虑。
医生收入压倒性的高，远高于其他专业，甚至包括律师。还是学医收入高。
读书有用啊。读书越多，收入越高。

BTW, I read this from Ian Foster's blog:

"Indeed, it was outsiders—those with expertise at the periphery of a problem's field—who were most likely to find answers and do so quickly."

Java I/O

2006-12-01T21:45:00.000Z

This presentation gives a good overview of Java I/O, from which I have learned:

FileInputStream/FileOutputStream: for sequential I/O and simple to buffer
RandomAccessFile: for full random-access
FileChannel: adds support for NIO byte buffers
MappedByteBuffer: for memory-mapped I/O
Buffering: for instance, use BufferedOutputStream for buffering FileOutputstream
Forcing I/O, e.g. flush() or FileChannel.force(), equals to the "fsync" system call
No direct I/O in Java

Google File System

2006-11-30T22:06:00.000Z

The Google File System is a scalable distributed file system for large distributed data-intensive applications, being used within Google. The paper, published in the 19th ACM Symposium on Operating Systems Principles, is a well written computer engineering paper. System builders should read it.

Quoted from its conclusion:

"We started by reexamining traditional file system assumptions in light of our current and anticipated application workloads and technological environment. Our observations have led to radically different points in the design space. We treat component failures as the norm rather than the exception, optimize for huge files that are mostly appended to (perhaps concurrently) and then read (usually sequentially), and both extend and relax the standard file system interface to improve the overall system."

Lesson learnt:

Application driven.
Base on the technological environment. Base on the facts of performance characteristics. Performance benchmark is the foundation for system design.
Application/file system (or OS, or underlying infrastructure, etc.) co-design.
Optimize for what should be optimized.

"Our system provides fault tolerance ..."

Lesson learnt:

Reliability is a must for any system. System must work. Never forget reliability, persistence, fault tolerance during system design.

"... This makes possible a simple, centralized master that does not become a bottleneck. ..."

Lesson learnt:

Down to earth.
...I had already come to the conclusion that in the practise of computing, where we have so much latitude for making a mess of it, mathematical elegance is not a dispensible luxury, but a matter of life and death. - Edsger Wybe Dijkstra ["My hopes of computing science" (EWD 709)]

elegant: [...]ingeniously simple and effective. - Concise Oxford Dictionary

This elegance makes perfect sense in the domain of system design. May we say in system design, elegance is not a dispensible luxury, but a matter of life and death.

A Korean Artificial Beauty

2006-11-17T10:58:00.000Z

A authentic dream face based on popular Korean stars.

A Very Interesting Story

2006-11-16T20:18:00.000Z

Copied from Ian Foster's blog.

<CopiedContent>

Hadoop on EC2

Here's something neat (and details here).

Hadoop, an open source clone of Google FS and MapReduce, can be run on top of Amazon EC2, a hosting service that allows leasing servers on an hourly basis.

As Greg Linden goes on to say:

Developers may now be able to rapidly bring up hundreds of servers, run a massive parallel computation on them using Hadoop's MapReduce implementation, and then shut down all the instances, all with low effort and at low cost. Very cool.

My colleague Tim Freeman points out that you can run those same VMs on your own resources using the Globus Workspace service.

</CopiedContent>

I got a feeling that parallel computing becomes more and more available, and has better and better programmability.

Provenance of Life

2006-11-16T17:17:00.001Z

We are moving. Our current landlord claims the carpet is "new". Although it does not look like a new one at all, and I remember it was not very clean when I moved in one year ago, we do not have any evidence to support our point. The landlord also blames us that we did not report problems to them promptly. Having consulted the university accommodation officer, we now learn that we had better write to them and keep a photocopy of the letter.

The evidence matters. The evidence could be pictures, receipts, letters, or anything concrete, not transient, but retrievable. Furthermore, the evidences should not be isolated. They should be logically linked together, and finally lead to some conclusion. Put it in another way, when we see an event in the life, we would like to "see" the complete process that leads to this event. By "see" we really mean to reconstruct it in a convincible way.

My boss is investigating a "provenance" project, which defines "the provenance of a piece of data is the process that led to that piece of data". The aim of the project is to "to conceive a computer-based representation of provenance that allows us to perform useful analysis and reasoning ...".

I argue we also need provenance support in our lives. Important facts should be documented in a retrievable and searchable way, for instance, in a computer-based way. We should keep recording provenance of life.

Google advocates searching instead of organizing. Ideally, as long as we record the provenance of life in our computer or on the internet, there should be a way to query and retrieve it.

I am interested in all techniques to improve productivity. It is great that some software could help to record the provenance of life an query over it, which will definitely improve our productivity of life.

来点喜气与财气

2006-11-11T11:01:00.000Z

拷贝自二闲堂.

The Outcome of MPT

2006-11-08T10:11:00.000Z

Here is a visualized outcome of MPT. The x-axis is the number of processors used in the throuthput test (1 - 32). The y-axis is the number of threads per processor (1 - 16). The z-axis is the measured throughput. More precisely, it is the number of service discovery (by name) requests served in 30 minutes when there are 5000 service descriptions registered in GRIMOIRES. When 32 processors and 16 threads per processor are used, GRIMOIRES can serve 62,877 service discovery requests in 30 minutes , i.e., 34.9 per second. Each request costs 28.6 milliseconds.

旅游大计

2006-11-07T14:17:00.000Z

星期六一早起床, 阳光明媚, 万物复苏. 乘机定下今后N年的旅游大计. 以求心中有数, 目标明确.

这里我来谈三点.

(一) 回顾过去, 成绩很大.

香港, 东方之珠, 生活5年, 非常好;
倫敦, 逝去帝国的首都, 非常好;
巴黎, 文化艺术之都, 非常好;
瑞士, 美丽的高山, 湖泊, 非常好;
芝家哥, the mile of magnificence, 不错;
加拿大BC, 温哥华, 维多利亚, 不错, 从温哥华到维多利亚的水路很美;
洛山基, 听说治安很差, 硬是没敢进downtown;
胜地呀哥, 还是不错的, 还看到了航母.

(二) 面对现实, 压力很大.

这里又要谈两点, 一如何在时间上创造机会, 二是如何在经济上开源节流. 以后有时间再版详细谈.

(三) 展望将来, 希望很大.

埃及, 非常想去看看古埃及的伟大;
日本, 又爱又恨, 要去走走;
台湾, 另类中国, 要去看看;
纽约, 要去;
拉丝围家四, 要去;
欧洲, 那是要经常过去看看的, 意大利, 西班牙, 德国, 北欧, ..., 法国和瑞士也要再去;
西藏的债也是要还的;
非洲, 南美, 大洋洲, 有机会都要去踩上几个脚印.

The Reasons to Have a Blog

2006-11-07T14:06:00.000Z

According to Wikipedia, the term "blog" is a contraction of "web log". From its original name, blog can be inferred to have two characteristics that correspond to the two purposes for me to blog.

Blog is a log that is organized as a chronicle. I can use blog to track events in my life and to record my thoughts. In this sense, the blog is just like a diary.

Blog is web based thus it is intended to be read by people. Blog can be a good communication tool to present the blogger. Blog is actually a bilateral communication tool because it allows other to comment.

Monitor the Blog and Benchmark the Program

2006-11-02T20:15:00.000Z

I am using Google Analytics to monitor visits to my blog. See the picture, in which the dots indicates where somebody has visited my blog. There is even one returning visitor. Not bad!

It is a good practice to start monitoring since the website is established. Just like it is a good practice to start benchmarking since the program is prototyped.

Monitoring is an inherent requirement of a website. Thus it should become a part of website infrastructure. I.e., when you establish a website, your website is automatically being monitored. No webmaster effort is involved.

Just like benchmarking is an inherent requirement of a program. Benchmarking should become a part of development environment. When you prototype a program, it costs you zero or little effort to benchmark the program. Some benchmark specific code should be automatically added into your program. Thus according to the benchmark data, you can make a decision on whether to take a certain refactoring or not. Will Aspect-Oriented Programming help on this?

Use VMware to Release Server-Side Software

2006-11-01T20:50:00.000Z

I am using WMware's free VMware Server to create a GRIMOIRES virtual appliance. I admit it is not very easy to install GRIMOIRES. And I believe it is not easy to install any server-side software, probably because of the tedious and error prone procedure to configure, for instance, backend database, and security.

While VMware is able to relieve this pain. We create a VMware virtual appliance, which includes OS and our software. Our customer simply downloads it, and is able to play it, thus avoiding the painful installation procedure. The size of the VMware virtual appliance could be much much bigger than the size of our software. But who cares. Our computer spends more time in downloading, but we spend much less time in installation.

Massive Parallel Test Kit

2006-10-31T20:59:00.000Z

I have implemented a Massive Parallel Test Kit (MPT) whose major goal is to measure the throughput of a server. While in my case, MPT is used to measure the throughput of our web service registry, GRIMOIRES.

MPT has the following features:

MPT is an MPI program. So it is able to create a large number of real parallel clients on a computer cluster to generate a large number of requests simultaneously. That is why it is claimed to be "massive parallel". By the way, it is really convenient to program a cluster using MPI.
Each MPI process invokes a Java program, ThroughputTest. ThroughputTest provides basic functionalities required to run a throughput test. Above all, it supports user provided plug-in, called Worker, which implements the actual test business logic.
In fact, Worker extends Thread. Thus two forms of parallelism are leveraged in MPT: multi-processing at the MPI level, and multi-threading in at the Java level.
MPT has a single control point. The degree of multi-processing, the degree of multi-threading, and the test time are all set as the parameters for the MPI program. The latter two are then passed as environment variables to ThrouputTest.
The fairness of the service can be revealed by calculating the deviation of the throughputs measured by individual MPI processes.

I believe MPT is a useful tool to conduct a stress test for web services. A cluster is useful for a building and testing environment for web services. After more tests of MPT, I will release it at the GRIMOIRES project website.

传说中在大陆花10万元拍的婚纱照

2006-10-30T16:55:00.000Z

More photos

Many First Times Today

2006-10-30T00:04:00.000Z

Happy today! Lovely weather. We had many first times today. For the first time, we had our Hong Kong style dim sum in UK. It is not bad. For the first time, we drove through a bridge asking for a payment, 50p. There is a good view along the bridge and nice drive.

My Very First Blog

2006-10-28T21:38:00.000+01:00

This is my first blog.

Testing

测试

Keep testing

继续测试