Remote learning, graduate projects, and open source

Last week, an ex-colleague asked me if I could be a reviewer on a graduation project at National University where he is now a professor. Flattered, and finding myself with a little bit more time on my hands than usual, I accepted.

The goal of the project was to let a team of seven students build a real-life application as a learning experience about how software development is done in the industry, for a "real" customer. The task was to develop a tracking application for the Boys & Girls Clubs of the Sequoias.

There were several interesting aspects of this project: first of all, the end result was a working application that in quality and functionality exceeds a good portion of similar applications built by IT departments. Key to this was the use of off-the-shelf frameworks such as Spring MVC, Hibernate, BIRT, and jQuery. I'm glad that students are using frameworks like these rather than reinventing their own, although that might have been even more fun.

Another interesting aspect was that the project was done as an open source project on code.google.com. The practical advantages are the infrastructure is provided completely ready by the code forge (in this case code.google.com), so that little time needs to be spent on setting up an infrastructure. This is especially important for students that are remote and hence don't have a shared work environment (e.g. office). Of course there's a lot more work to be done to make it a successful open source project, but that's not important in the scope of the project. I'm glad universities expose students to an open source way of development.

Lastly, the presentation of the project was done using a tool called ClassLive Pro. This is a JNLP application that provides similar functionality to WebEx. Although I've developed a few JNLP applications myself, and I'm convinced of its great utility, I haven't encountered any other JNLP applications so I was glad I ran into this application.

A SEED certificate

With the Oracle acquisition looming on the horizon, and the uncertainty that comes with it, it feels like many organizations within Sun are bringing projects in a state in which they can be transitioned in a clean way. Transitioned meaning adopted or abandoned.

I also find myself doing this: for instance I'm trying to put the finishing touches on JMSJCA, and trying to put Hulp in a state where other people can use it, etc.

Maybe it was in that light that a few days ago, I got a certificate from the SEED organization within Sun. SEED is a mentoring program within Sun in which I had the good fortune to participate. I also was a SEED mentor. Or perhaps I should let the speak for itself.

Lessons of an interesting deadlock problem

Two years ago I wrote a blog entry about Nested Diagnostics Contexts in GlassFish. The approach was based on custom Logger objects. Now, two years later when running GlassFish on AIX, an issue with that turned up: a deadlock. Digging into it, there are two lessons I'd like to share: one about deadlocks, and the other one about workarounds.

Deadlock in the JDK

At the time of my writing the NDC facility, calling addLogger() and getLogger() from different threads may cause a deadlock in the JDK classes. Why is this?

This is how the code in the Sun JDK was:

public class Logger {
...
    public static synchronized Logger getLogger(String name) {
	LogManager manager = LogManager.getLogManager();
	Logger result = manager.getLogger(name);
	if (result == null) {
	    result = new Logger(name, null);
	    manager.addLogger(result);
	    result = manager.getLogger(name);
	}
	return result;
    }
}

public class LogManager {
...
    public synchronized boolean addLogger(Logger logger) {
...
        Logger plogger = Logger.getLogger(pname);
    }
...
}

There is a problem if there are two threads calling these two methods. Say Thread 1 calls Logger.getLogger() in its application code, e.g.

public void doSomething() {
  Logger.getLogger("com.stc.test").info("doSomething!");
}

Now let's consider another thread Thread 2 tries to register a custom logger using addLogger(), e.g.

LogManager.getLogManager().addLogger(new Logger() {
...
});

There are two locks that come into the picture: one is the lock on Logger.class, and the other one on LogManager.getLogManager(). Thread 1 simply calls Logger.getLogger() which first locks Logger.class, and while having this lock, will call LogManager.addLogger(). This call will get a lock on LogManager.getLogManager(). This will try to lock Logger.class again, which is no problem because it already had the lock on that. Now consider Thread 2: it will first lock LogManager.getLogManager() through its call to addLogger(), and while having this lock, it will try to lock Logger.class through its call to Logger.getLogger().

So Thread 1 will lock first A and then B, while Thread 2 will first lock B and then A. A classical deadlock situation.

This is clearly a bug in the JDK. With the knowledge of this bug, as the developer of the custom logger, we can use a simple workaround for this problem. Before calling addLogger(), first lock Logger.class. By doing that, we can guarantee that the locks are called in the same order: first Logger.class and then LogManager.getLogManager().

This bug was actually reported by a customer, see bug report, and was fixed in 6u11 and 5.0u18. The LogManager now no longer calls Logger.getLogger(), so there's no locking of Logger.class anymore.

Deadlock in GlassFish

GlassFish installs its own LogManager, and there's some locking going on there too. It has an internal lock which it locks in addLogger():

    public boolean addLogger(Logger logger) {
...
    super.addLogger(logger);
...
    synchronized (_unInitializedLoggers) {
...
        doInitializeLogger(logger);
    }
}

And this calls Logger.getLogger() again.

    protected void doInitializeLogger(Logger logger) {
...
    Logger newLogger =
        Logger.getLogger(logger.getName(), 
            getLoggerResourceBundleName(logger.getName()));
...
    }

As a developer of the custom logger, we can still use the same fix: first lock Logger.class before calling addLogger() so that the locking sequence becomes: Logger.class, transient lock on LogManager.getLogManager() followed by a lock on _unInitializedLoggers, lock on Logger.class().

A fix in GlassFish would simply make addLogger() synchronized. An alternative fix is to make the synchronized block smaller: it should not extend over doInitializeLogger().

Deadlock with the IBM JDK

All was well until this was run on the IBM JDK. As it turns out, Logger.getLogger() doesn't lock Logger.class, it locks LogManager.getLogManager(). Now there is a new deadlock: Thread 1 calls Logger.getLogger() and by doing so, locks LogManager.getLogManager(), and then tries to lock _unInitializedLoggers and later tries to lock LogManager.getLogManager() again. Thread 2 calls addLogger() and by doing so locks first _unInitializedLoggers which later tries to lock LogManager.getLogManager(). The problem is back: the sequence of locking is reversed.

Lessons learned

The first lesson is that of the deadlock itself: one should be extremely careful when locking an object, and then calling into another object, especially if that object is meant to be a baseclass in a public API. Both Logger and LogManager may be overridden, and are clearly very public.

The second lesson is that instead of building workarounds, try to get a fix the original bug. Or at least try to have the original issue addressed while adding a workaround. In my case, I never filed a ticket against the JDK or against GlassFish. I should have known better.

... or not?

At this very moment, GlassFish v2.1.1 is about to be shipped, and we need GlassFish v2.1.1 on AIX for the upcoming CAPS release. A bug fix in GlassFish v2.1.1 is probably not going to get in. So I'm making yet another workaround: don't use a custom logger at all as to avoid the whole addLogger() problem. Instead, use standard Logger objects, but provide a custom filter that manipulates the log record. This will work on both unpatched versions of GlassFish, and also on unpatched versions of the JDK. Still, I hope the GlassFish problem will be fixed.

JavaOne 2009 TS-5341: Rethinking the ESB: lessons learned challenging the limitations and pitfalls – audio recording uploaded

At JavaOne 2009 I gave a number of presentations. One was "Rethinking the ESB: lessons learned challenging the limitations and pitfalls" which I did with Andi Egloff. JavaOne sessions were not recorded this year, that is not by the JavaOne Organization. But I used an mp3 player to record the sessions I was involved in. I finally processed the audio and uploaded it. I also uploaded a PDF of the presentation:

• Presentation (PDF) of TS-5341 Rethinking the ESB: lessons learned challenging the limitations and pitfalls
• Audio recording (mp3) of TS-5341 Rethinking the ESB: lessons learned challenging the limitations and pitfalls

Message interceptors with JMS

Interceptors are a way to add behavior to a system without directly invoking this behavior in from code. The typical anti-example is that if you would want to log the entry and exit of methods on a class, you could do that of course by adding log statements in each and every method. The downside is that you would have to change each and every method, and that all these methods now have repetitive code in there. With the interceptor approach on the other hand, you could do that by adding an interceptor that is invoked before and after the method is invoked. In the code of the interceptor you would then add the log statement. The advantage is that the existing methods are not changed, that repetitive code is avoided, and that the logging concern is concentrated in one part of the code base rather than spread all over.

I think that this logging example is pretty dumb, but I guess it drives home the message of adding behavior non-invasively, and a separation of concerns. I'll discuss some more interesting examples in a minute.

Interceptors in EE

Interceptors have long been the domain of AOP (Aspect Oriented Programming) packages that add interceptors through byte code manipulation, either at runtime or as a post-compile step. For EE developers, things changed with the advent of EE5 in which interceptors were added to EJBs.

An interceptor can be added to all the methods or individual methods of an EJB by adding an annotation to either the class or to individual methods. In the following example, the NoConcurrency interceptor is invoked when the getUnclaimedAccounts() method is called.

@Interceptors(NoConcurrency.class)
public void getUnclaimedAccounts(String category) {...

The interceptor class is a simple Java class that has to have a method with the following signature:

@AroundInvoke
Object <METHOD>(javax.interceptor.InvocationContext) throws Exception

The interceptor method is called when a client calls an EJB method. The interceptor typically calls InvocationContext.proceed(). This will call into the EJB method (or the next interceptor, should there be one).

Interceptor classes are typically packaged as part of an EAR file: they are part of the application.

Interceptors are referenced in an EJB using the @Interceptors annotation. They can also be referenced from the EJB's deployment descriptor. In either case, the application needs to be modified to specify interceptors.

Another limitation of EJB interceptors is that they only work on EJB methods. When common behavior needs to be added to other interactions, for instance if an EJB invokes something like an outbound resource adapter, a different mechanism must be used.

Interceptors in CAPS

A Java CAPS customer had an interesting case for using interceptors. The customer had hundreds of integration applications that all received and sent JMS messages. If a message cannot be processed, the message is sent to an error handling system using JMSJCA's dead letter queue facility. An operator will monitor the dead letter queue, and may resubmit the message. If a resubmitted message is processed successfully, the error handling system needs to be notified of that fact. One way of adding this behavior would be to add a chunk of code to all MDBs. That would of course lead to a lot of duplicated code. It also leads to mixing this system level behavior with the business logic in the integration applications. This is clearly undesirable.

Because Repository based projects in Java CAPS generate all the MDB code and assembles the applications, it is difficult to make use of EJB interceptors. What is needed is to specify the use of interceptors on a global basis, that is outside of the applications, and preferably for all applications at the same time.

Another requirement is that the customer wanted to add common behavior not just when messages are received, but also when messages were sent from an EJB. Upon sending a message, payload validation should happen. Since EJB interceptors can only be used for EJB methods, this was another reason that EJB interceptors could not be used.

The solution was to add a new feature to JMSJCA to support interceptors.

Interceptors in JMSCA

Interceptors in JMSJCA are invoked not only before / after a message is delivered to the MDB, but also any time a message is sent.

Rather than introducing a new Java interface, which would bring with it the complication of compile time dependencies on JMSJCA jars, the JMSJCA interceptors use the same annotations as in EJB3: any class with a method with the right signature and the @AroundInvoke annotation can serve as an interceptor.

Which classes will JMSJCA consider as interceptors? For this, it uses the service locator mechanism from JDK 6. This is done without introducing dependencies on JDK 6 by the way. With the service locator mechanism, a jar that holds the interceptor should have a special manifest file that lists the class names of the interceptors. The name of this manifest should be META-INF/services/myinterceptor where myinterceptor is the name of the interceptor. This name needs to be specified in the JMSJCA configuration. If no name is specified, the default name jmsjca.interceptor is used. This means that interceptors using the service name jmsjca.interceptor are automatically loaded and invoked if not overridden in the JMSJCA configuration.

With this, it's easy to add interceptors to all applications running in the application server, whenever they send or receive messages through JMSJCA, without having to change any application.

For more information, see http://jmsjca.dev.java.net/ .

JavaOne 2009

I'm writing this on my way to JavaOne 2009. San Francisco airport was fogged in so my flight was cancelled. I have some time to burn: I'm waiting for a train to take me from Santa Clara to San Francisco.

As usual it has been immensely busy in the two months before JavaOne.

For this JavaOne, I'm the lead of the SOA/Services track. A track lead is responsible for assembling a program of sessions and BOFs. Or in more practical terms, going through the proposals with the other volunteers that make up the selection committee, and picking the proposals so that not only the quality but also the variety of the track is optimal. The selection committee consists of subject matter experts. There are three from Sun and four from other companies: RedHat, Cap Gemini, Adobe.

This year there again were many hundreds of proposals for sessions and BOFs. Going through them to make a first cut took a lot of time. Refining the first cut to the final selection is difficult: there were many good and interesting proposals. But in the end only 22 sessions could prevail. Getting consensus on the selections, scheduling them, ensuring that presentations are ready, reviewing the presentations, the inevitable change requests, etc. took a lot more time than I expected. But I think these efforts have paid off: I think we have a very interesting lineup of presentations.

My second JavaOne activity is presenting at Java University, the day before JavaOne. Like last year, Joe Boulenouar invited me to present a few hours on SOA and integration middleware. The course title is "Using Java EE and SOA to Help Architect and Design Robust Enterprise Applications." Last year there were four hundred or so people in attendance. I haven't heard yet how many people there will be this year. Perhaps fewer on account of the poor the economy.

Thirdly, I'm presenting a session titled "Glassfish ESB: get your apps on the bus". The original plan was that Keith Babo would pull that wagon, but unfortunately he had to pull out. I'm now doing that session with Sujit Biswas. In this session I'm making a case for ESBs or integration middleware in general. This has led me to some more thoughts on how we can widen the applicability of ESBs. More about that in a future post.

I'm also doing a session titled "Rethinking the ESB: lessons learned challenging the limitations and pitfalls". Andi Egloff is co-presenter. The idea of this session is to show a number of things that went wrong in previous product releases and how we changed that in subsequent releases. The interesting part for the audience is that it gives a set of requirements or "things to avoid" in middleware products.

The fifth activity I'm involved in is the OpenESB community BOF. The idea of this BOF is to discuss experiences with OpenESB and discuss what's on the wish list for OpenESB.

So, yes, it was busy running up to JavaOne. I'm sure that JavaOne itself will also be very busy: not only will there be the presentations I need to give, others I have to be in attendance for, but there will also be the numerous customer meetings.

I think I'll need a break after JavaOne!

The java.lang.LinkageError: loader constraint violation" demystified

My colleague Murali Pottlapelli ran into an interesting problem the other day. He added Rhino to his BPEL Service Engine, and saw this error happen:

java.lang.LinkageError: loader constraint violation: loader (instance of <bootloader>) previously initiated loading for a different type with name "org/w3c/dom/UserDataHandler"

The weird thing was that this exception was thrown from a call to Class.getMethods() on a class shipped with the JVM!

Googling this problem revealed that there are a lot of people running into this issue, often when using OSGi. Most search results referred to email lists postings where people ran into this problem. None of the web pages properly explained what the problem was. Intuitively I felt we could solve our problem by moving a jar to a different classloader, but was that merely hiding the problem? As with my post on "How to fix the dreaded "java.lang.OutOfMemoryError: PermGen space" exception (classloader leaks)", I was convinced that understanding the problem is key. So Murali and I set out to dig in this problem deeper until we completely grasped it.

As it turns out, there are some aspects about this problem that make it very confusing:

1. In a dynamic component system, a change in one component may cause the other components to fail in areas that used to work properly before. At the same time, the changed component appears to be working properly.
2. The order in which components are activated determines where and how the problem shows up.
3. The effects of the problem may show up in innocuous and seemingly unrelated calls such as Class.getMethods().

In the following sections I'll first illustrate the problem, and then explain in detail what's causing the problem.

The problem

Let's look at a model example. In stead of looking at an OSGi example or a JBI example, let's look at EE because it will be more familiar. Imagine we have an EAR file with an EJB and two WAR files. The WAR files are identical, and have a servlet that uses an EJB to log in. As such we have three classes:

public static class User {
}

public static class LoginEJB {

  static {
    System.out.println("LoginEJB loaded");
  }

  public static void login(User u) {
  }
}

public static class Servlet {
  public static void doGet() {
    User u = new User();
    System.out.println("User in " + u.getClass().getClassLoader());
    LoginEJB.login(u);
  }
}

Let's say that one WAR is configured with a self-first classloader, and the other one uses the default parent-first class loading model.

Now consider these three scenarios:

1. We log in using the parent-first servlet, and then inspect the EJB with Class.getMethods(). Everything works as expected, but when we then try to login on the second servlet, we see the linkage error.
2. We log in using the self-first servlet. Then when we call Class.getMethods() on the EJB, this fails. Also, we can no longer log in on the parent-first servlet!
3. We first call Class.getMethods() on the EJB. We can no longer login using the self-first servlet, but the parent-first servlet still works.

What is going on? To explain, let's first revisit some classloader basics. If parent-first and self-first is in your daily vocabulary, feel free to skip the next section.

Self-first versus parent-first delegation

What is meant with self-first delegating classloaders? Here's the skinny on classloaders. In Java you can create your own classloader for two reasons: this allows multiple versions of the same class to co-exist in memory, as is often found in OSGi. It also allows classes to be unloaded, as is found in application servers. A classloader typically represents a set of jars that make up the module, the component, or the application. Each classloader must have a parent classloader. Hence, classloaders form a tree with the bootstrap classloader as the root. See the picture above.

When a classloader is asked to load a class, it can first ask its parent to load the class. If the parent fails to load the class, that classloader will then try to load the class. In this scheme, called parent-first class loading, common classes are always loaded by the parent classloader. This allows one application or module to talk to another application or module in the same VM.

Instead of asking the parent classloader first, a classloader can also try to find a class itself first, and only if it cannot find the class would it ask the parent classloader to find the class. A self-first classloader allows for an application to have a different version of a class than found in the parent classloader.

Classloader lab

To show what's going on, I've developed a small demo that emulates the scenario with the two WARs and the EJB. Key in this demo is a custom classloader. The constructor takes a list of classes that should be defined by that classloader, i.e. the classloader behaves as self-first for those classes, and delegates to the parent classloader for the other classes. The custom classloader is listed in the code at the bottom of this post.

This is how the system is setup: a classloader for the EJB that loads the LoginEJB and the User class. A classloader for the parent-first WAR that loads the Servlet only, and a self-first classloader that loads the Servlet and the User class.

CustomCL ejbCL = new CustomCL("EJB  ", Demo.class.getClassLoader(), "com.stc.Demo$User", "com.stc.Demo$LoginEJB");

CustomCL pfWebCL = new CustomCL("PFWeb", ejbCL, "com.stc.Demo$Servlet");

CustomCL sfWebCL = new CustomCL("SFWeb", ejbCL, "com.stc.Demo$User", "com.stc.Demo$Servlet");

The custom classloader prints all class loading requests so it can be easily seen what's happening.

Classloading eagerness

What exactly happens when the LoginEJB class is loaded? In the demo program, the following line causes the following output:

ejbCL.loadClass("com.stc.Demo$LoginEJB", true).newInstance();

EJB  : Loading com.stc.Demo$LoginEJB in custom classloader
EJB  : super.loadclass(java.lang.Object)
EJB  : super.loadclass(java.lang.System)
EJB  : super.loadclass(java.io.PrintStream)
LoginEJB loaded

When the JVM loads the LoginEJB class, it goes over references in the class and loads those classes too: the java.lang.Object class because it's the super class of the EJB, and the java.lang.System and java.io.PrintStream class because they are used in the static block. That these "JVM" classes are loaded is in itself remarkable and shows an interesting aspect of how classloading works. "JVM" classes are not treated specially, and it is not relevant that they are already loaded in the bootstrap classloader and are used all over the place.

When the EJB classloader receives the request to load these "JVM" classes, that classloader of course delegates those requests to the parent classloader. In fact, it's a requirement to delegate all class load requests to the parent classloader for all classes that are in java.* and javax.*.

Something also remarkable is what is not loaded: the User class. Apparently, the fact that this class is used in a method is not enough to cause this class to be loaded when the EJB class is loaded. It is difficult to predict what classes are loaded as the result of loading a particular class. I think the spec leaves a lot of room to implementers to decide when to do so.

It's important to realize that when the EJB class is loaded, the User class is not loaded yet.

Linking classes

Next, let's use the parent-first Servlet to log in:

pfWebCL.loadClass("com.stc.Demo$Servlet", false).getMethod("doGet").invoke(null);

PFWeb: Loading com.stc.Demo$Servlet in custom classloader
PFWeb: super.loadclass(java.lang.Object)
EJB  : already loaded(java.lang.Object)
PFWeb: super.loadclass(com.stc.Demo$User)
EJB  : Loading com.stc.Demo$User in custom classloader
PFWeb: super.loadclass(java.lang.System)
EJB  : already loaded(java.lang.System)
...
Logging in with User loaded in EJB
PFWeb: super.loadclass(com.stc.Demo$LoginEJB)
EJB  : already loaded(com.stc.Demo$LoginEJB)

Ignoring the "JVM" classes, it can be seen that the Servlet causes the User class to be loaded, and that that class is loaded in the EJB classloader. Nothing unexpected here.

If subsequently the self-first Servlet gets a go, the following happens:

sfWebCL.loadClass("com.stc.Demo$Servlet", false).getMethod("doGet").invoke(null);

SFWeb: Loading com.stc.Demo$Servlet in custom classloader
SFWeb: super.loadclass(java.lang.Object)
EJB  : already loaded(java.lang.Object)
SFWeb: Loading com.stc.Demo$User in custom classloader
SFWeb: super.loadclass(java.lang.System)
...
Logging in with User loaded in SFWeb
SFWeb: super.loadclass(com.stc.Demo$LoginEJB)
EJB  : already loaded(com.stc.Demo$LoginEJB)
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
Caused by: java.lang.LinkageError: loader constraints violated when linking com/stc/Demo$User class
    at com.stc.Demo$Servlet.doGet(Demo.java:82)
    ... 6 more

The User class is loaded and defined in the self-first WAR classloader, and when the login() method is called, an instance of that class is passed to the EJB. Why does the linkage error happen?

When the parent-first servlet invoked the EJB, it passed in a User object. At that moment, the JVM links the reference to com.stc.Demo$User to a class instance. A class instance is identified using the fully qualified name and the classloader instance in which it was loaded. Upon invocation of login(User), the JVM will check that the class object of the User passed in, matches the class object that was linked to com.stc.Demo$User. If they don't match, the JVM will throw a LinkageError.

This linking happened on the first invocation. We can also force the linking to happen by calling LoginEJB.class.getMethods(). I can illustrate that by changing the test program to fist inspect the EJB class, and then make the self-first servlet to login.

System.out.println("Loading EJB");
ejbCL.loadClass("com.stc.Demo$LoginEJB", true).newInstance();
System.out.println("Examining methods of LoginEJB");
ejbCL.loadClass("com.stc.Demo$LoginEJB", false).getMethods();
System.out.println("Logging in, self-first");
sfWebCL.loadClass("com.stc.Demo$Servlet", false).getMethod("doGet").invoke(null);

Loading EJB
EJB  : Loading com.stc.Demo$LoginEJB in custom classloader
LoginEJB loaded
Examining methods of LoginEJB
EJB  : already loaded(com.stc.Demo$LoginEJB)
EJB  : Loading com.stc.Demo$User in custom classloader
Logging in
SFWeb: Loading com.stc.Demo$Servlet in custom classloader
SFWeb: Loading com.stc.Demo$User in custom classloader
Logging in with User loaded in SFWeb
SFWeb: super.loadclass(com.stc.Demo$LoginEJB)
EJB  : already loaded(com.stc.Demo$LoginEJB)
java.lang.reflect.InvocationTargetException
Caused by: java.lang.LinkageError: loader constraints violated when linking com/stc/Demo$User class

The login fails because the LoginEJB.class.getMethods() invocation causes the com.stc.Demo$User reference to be linked with the User class loaded in the EJB classloader. When the self-first servlet invokes the method, the two class objects don't match, causing the Error to be thrown.

Small mistake results in "poisoning" a shared class

By now it should be obvious that the User class should not have been packaged in the self-first WAR. What may not be obvious yet, is that this small mistake has big consequences. If a login happens on the self-first servlet before anything else, the linking happens with the erroneous User class object from the self-first classloader. This will cause the login of the parent-first WAR to fail. It will also cause the LoginEJB.class.getMethods() invocation to fail.

System.out.println("Loading EJB");
ejbCL.loadClass("com.stc.Demo$LoginEJB", true).newInstance();
System.out.println("Logging in, self-first");
sfWebCL.loadClass("com.stc.Demo$Servlet", false).getMethod("doGet").invoke(null);
System.out.println("Examining methods of LoginEJB");
ejbCL.loadClass("com.stc.Demo$LoginEJB", false).getMethods();

Loading EJB
EJB  : Loading com.stc.Demo$LoginEJB in custom classloader
LoginEJB loaded
Logging in, self-first
SFWeb: Loading com.stc.Demo$Servlet in custom classloader
SFWeb: Loading com.stc.Demo$User in custom classloader
Logging in with User loaded in SFWeb
SFWeb: super.loadclass(com.stc.Demo$LoginEJB)
EJB  : already loaded(com.stc.Demo$LoginEJB)
Examining methods of LoginEJB
EJB  : already loaded(com.stc.Demo$LoginEJB)
EJB  : Loading com.stc.Demo$User in custom classloader
java.lang.LinkageError: Class com/stc/Demo$User violates loader constraints
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:465)
    at com.stc.Demo$CustomCL.findClass(Demo.java:36)
    at com.stc.Demo$CustomCL.loadClass(Demo.java:54)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2395)
    at java.lang.Class.privateGetPublicMethods(Class.java:2519)
    at java.lang.Class.getMethods(Class.java:1406)
    at com.stc.Demo.test1(Demo.java:98)
    at com.stc.Demo.main(Demo.java:111)

The first login with the self-first WAR has effectively poisoned the EJB, making it unusable. In everyday life, the problem is likely not so clear cut. For instance, a developer adds a jar to a component, or changes the classloading delegation model of a component, and all tests with that component may succeed. The problem may only show up in next day's build when integration tests are run. And as the example above shows, the stacktrace does not tell much about where the real cause of the error is.

Formalization and references

A formal description of loading constraints can be found in detail in section 5.3.4 of The Java Virtual Machine Specification (online available at http://java.sun.com/docs/books/jvms/second_edition/html/ConstantPool.doc.html). In simple words, a linkage error can occur if two different classes interact with each other, and in this interaction the classes refer to types with the same symbolic name but with different class objects. In the example,  the self-first servlet referred to EJB:LoginEJB.login(sfWeb:User), but the EJB's representation was EJB:loginEJB(EJB:User).

Other places where linkage errors may occur are in class hierarchies. If class Derived overrides a method f(A a, B b) in class Super, A and B as seen from Super must be the same A and B as seen from Derived. References to static variables are another example.

More information can also be found in the book Inside the Java Virtual Machine by Bill Venners. Chapters from this book are also available at Artima.

Classloader lab

If you like to experiment, here's the source of the Demo program. (click to expand)

package com.stc;
 
 import java.io.InputStream;
 import java.util.Arrays;
 import java.util.HashSet;
 import java.util.Set;
 
 /**
  * Demonstrates Linkeage errors
  * 
  * @author fkieviet
  */
 public class Demo {
     /**
      * A self-first delegating classloader. It only loads specified classes self-first;
      * other classes are loaded from the parent. 
      */
     private static class CustomCL extends ClassLoader {
         private Set<String> selfFirstClasses;
         private String label;
 
         public CustomCL(String name, ClassLoader parent, String... selfFirsNames) {
             super(parent);
             this.label = name;
             this.selfFirstClasses = new HashSet<String>(Arrays.asList(selfFirsNames));
         }
 
         public Class<?> findClass(String name) throws ClassNotFoundException {
             if (selfFirstClasses.contains(name)) {
                 try {
                     byte[] buf = new byte[100000];
                     String loc = name.replace('.', '/') + ".class";
                     InputStream inp = Demo.class.getClassLoader().getResourceAsStream(loc);
                     int n = inp.read(buf);
                     inp.close();
                     System.out.println(label + ": Loading " + name + " in custom classloader");
                     return defineClass(name, buf, 0, n);
                 } catch (Exception e) {
                     throw new ClassNotFoundException(name, e);
                 }
             }
 
             // Is never executed in this test
             throw new ClassNotFoundException(name);
         }
 
         public Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException {
             if (findLoadedClass(name) != null) {
                 System.out.println(label + ": already loaded(" + name + ")");
                 return findLoadedClass(name);
             }
 
             // Override parent-first behavior into self-first only for specified classes
             if (selfFirstClasses.contains(name)) {
                 return findClass(name);
             } else {
                 System.out.println(label + ": super.loadclass(" + name + ")");
                 return super.loadClass(name, resolve);
             }
         }
 
         public String toString() {
             return label;
         }
     }
 
 
     public static class User {
     }
     
     public static class LoginEJB {
         static {
             System.out.println("LoginEJB loaded");
         }
         public static void login(User u) {
         }
     }
     
     public static class Servlet {
         public static void doGet() {
             User u = new User();
             System.out.println("Logging in with User loaded in " + u.getClass().getClassLoader());
             LoginEJB.login(u);
         }
     }
     
     public static void test1() throws Exception {
         CustomCL ejbCL = new CustomCL("EJB  ", Demo.class.getClassLoader(), "com.stc.Demo$User", "com.stc.Demo$LoginEJB");
         CustomCL pfWebCL = new CustomCL("PFWeb", ejbCL, "com.stc.Demo$Servlet");
         CustomCL sfWebCL = new CustomCL("SFWeb", ejbCL, "com.stc.Demo$User", "com.stc.Demo$Servlet");
 
         System.out.println("Loading EJB");
         ejbCL.loadClass("com.stc.Demo$LoginEJB", true).newInstance();
         
         System.out.println("Logging in, self-first");
         sfWebCL.loadClass("com.stc.Demo$Servlet", false).getMethod("doGet").invoke(null);
         
         System.out.println("Examining methods of LoginEJB");
         ejbCL.loadClass("com.stc.Demo$LoginEJB", false).getMethods();
 
         System.out.println("Logging in");
         pfWebCL.loadClass("com.stc.Demo$Servlet", false).getMethod("doGet").invoke(null);
     }
     
     public static void main(String[] args) {
         try {
             test1();
         } catch (Throwable e) {
             e.printStackTrace(System.out);
         }
     }
 }
 

Time running backwards in Microsoft's Hyper-V

You would think that subsequent calls to System.nanoTime() would return ever increasing values. After all, time cannot run backwards. However, when you run a JVM on the Hyper-V virtualization platform, it turns out that time may actually run backwards.

Hence, if you use this timer to measure time differences, you need to account for negative differences. This was a problem that I had not accounted for in the Hulp Measuring package.

Interview with DZone

Last week Masoud Kalali asked me for an interview about OpenESB. Today the interview was published. Nothing new for those that already known OpenESB and Java CAPS, but for people who look at it for the first time it may provide some useful background information.

An interactive Regular Expression Editor

The other day I had to filter an HTTP access log. I figured I could write a dozen or two lines of Java code to do the job, to be written with a dozen or so test-fix cycles. But I also knew that it could be done in just a few lines when using a regular expression. The things that I don't like about using regular expressions in Java code is the fact that the regular expression becomes difficult to edit because the backslash and quote characters need to be escaped with an extra backslash, and secondly I typically need even more test-fix cycles to get the expression right. Thirdly, a non-trivial regular expression becomes difficult to read.

It occurred to me that much of my reservations could be eliminated with an interactive editor that would allow me to edit the regular expression in plain text, and that would show me the results immediately. Since I've been wanting to look at NetBeans' Matisse editor, I thought I'd be able to whip out an editor in less than half an hour, and by doing so, save time not just for the project I was working on, but also for whenever I need to edit a regular expression in the future.

The Matisse editor in NetBeans is as easy as advertised. It took me only minutes to put together a basic editor. To my satisfaction, the time savings in editing regular expressions immediately proved to be enormous!

That was a few weeks ago. This week I was on vacation, and to kill time in the airplane, I've been upgrading this editor further. What I've added is the ability to convert the regular expression in the editor and convert it into Java. That is, the tool now takes care of escaping special characters such as backslashes. The tool is now sort of a code generator. But how about round-trip engineering? I also threw in an option to take a snippet of Java code with a regular expression, and remove the escaping characters. This makes it easy when you later need to revisit a regular expression: you copy the expression from the Java code, paste it into the editor, hit convert and you can edit the plain text regular expression. When done, generate the expression to Java code, paste it back in the Java source file.

Sounds easy, no? What else could you wish for? Of course the whole thing could be integrated into NetBeans and / or Eclipse. Maybe an exercise for another plain trip.

Use the Interactive Regular Expression Editor!

I’ve made the application into a JNLP application, so you can launch it from the browser.

Launch Interactive Regular Expression Editor

How do you use the editor? The screen consists of three edit panes. The middle editor is where you edit the regular expression. Any time that you change the expression, the sample input in the top pane is evaluated, and the results are displayed in the bottom edit pane. When you're done editing the expression, click the "generate" button, and Java code will appear in the bottom pane. Copy that code and use it in your Java programs. If you later on need to edit the expression again, simply paste the Pattern definition in the regular expression pane, and hit the "extract" button. The code in the middle pane will be replaced with the clear text version.

Tips for working with Regular Expressions

Here are some tips for working with regular expressions: use the Pattern.COMMENTS flag:  it allows you to  break up your regular expression into multiple lines. Doing so makes it possible to document the parts that the regular expression consists of. This leads me to the second tip: document the parts of your expression so that you'll make it a lot easier on yourself if you later have to revisit the expression. Commenting is easy: use the # sign just like you would use // in Java. Whitespace in the expression is ignored, so if you want to insert a literal space, escape it with a backslash.

Parsing text with regular expressions

I knew that regular expressions are great for matching text, but now that I have the interactive regular expression editor, I also came to appreciate the regular expressions to parse strings -- I mean to extract fragments out of a string. It surely is a lot easier than finding positions in a string and then using String.subString().

For example, here is a regular expression with capturing groups to parse the GlassFish log file:

# Begin marker
\[\#

# Date and time
\|  (\d\d\d\d-\d\d-\d\d)\D(\d\d:\d\d:\d\d\.\d\d\d\D\d\d\d\d)

# Level
\|  (.+?)

# Product
\|  (.+?)

# Category
\|  (.+?)

# Key-value pairs
\|  (.+)?

# Msg text
\|  (.+?)

# Optional stack trace
(^\t at \s \p{javaLowerCase} .*  \.java .*)?

# End marker
\|\#\]

This will result in 8 groups. This looks like a difficult expression, but with the Interactive regular expression editor, it's pretty simple to write this expression.And showing off the Interactive regular expression editor, this is what Java code it produces:

public static Pattern REGEX = Pattern.compile("# Begin marker\r\n" + 
  "\\[\\#\r\n" + 
  "\r\n" + 
  "# Date and time\r\n" + 
  "\\|  (\\d\\d\\d\\d-\\d\\d-\\d\\d)\\D(\\d\\d:\\d\\d:\\d\\d\\.\\d\\d\\d\\D\\d\\d\\d\\d)\r\n" + 
  "\r\n" + 
  "# Level\r\n" + 
  "\\|  (.+?)\r\n" + 
  "\r\n" + 
  "# Product\r\n" + 
  "\\|  (.+?)\r\n" + 
  "\r\n" + 
  "# Category\r\n" + 
  "\\|  (.+?)\r\n" + 
  "\r\n" + 
  "# Key-value pairs\r\n" + 
  "\\|  (.+)?\r\n" + 
  "\r\n" + 
  "# Msg text\r\n" + 
  "\\|  (.+?)\r\n" + 
  "\r\n" + 
  "# Optional stack trace\r\n" + 
  "(^\\t at \\s \\p{javaLowerCase} .*  \\.java .*)?\r\n" + 
  "\r\n" + 
  "# End marker\r\n" + 
  "\\|\\#\\]", 0 | Pattern.COMMENTS | Pattern.DOTALL | Pattern.MULTILINE);

Blogger.com works after all

I had this issue with blogspot where it would insert a great number of line breaks just before a table.  As I found out  (thanks to Edward Chou), this behavior can be turned off in the settings page of Blogspot.

I also found out how to create Atom feeds from labels. The help text on Google is incorrect, and I found out to reference a category like Sun, the corresponding URL is: http://frankkieviet.blogspot.com/feeds/posts/default/-/Sun

Now I’m wondering if I should stick with Blogspot or with Wordpress…