How to enlist and delist XA resources: trial and error

Once in a while a problem pops up with how a middleware stack supports XA. The problem I'm discussing here is the one of how to exactly enlist and delist XAResource-s with isSameRM()returning true. The XA spec is vague on this very topic, so this warrants some exploration.

First of all, what do I mean with "enlist and delist XAResource-s with isSameRM()returning true?"

Enlisting and delisting

When using an XA capable system, the connection needs to be enlisted in the transaction before it is used. A typical sequence on the XAResource, let's call it r1, is:

   1: r1.start(xid, TMNOFLAGS);

   2: ... // use (r1)

   3: r1.end(xid, TMSUCCESS);

   4: r1.commit(xid, true);

With a second resource in the mix, the call sequence may become:

   1: r1.start(xid1, TMNOFLAGS);

   2: ... // use r1

   3: r2.start(xid2, TMNOFLAGS);

   4: ... // use r2

   5: r2.end(xid2, TMSUCCESS);

   6: r1.end(xid1, TMSUCCESS);

   7: r1.prepare(xid1);

   8: r2.prepare(xid2);

   9: p1.commit(xid1, false);

  10: p2.commit(xid2, false);

The transaction is now escalated to a full two-phase transaction. But if r1 and r2 represent the same resource manager, a shortcut can be taken: one resource can "piggy back ride" on the other resource. The isSameRM() method can be used to check if this should be attempted. If it returns true, the sequence may become:

   1: // MULTIPLE ACTIVE

   2: r1.start(xid, TMNOFLAGS);

   3: r1.isSameRM(r2); // returns true

   4: r2.start(xid, TMJOIN);

   5: r2.end(xid, TMSUCCESS);

   6: r1.end(xid, TMSUCCESS);

   7: r1.commit(xid, true);

The transaction now again is a single-phase transaction, and the performance difference with the two-phase commit case is usually very significant.

This sequence doesn't work for a number of popular systems, e.g. MQSeries and Oracle. Unfortunately the XA specification, but that specification is not quite clear at all about what resource managers should be able to do, so there is to some extent some trial and error involved.

Trial and error

How should isSameRM() be used then? By testing a number of different systems, it turns out that for some systems, there can be only one enlisted XAResource active in the transaction. If a second XAResource joins the transaction, the first one should be deactivated. Here is an example:

   1: // SINGLE ACTIVE 1

   2: r1.start(xid, TMNOFLAGS);

   3: r1.isSameRM(r2); // should return true

   4: ... // use r1

   5: r1.end(xid, TMSUSPEND);

   6:  

   7: r2.start(xid, TMJOIN);

   8: ... // use r2

   9: r2.end(xid, TMSUCCESS);

  10:  

  11: r1.start(xid, TMRESUME);

  12: ... // use r1 again

  13: r1.end(xid, TMSUCESS);

  14:  

  15: r1.commit(xid, true);

A variation of this is calling TMSUCCESS instead of TMSUSPEND. In that case TMJOIN should be called instead of TMRESUME:

   1: // SINGLE ACTIVE 2

   2: r1.start(xid, TMNOFLAGS);

   3: r1.isSameRM(r2); // should return true

   4: ... // use r1

   5: r1.end(xid, TMSUCCESS);

   6:  

   7: r2.start(xid, TMJOIN);

   8: ... // use r2

   9: r2.end(xid, TMSUCCESS);

  10:  

  11: r1.start(xid, TMJOIN);

  12: ... // use r1 again

  13: r1.end(xid, TMSUCESS);

  14:  

  15: r1.commit(xid, true);

I ran these tests on a number of systems, and here is what I found:

	isSameRM()	Test: multiple active	Test: single active 1	Test: single active 2
STCMS	yes	yes	yes	yes
JMQ	as of 4.4	yes	no: throws on line 7	yes
WebSphereMQ 6	yes	no: blocks on line 4	no: blocks on line 7	yes
Derby 10.5.3.0	yes	no: blocks on line 4	yes	yes
Oracle 11.1.0.6	yes	no: blocks on line 4	yes	yes
MySQL 5.1	yes	no: throws on line 4	no: throws on line 5	no: throws on line 7

As can be seen, the SINGLE ACTIVE 2 code sequence works best.

As a side note, MySQL is showing surprising behavior: TMJOIN and TMSUSPEND are not supported (as documented in the MySQL documentation), so why does it bother to return true on isSameRM()? Behavior like this makes it difficult to write portable code: a container now has to provide a wrapper around the DataSource that corrects for this. It would have been much better if it simply had returned false on isSameRM().

How does this relate to what application code can do in for instance a Java EE container? That's the topic of a different blog post.

Regex editor now on code.google.com

By popular demand, the sources of the Regular Expression Editor I discussed in a previous post, is now available on code.google.com.

Book "SOA with Java": Rough cut now online

A few months ago I got a request from Satadru Roy, one of the authors of the book "SOA with Java", to write a section on Open Source ESBs for this book.

This was a good opportunity to advertise OpenESB, so I eagerly said yes. Satadru suggested that a practical example should be the core of the chapter, so I sat down with two of my colleagues, Murali Pottlapelli and Sujit Biswas, to discuss how the example would look like.

The result is a 17 page chapter that gives an overview of OpenESB. The book is now available for review on Safari. This link will probably be invalidated when the review period ends, but perhaps the link to the book itself is more permanent (the book is also on Amazon).

Remote learning, graduate projects, and open source

Last week, an ex-colleague asked me if I could be a reviewer on a graduation project at National University where he is now a professor. Flattered, and finding myself with a little bit more time on my hands than usual, I accepted.

The goal of the project was to let a team of seven students build a real-life application as a learning experience about how software development is done in the industry, for a "real" customer. The task was to develop a tracking application for the Boys & Girls Clubs of the Sequoias.

There were several interesting aspects of this project: first of all, the end result was a working application that in quality and functionality exceeds a good portion of similar applications built by IT departments. Key to this was the use of off-the-shelf frameworks such as Spring MVC, Hibernate, BIRT, and jQuery. I'm glad that students are using frameworks like these rather than reinventing their own, although that might have been even more fun.

Another interesting aspect was that the project was done as an open source project on code.google.com. The practical advantages are the infrastructure is provided completely ready by the code forge (in this case code.google.com), so that little time needs to be spent on setting up an infrastructure. This is especially important for students that are remote and hence don't have a shared work environment (e.g. office). Of course there's a lot more work to be done to make it a successful open source project, but that's not important in the scope of the project. I'm glad universities expose students to an open source way of development.

Lastly, the presentation of the project was done using a tool called ClassLive Pro. This is a JNLP application that provides similar functionality to WebEx. Although I've developed a few JNLP applications myself, and I'm convinced of its great utility, I haven't encountered any other JNLP applications so I was glad I ran into this application.

A SEED certificate

With the Oracle acquisition looming on the horizon, and the uncertainty that comes with it, it feels like many organizations within Sun are bringing projects in a state in which they can be transitioned in a clean way. Transitioned meaning adopted or abandoned.

I also find myself doing this: for instance I'm trying to put the finishing touches on JMSJCA, and trying to put Hulp in a state where other people can use it, etc.

Maybe it was in that light that a few days ago, I got a certificate from the SEED organization within Sun. SEED is a mentoring program within Sun in which I had the good fortune to participate. I also was a SEED mentor. Or perhaps I should let the speak for itself.

Lessons of an interesting deadlock problem

Two years ago I wrote a blog entry about Nested Diagnostics Contexts in GlassFish. The approach was based on custom Logger objects. Now, two years later when running GlassFish on AIX, an issue with that turned up: a deadlock. Digging into it, there are two lessons I'd like to share: one about deadlocks, and the other one about workarounds.

Deadlock in the JDK

At the time of my writing the NDC facility, calling addLogger() and getLogger() from different threads may cause a deadlock in the JDK classes. Why is this?

This is how the code in the Sun JDK was:

public class Logger {
...
    public static synchronized Logger getLogger(String name) {
	LogManager manager = LogManager.getLogManager();
	Logger result = manager.getLogger(name);
	if (result == null) {
	    result = new Logger(name, null);
	    manager.addLogger(result);
	    result = manager.getLogger(name);
	}
	return result;
    }
}

public class LogManager {
...
    public synchronized boolean addLogger(Logger logger) {
...
        Logger plogger = Logger.getLogger(pname);
    }
...
}

There is a problem if there are two threads calling these two methods. Say Thread 1 calls Logger.getLogger() in its application code, e.g.

public void doSomething() {
  Logger.getLogger("com.stc.test").info("doSomething!");
}

Now let's consider another thread Thread 2 tries to register a custom logger using addLogger(), e.g.

LogManager.getLogManager().addLogger(new Logger() {
...
});

There are two locks that come into the picture: one is the lock on Logger.class, and the other one on LogManager.getLogManager(). Thread 1 simply calls Logger.getLogger() which first locks Logger.class, and while having this lock, will call LogManager.addLogger(). This call will get a lock on LogManager.getLogManager(). This will try to lock Logger.class again, which is no problem because it already had the lock on that. Now consider Thread 2: it will first lock LogManager.getLogManager() through its call to addLogger(), and while having this lock, it will try to lock Logger.class through its call to Logger.getLogger().

So Thread 1 will lock first A and then B, while Thread 2 will first lock B and then A. A classical deadlock situation.

This is clearly a bug in the JDK. With the knowledge of this bug, as the developer of the custom logger, we can use a simple workaround for this problem. Before calling addLogger(), first lock Logger.class. By doing that, we can guarantee that the locks are called in the same order: first Logger.class and then LogManager.getLogManager().

This bug was actually reported by a customer, see bug report, and was fixed in 6u11 and 5.0u18. The LogManager now no longer calls Logger.getLogger(), so there's no locking of Logger.class anymore.

Deadlock in GlassFish

GlassFish installs its own LogManager, and there's some locking going on there too. It has an internal lock which it locks in addLogger():

    public boolean addLogger(Logger logger) {
...
    super.addLogger(logger);
...
    synchronized (_unInitializedLoggers) {
...
        doInitializeLogger(logger);
    }
}

And this calls Logger.getLogger() again.

    protected void doInitializeLogger(Logger logger) {
...
    Logger newLogger =
        Logger.getLogger(logger.getName(), 
            getLoggerResourceBundleName(logger.getName()));
...
    }

As a developer of the custom logger, we can still use the same fix: first lock Logger.class before calling addLogger() so that the locking sequence becomes: Logger.class, transient lock on LogManager.getLogManager() followed by a lock on _unInitializedLoggers, lock on Logger.class().

A fix in GlassFish would simply make addLogger() synchronized. An alternative fix is to make the synchronized block smaller: it should not extend over doInitializeLogger().

Deadlock with the IBM JDK

All was well until this was run on the IBM JDK. As it turns out, Logger.getLogger() doesn't lock Logger.class, it locks LogManager.getLogManager(). Now there is a new deadlock: Thread 1 calls Logger.getLogger() and by doing so, locks LogManager.getLogManager(), and then tries to lock _unInitializedLoggers and later tries to lock LogManager.getLogManager() again. Thread 2 calls addLogger() and by doing so locks first _unInitializedLoggers which later tries to lock LogManager.getLogManager(). The problem is back: the sequence of locking is reversed.

Lessons learned

The first lesson is that of the deadlock itself: one should be extremely careful when locking an object, and then calling into another object, especially if that object is meant to be a baseclass in a public API. Both Logger and LogManager may be overridden, and are clearly very public.

The second lesson is that instead of building workarounds, try to get a fix the original bug. Or at least try to have the original issue addressed while adding a workaround. In my case, I never filed a ticket against the JDK or against GlassFish. I should have known better.

... or not?

At this very moment, GlassFish v2.1.1 is about to be shipped, and we need GlassFish v2.1.1 on AIX for the upcoming CAPS release. A bug fix in GlassFish v2.1.1 is probably not going to get in. So I'm making yet another workaround: don't use a custom logger at all as to avoid the whole addLogger() problem. Instead, use standard Logger objects, but provide a custom filter that manipulates the log record. This will work on both unpatched versions of GlassFish, and also on unpatched versions of the JDK. Still, I hope the GlassFish problem will be fixed.

JavaOne 2009 TS-5341: Rethinking the ESB: lessons learned challenging the limitations and pitfalls – audio recording uploaded

At JavaOne 2009 I gave a number of presentations. One was "Rethinking the ESB: lessons learned challenging the limitations and pitfalls" which I did with Andi Egloff. JavaOne sessions were not recorded this year, that is not by the JavaOne Organization. But I used an mp3 player to record the sessions I was involved in. I finally processed the audio and uploaded it. I also uploaded a PDF of the presentation:

• Presentation (PDF) of TS-5341 Rethinking the ESB: lessons learned challenging the limitations and pitfalls
• Audio recording (mp3) of TS-5341 Rethinking the ESB: lessons learned challenging the limitations and pitfalls