Wednesday, May 16, 2007

JavaOne 2007

All of last week I was at JavaOne. It was an exhausting but very interesting week. Like last year, there were many interesting sessions, too many to list them here. Let me just mention the one I enjoyed most was the one by Neal Gafter on Closures for the Java Programming Language (BOF-2358). I can't wait until they're in the Java language!

Not only did I attend sessions and BOFs, I also presented BOFs. Three of them to be precise. I recorded the audio on my MP3 player. Unfortunately the quality of the audio is pretty bad. I'm posting the audio recordings below. I'm also posting the slides. Here they are:

BOF8847: Developing Components for Java Business Integration: Binding Components and Service Engines

Presented by Frank Kieviet, Alex Fung, Sherry Weng, and Srinivasan Chikkala
Attendance: about 100

You cannot cover how to write JBI components in just 45 minutes. We were also not sure about what the audience was interested in. That's why we assumed that the audience would consist mostly of people who have never written a JBI component before, and are relatively new to JBI. That's why we decided to talk mostly about general information on JBI and JBI components, and highlight the power of JBI and discuss how to go about developing one.

As an experiment I wanted to try a new format (at least new for me): rather than slicing up the session into four parts of 10 minutes, we cast the session into a "discussion forum". Of course the questions and answers (and even the jokes) were well rehearsed.

Unfortunately, the audio/visual people that control the meeting rooms, had forgotten to start the session timer. As a result the audio was cut unexpectedly just a minute before we could finish up.

Nevertheless, I think it was an interesting session.

Presentation JavaOne07-BOF8847 (pdf)

Audio JavaOne07-BOF8847 (mp3)


BOF8745: Leveraging Java EE in JBI and vice versa

Presented by Frank Kieviet and Bhavanishankara Sapaliga
Attendance: about 60

This BOF was originally to be presented by Vikas Awasthi and Bhavanishankara Sapaliga, but Vikas couldn't make it, so I replaced him. We focused the session on how JBI and EE can play together, trying to make it interesting for both JBI application developers as well as for EE developers. At the end I ran a demo with NetBeans showing three different scenarios. The demo-gods were with me: the demo went very smoothly. Unfortunately I forgot to demo how to add an EJB to a composite application. Another valuable lesson learned.

Presentation JavaOne07-BOF8745 (pdf)

Audio JavaOne07-BOF8745 (mp3)


BOF9982: The java.lang.OutOfMemoryError: PermGen Space error demystified

Presented by Edward Chou and Frank Kieviet
Attendance: about 116

This session was on Thursday night at 10pm. That night was the JavaOne After dark bash. Free beers, music and snacks for everyone. Therefore we didn't expect much of an attendance: memory leaks are a rather dry subject, and why leave the party early to go to this session? Also, some of our thunder had been stolen by SAP who demo-ed a tool to track memory leaks in a morning-session earlier that week. So we were quite surprised when about 116 people turned up for our session. Most stayed until the very end, and there were also quite a few interesting questions. Apparently a lot of people struggle with memory leaks in permgen space -- in my presentation I mention that I get about a hundred hits on my blog every day from people who search for this memory exception in Google.

Presentation JavaOne07-BOF9982 (pdf)

Audio JavaOne07-BOF9982 (mp3)

Tuesday, May 1, 2007

JavaOne / memory leaks revisited...

Memory leaks in print 

A few months ago, Gregg Sporar together with A. Sundararajan started an article on memory leaks in the magazine Software Test & Performance. While writing that, he stumbled upon my blog and decided to cover the "java.lang.OutOfMemoryError: PermGen space" exception too. I offered to collaborate on the article. The article eventually grew so much it was split in two. Part one was published a month ago. Yesterday, part two was published.

Memory leaks at JavaOne

Edward Chou submitted a proposal for a BOF at JavaOne 2007. He and I will be presenting a BOF on the "java.lang.OutOfMemoryError: PermGen space" exception. I'll try to record the session with my MP3 player and post it on my blog.

In preparation for our presentation, we've been looking at some real-life examples of permgen memory leaks. We took a few memory dumps that came from actual customers in actual production environments. We discovered a few more improvements we could make to jhat: it was already fairly simple to track the leaks with jhat; with these changes it becomes really simple. We were actually quite surprised how simple. More on that in a future entry, either on my blog or on Edward's.

More at JavaOne

Speaking about JavaOne... I have my hands full. Next to the memory leaks BOF, I'm also presenting a BOF on JBI ("How to develop JBI components") and I'll be co-presenting another BOF on "EE and JBI."


Tuesday, February 13, 2007

Using Resource Adapters outside of EE containers

Java EE (J2EE) application servers usually are the ideal choice to host your back-end applications, especially if your applications require transactions, security or state management. Running in an application server, your application can easily connect to external systems provided by Resource Adapters. E.g. if your application would have to connect to a CRM (e.g. PeopleSoft), to an ERP (e.g. SAP) or to a system such as JMS, or even to a database (e.g. Oracle), it would use a Resource Adapter. Through this, the application would use advanced features such as connection pooling, connection failure detection and recovery, transactions, security propagation, etc.

As said, EE or J2EE application servers are ideal containers for hosting business side logic. But it's not the best choice in literally every situation. There are cases where you would prefer to write your application as a stand alone Java program. There's nothing strange about that: you should always use the best tools for the job at hand, and no single tool is best in all situations.

Now say that you need to write a stand-alone Java application, and in that application you would need to connect to an external system. Wouldn't it be nice to be able to use an off-the-shelf Resource Adapter for this connectivity, so that you would not have to hand-code features such as connection pooling, connection failure detection and recovery etc? As I will show, this is not that difficult.

Hacking a Resource Adapter

Resource Adapters are distributed in RAR files, i.e. a file with a .rar extension. A RAR is nothing more than a ZIP file. When you open up a RAR, you will see a bunch of jars and a descriptor file with the name ra.xml. In a nutshell, this is what you need to do to use a Resource Adapter:

  1. Add the jars to your application's classpath
  2. Instantiate the Resource Adapter classes (you can find out which ones by examining the ra.xml)
  3. Configure the Resource Adapter by calling a few setter methods (you can find which ones by examining the ra.xml)
  4. Activate the Resource Adapter (you can find out how by reading the Java Connector API specification, or just read on)

These four steps are basically what the application server does when it loads a Resource Adapter. There's a lot of logic involved in this, but in a stand-alone Java application we can take a lot of short-cuts so that the logic that we need to write is not too involved.

There's quite a big difference in how a Resource Adapter is used to provide outbound connectivity versus inbound connectivity. With outbound connectivity, I mean a situation where an application obtains a connection to an external system and reads or writes data to it. With inbound I mean a situation where the Resource Adapter listens for events from the external system and calls into your application when such an event occurs.

As an example we will use the JMSJCA resource adapter. This is an open-source adapter for JMS connectivity to various JMS servers.

Outbound connectivity

In a nutshell what we need to do is instantiate the Resource Adapter class, configure it, instanatiate a Managed Connection Factory, and from that obtain the application-facing connection factory.

When you open up the ra.xml file, you can find out class implements the ResourceAdapter interface:

    <resourceadapter>
        <resourceadapter-class>com.stc.jmsjca.unifiedjms.RAUnifiedResourceAdapter</resourceadapter-class>

The Resource Adapter class has per the specification a no-args constructor and should implement the javax.resource.spi.ResourceAdapter interface. You could instantiate the ResourceAdapter simply by doing this:

    com.stc.jmsjca.unifiedjms.RAUnifiedResourceAdapter ra = new com.stc.jmsjca.unifiedjms.RAUnifiedResourceAdapter();

The drawback of this is that, although the chances of that being small, if the classname changes in future versions of the Resource Adapter, your application would no longer compile. A better approach would be to read the classname dynamically from the ra.xml. I'll leave that as "an excercise for the reader".

The Resource Adapter is a Java Bean with getters and setters. This is how you can configure the Resource Adapter. For example, we could set the connection URL as follows:

    ra.setConnectionURL("stcms://localhost:18007");

Next, we need to instantiate a  ManagedConnectionFactory.  Again from  the ra.xml, you can find the classname:

    <outbound-resourceadapter>
<connection-definition>
<managedconnectionfactory-class>com.stc.jmsjca.core.XMCFUnifiedXA</managedconnectionfactory-class>
 

Again, the ManagedConnectionFactory must have a no-arg constructor and is a Java bean, so you can simply instantiate and configure one as follows:

    com.stc.jmsjca.core.XMCFUnifiedXA mcf = new com.stc.jmsjca.core.XMCFUnifiedXA();
mcf.setUserName("Administrator");
mcf.setPassword("STC");

Next, you may need to associate the newly created ManagedConnectionFactory with the ResourceAdapter. Those that require this association (most likely all of them), implement the javax.resource.spi.ResourceAdapterAssociation interface. This leads to the following code:

    mcf.setResourceAdapter(ra);

Lastly, you create the application-facing connection factory. In case of JMS, this is a javax.jmx.ConnectionFactory. You can find evidence of this in the ra.xml:

   <outbound-resourceadapter>
<connection-definition>
<managedconnectionfactory-class>com.stc.jmsjca.core.XMCFUnifiedXA</managedconnectionfactory-class>
...
<connectionfactory-interface>javax.jms.ConnectionFactory</connectionfactory-interface>
<connectionfactory-impl-class>com.stc.jmsjca.core.JConnectionFactoryXA</connectionfactory-impl-class>
<connection-interface>javax.jms.Connection</connection-interface>
</connection-definition>

This leads to the following code:

javax.jms.ConnectionFactory f = (javax.jms.ConnectionFactory) mcf.createConnectionFactory();

Now, putting it all together, this is what you would need to create a JMS connection factory from the JMSJCA Resource Adapter:

    com.stc.jmsjca.unifiedjms.RAUnifiedResourceAdapter ra = new com.stc.jmsjca.unifiedjms.RAUnifiedResourceAdapter();
com.stc.jmsjca.core.XMCFUnifiedXA mcf = new com.stc.jmsjca.core.XMCFUnifiedXA();
ra.setConnectionURL("stcms://localhost:18007");
ra.setUserName("Administrator");
ra.setPassword("STC");
ra.setOptions("JMSJCA.NoXA=true");
mcf.setResourceAdapter(ra);
javax.jms.ConnectionFactory f = (javax.jms.ConnectionFactory) mcf.createConnectionFactory();

And that's all there's to it

As I mentioned, one of the advantages of using a Resource Adapter over using a client runtime directly, is that an Resource Adapter typically provides connection pooling and other nifty features. JMSJCA for instance provides a powerful and configurable connection manager with blocking behavior, time-out behavior, connection failure detection, etc. It even enlists the connection in the transaction if it detects that there is a transaction active when the connection is created.

Not all resource adapters will provide such a comprehensive connection manager: check the documentation of the resource adapter that you're planning to use. If it doesn't provide a connection manager to your liking, you can provide your own connection manager. If you need to write one, take a look at the one in JMSJCA: you may want to use it as a starting point.

Inbound connectivity

Inbound connectivity is where a resource adapter is used to receive messages from an external system; the resource adapter delivers these messages to a Message Driven Bean. In the case of JMS, this is javax.jms.MessageListener, with its void onMessage(javax.jms.Message) method. For other types of resource adapers, you will find other Message Driven Bean interfaces. Look in the ra.xml to find out which one.

Seting up inbound connectivity is a bit more involved than outbound connectivity. That is because with inbound, you need to explain to Resource Adapter how it should obtain a new instance of the Message Driven Bean, and how to obtain a thread that will call the onMessage() method or equivalent method.

We start with instantiating and configuring a ResourceAdapter object; the class can be found in ra.xml as I showed in the Outbound Connectivity section:

    com.stc.jmsjca.unifiedjms.RAUnifiedResourceAdapter ra = new com.stc.jmsjca.unifiedjms.RAUnifiedResourceAdapter();
ra.setConnectionURL("stcms://localhost:18007");
ra.setUserName("Administrator");
ra.setPassword("STC")

Next, we need to call start() on the ResourceAdapter object. This method takes a javax.resource.spi.BootstrapContext object. You need to provide an implementation for this class; the most important method that you need to implement is the public javax.resource.spi.work.WorkManager getWorkManager() method. As you can see, this method should return a WorkManager object. This class has a number of methods that provide access to a threadpool. It would help at this point if you have a bit of knowledge of the internals of the Resource Adapter that you're trying to work with: some Adapters don't use a WorkManager, and the ones that do, typically only use one of the methods on the WorkManager object. For instance, here's a WorkManager that could be used with JMSJCA:

public class XWorkManager implements WorkManager {
private java.util.concurrent.Executor mPool;

public XWorkManager(int poolsize) {
mPool = new PooledExecutor(new LinkedQueue(), poolsize);
}

public void scheduleWork(Work work) throws WorkException {
try {
mPool.execute(work);
} catch (InterruptedException e) {
throw new WorkException(e);
}
}

// other methods just throw an exception
}

Fortunately we can make use of the java.util.concurrent tools introduced in JDK 5.0 for a threadpool. Next, we need to provide an implementation for javax.resource.spi.endpoint.MessageEndpointFactory. This is an interface with only two methods. One is there to indicate if the message delivery should be transacted, and the other one to is there to create a MessageEndpoint. This is a class that is a proxy around the Message Driven Bean. This proxy should implement the onMessage() or equivalent method which should simply delegate to the MessageListener or equivalent object in your application. The proxy should also implement three additional methods:

    void afterDelivery()
void beforeDelivery(Method method)
void release()

The beforeDelivery() and afterDelivery() methods are called just before the Resource Adapter calls the onMessage() or equivalent method. You could start and commit a transaction in these methods; if you're not using transactions, you can just leave these methods unimplemented. The release() method is called by the Resource Adapter when it's done using a Message Driven Bean. If you don't implement some sort of pooling mechanism for Message Driven Beans, and you probably won't, you can leave this method empty as well.

Now that you have implementations for the BootstrapContext, WorkManager, MessageEndpointFactory, and MessageEndpoint, you can finally tell the Resource Adapter to start delivering messages to your Message Driven Bean. You do that by calling the void endpointActivation(javax.resource.spi.endpoint.MessageEndpointFactory, javax.resource.spi.ActivationSpec) method on the ResourceAdapter. The ActivationSpec object is implemented by the Resource Adapter; you can find the classname in the ra.xml file. It is a Java bean that is used to configure the message delivery. For instance, in the case of JMS, you specify from which queue or topic to get messages.

As you can see, the inbound connectivity case is not as straight forward as the outbound connectivity case, but still very much doable.

Real examples

For a full sample source listing, take a look at the test suite in JMSJCA You can also look at the JMSBC as part of the Open JBI Components project. This Binding Component uses the JMSJCA Resource Adapter as described in this blog entry. By doing so, a lot of development effort was saved dealing with all the idiosyncracies of various JMS server implementations, connection management, etc.

Tuesday, January 16, 2007

JMSJCA, a feature rich JMS Resource Adapter, is now a java.net project

JMSJCA is now a java.net project. It can be found here: http://jmsjca.dev.java.net.

The project is currently used in Java CAPS, JMS Grid and the JMS BC as part of the open-jbi-components project.

The connector can be used as a J2EE 1.4 Resource Adapter, but its libraries can also be used  as an abstraction layer to JMS servers from  non J2EE-code. As such, the adapter acts like a library that hides the complexities of transactions, concurrency, connection failure detection, JMS server implementation idiosyncracies, etc. That is how it is used in the JMS BC as part of the open-jbi-components project.

Sunday, January 14, 2007

Logless transactions

A few months ago, in my blog entry Transactions, disks, and performance I went into the importance of minimizing the number of writes. Transaction logging is one of those cases where minimizing the number of writes greatly enhances performance. In this entry, I'll describe a way to avoid transaction logging altogether.

What is transaction logging? Transaction logging refers to persisting the state of a two-phase transaction so that in the event of a crash, the transaction can either be committed or rolled back (recovered). I won't go into the details of what XA is; more information about XA transactions can be found elsewhere, e.g. in Mike Spille's XA Exposed.

Let me illustrate what recovery is using a "diagram". Consider an XA two phase transaction with three Resource Managers (RMa, RMb, and RMc). To indicate what happens at what time, I'll put all actions in a table; each row corresponds to a different time.

time
RMa
RMb
RMc
Coordinator
t1
start(xid1a, TMNOFLAGS)



t2

start(xid1b, TMNOFLAGS)

t3


start(xid1c, TMNOFLAGS)
t4
end(xid1a, TMSUCCESS)



t5

end(xid1b, TMSUCCESS)

t6


end(xid1c, TMSUCCESS)
t7
prepare(xid1a)



t8

prepare(xid1b)

t9


prepare(xid1c)
t10



log
t11
commit(xid1a, false)



t12

commit(xid1b, false)

t13


commit(xid1c, false)
t14



delete from log

At t10 the transaction manager records the decision to commit to the log. Let's say that the system crashes after t10, say between t11 and t12. When the system restarts, it will call recover() on all known Resource Managers and it will read the transaction log. In the transaction log it will find that xid1x was marked for commit. Through recover() it will find that xid1b and xid1c are in doubt. It knows that these two need to be committed because of the commit decision in the log.

What happens if the system crashes before the commit decision is written to the log, for example between t8 and t9? Upon recovery, the recover() method of RMa, RMb and RMc return xid1a and xid1b (but not xid1c because prepare was not called on RMc yet). The transaction manager will rollback RMa and RMb because no commit decision was found in the log.

SeeBeyond's Logless XA Transactions

Let's take a look at the recover() method on the XAResource. This method returns an array of Xid objects. Each Xid object holds two byte[] arrays. These two arrays represent the global transaction ID and the branch qualifier. They are typically random numbers picked by the transaction manager. The Resource Managers that receive these Xids should use these objects as identifiers and return them in the recover() method unmodified.

At SeeBeyond, Jerry Waldorf and Venugopalan Venkataraman came up with an idea to use the storage space in the byte[] arrays of the Xid as a way to persist the transaction state. Here's how it works. Let's modify the above example by removing transaction logging:

time
RMa
RMb
RMc
Coordinator
t1
start(xid1a, TMNOFLAGS)



t2

start(xid1b, TMNOFLAGS)

t3


start(xid1c, TMNOFLAGS)
t4
end(xid1a, TMSUCCESS)



t5

end(xid1b, TMSUCCESS)

t6


end(xid1c, TMSUCCESS)
t7


prepare(xid1c)

t8

prepare(xid1b)

t9
prepare(xid1a)


t10


commit(xid1c, false)
t11

commit(xid1b, false)

t12
commit(xid1a, false)



A commit decision is still being made, but this decision is no longer persisted in a separate transaction log. In stead, it is persisted in xid1a. If the system finds xid1a upon recovery, it knows that a commit decision was made. If it doesn't find xid1a, it knows that a commit decision was not made. Note that the order in which both prepare and commit are called on the three Resource Managers is very important.

As in the first example, if the system crashes before a commit decision has been made, it will rollback any resources upon recovery. E.g. if the system crashes between t8 and t9, it will encounter xid1c and xid1b and will call rollback() on these because it cannot find a record of a commit-decision for xid1, i.e. it cannot find xid1a. Hence, xid1b and xid1c need to be rolled back.

If the system crashes after a commit decision has been made, for example between t10 and t11, it will find xid1b and xid1a. Since xid1a signifies a commit decision, both xid1b and xid1a should be committed.

So far so good. But how does the transaction manager know that if it encounters xidb it should look for xida to figure out if a commit decision was made? This is where the transaction manager uses the byte[] of the Xid: it stores this information in one of them.

Complicating factors

A problem in this scheme occurs when the prepare(xid1a) method returns XA_RDONLY. If that happens, commit(xid1a, false) cannot be called, and RMa will not return xid1a upon calling recover(). Recall that xid1a had special significance! Hence it is important to order the Resource Managers such that the first one on which prepare() is called, is both reliable and will not return XA_RDONLY. However, in normal EE applications, the application prescribes in which order resources are enlisted in a transaction. Hence, to use this logless transaction scheme, the application server either needs to be extended with a way to specify resources a priori, or the application server needs to be extended with a learning capability so that it knows which resources are enlisted in a particular operation so that it can pick the right resource manager to write the commit decision to.

The SeeBeyond logless transaction approach is one of the ways that transaction logging can be made less exensive. In a future blog, I'll cover additional ones.

Sunday, December 10, 2006

Short note: Running Java CAPS on Java SE 6

Today Java SE 6 was released. It comes with many new features and cool tools. One of them being jmap as described in a previous log on permgen exceptions. The Integration Server is not officially supported on SE 6 yet. However, if you want to run the Java CAPS integration server on SE 6, this is what you can do:

  1.     Install JDK 6 somewhere, e.g. c:\\java
  2.     Install the IS somewhere, e.g. c:\\logicalhost
  3.     Rename c:\\logicalhost\\jre to c:\\logicalhost\\jre.old
  4.     Copy c:\\java\\jre1.6.0 to c:\\logicalhost\\jre
  5.     Copy c:\\java\\jdk1.6.0\\lib\\tools.jar to c:\\logicalhost\\jre\\lib
  6.     Copy c:\\logicalhost\\jre\\bin\\javaw.exe to c:\\logicalhost\\jre\\bin\\is_domain1.exe
  7.     Copy c:\\logicalhost\\jre\\bin\\javaw.exe to c:\\logicalhost\\jre\\bin\\ isprocmgr_domain1.exe
  8.     Edit c:\\logicalhost\\is\\domains\\domain1\\config\\domain.xml and comment out these lines:
<!--
<jvm-options>-Dcom.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager=com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager</jvm-options>
<jvm-options>-Dorg.xml.sax.driver=com.sun.org.apache.xerces.internal.parsers.SAXParser</jvm-options>
<jvm-options>-Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl</jvm-options>
<jvm-options>-Dcom.sun.org.apache.xerces.internal.xni.parser.XMLParserConfiguration=com.sun.org.apache.xerces.internal.parsers.XIncludeParserConfiguration</jvm-options>
<jvm-options>-Djavax.xml.transform.TransformerFactory=com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl</jvm-options>
<jvm-options>-Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl</jvm-options>
<jvm-options>-Djavax.xml.soap.MessageFactory=com.sun.xml.messaging.saaj.soap.ver1_1.SOAPMessageFactory1_1Impl</jvm-options>
<jvm-options>-Djavax.xml.soap.SOAPFactory=com.sun.xml.messaging.saaj.soap.ver1_1.SOAPFactory1_1Impl</jvm-options>
<jvm-options>-Djavax.xml.soap.SOAPConnectionFactory=com.sun.xml.messaging.saaj.client.p2p.HttpSOAPConnectionFactory</jvm-options>
-->

i.e. add <!-- before and add --> after these lines.

Also comment out this line:

<!--        <jvm-options>-server</jvm-options>-->

After these changes you can run the Integration Server with Jave SE 6. These are not “official” recommendations (as mentioned, there’s no support for SE 6 just yet); also the lines commented out are optimizations, that need to be re-established for SE 6 tet, so don’t do any performance comparisons just yet.


 

Saturday, December 2, 2006

Moving out of people management

Last week it was four years ago that I started at SeeBeyond. At SeeBeyond, I managed products (the JMS server and J2EE application server in Java CAPS), technology and people. The latter was part of the culture at SeeBeyond: the only way to have influence on any product was to have a team of people reporting to you.

Having responsibility for a team has been interesting. Over the past four years, I've seen people grow. I've seen people "turn around" on whom I was ready to give up. This was very satisfying. With them and through them, I've grown as well. However, in the past year I felt it was time for me to move to the next level. Also, with the increasing number of people in my team (two originally, eight at one point, and six lately), there was less and less time to stay involved with technology at a deep enough level.

When Sun acquired SeeBeyond last year, I was classified as people manager because of the fact that I had people reporting to me. As it turned out, Sun's culture is quite different from SeeBeyond's: there is a dual career ladder with appreciation and growth opportunities for both people managers and individual contributors. But unlike SeeBeyond, people managers primarily manage people and are less involved with technology. 11 people per manager is seen as the norm. It are the technical individual contributors that manage and shape products.

Moving up to the next step on Sun's career ladder, I requested a "diagonal promotion": up one level and from the people management track to the technology track. This week I got my promotion: I'm now a Senior Staff Engineer. Does this mean I'm now a heads-down techie? No, of course not. Sure there will be no more managing reports, but I'll now devote more time influencing people in other teams even outside of the organization. And yes, hopefully there'll be more time to dive a little deeper into a piece of technology.