Friday, August 25, 2006

"When Connection.close() should not close", or the J2EE JCA ManagedConnection life cycle


One of the things I've struggled with most when I was developing the JMSJCA Resource Adapter at SeeBeyond, was the life cycle of the ManagedConnection. I wasted numerous hours going into fruitless directions simply because I did not fully grasp the intricacies of the ManagedConnection life cycle. If you're involved in developing Resource Adapters, you're likely to stumble onto the same problems, so read on!

What do you mean, close() should not close?

Let's take a look at how you might use a JMS Connection in an EJB:

@Resource(name="jms/cf") ConnectionFactory cf;
@Resource(name = "jmx/q1") Queue q;

@TransactionAttribute(TransactionAttributeType.REQUIRED)
public void myBusinessMethod() throws Exception {
Connection c = cf.createConnection();
Session s = c.createSession(false, Session.AUTO_ACKNOWLEDGE);
s.createProducer(q).send(s.createTextMessage("Hello world"));
c.close();
}

Should c.close() really close the connection?

Keep in mind that since J2EE 1.4, JMS providers interface with the application server using JCA. JMS connections should be pooled so that repeated execution of the statement fact.createConnection() is cheap. So the answer is no, the connection should not really be closed.

Should c.close() return the connection to the pool then, so that another EJB could use it? Remember that there is a transaction in progress, so the container should hold on to the connection until the transaction is completed. Only then should the container return the connection to the pool.

Are you appreciating yet the complexities that the application server has to deal with? If not, let's add the following to the method above:

@TransactionAttribute(TransactionAttributeType.REQUIRED)
public void myBusinessMethod() throws Exception {
ConnectionFactory fact = (ConnectionFactory) new InitialContext().lookup("jms/cf");
Connection c = fact.createConnection();
Session s = c.createSession(false, Session.AUTO_ACKNOWLEDGE);
s.createProducer(s.createQueue("q")).send(s.createTextMessage("Hello world"));
c.close();

otherMethod();
}


public void otherMethod() throws Exception {
ConnectionFactory fact = (ConnectionFactory) new InitialContext().lookup("jms/cf");
Connection c = fact.createConnection();
Session s = c.createSession(false, Session.AUTO_ACKNOWLEDGE);
s.createProducer(s.createQueue("q")).send(s.createTextMessage("Goodbye world"));
c.close();
}

When myBusinessMethod() is called, which in turn calls otherMethod(), how many connections are created? Good application servers like the Java CAPS Integration Server, the Sun Java System Application Server, and Glassfish use only one connection in this example. The connection that is used in otherMethod() is the same connection that was created in myBusinessMethod().

Let's look in more detail at what is happening under the covers, and why that is important.

Alphabet soup

Before we begin, let me reiterate some of the terms and abbreviations used. JCA stands for the Java Connector Architecture. It provides the interfacing between applications running in an application server, and external systems such as CRM packages, but also systems such as JMS.

A Resource Adapter is a set of classes that implement the JCA. The central interface for outbound communications is the ManagedConnection. An application, e.g. an EJB, never gets access to a a ManagedConnection directly. Instead, it gets a connection handle. The handle in the above example is the JMS Connection (actually, it is the JMS Session -- but let's not go into that right now). This Connection object is not the JMS Connection that is implemented by the JMS provider, but is a wrapper around such a connection. The wrapper is implemented by the Resource Adapter.

A ManagedConnection holds the physical connection to the external system, e.g. the JMS Connection and Session. Because the physical connection and hence the ManagedConnection is expensive to create, the application server tries to pool the ManagedConnections.

State transitions, and why they are important

We've already seen that a ManagedConnection can be in an idle state or pooled state and it can be in use. It would be very nice if the application server would tell the ManagedConnection when these state transitions happen. But it doesn't. Why would the ManagedConnection care to know when it is being used or when it is being returned to the pool? Let's look at an example in JMS. Let's change the example a little bit:

@TransactionAttribute(TransactionAttributeType.REQUIRED)
public void usingTemp() throws Exception {
ConnectionFactory fact = (ConnectionFactory) new InitialContext().lookup("jms/cf");
Connection c = fact.createConnection();
Session s = c.createSession(false, Session.AUTO_ACKNOWLEDGE);
Queue temp = s.createTemporaryQueue();
Message request = s.createTextMessage("541897-9841");
request.setJMSReplyTo(temp);
sendRequest(request);
Message reply = s.createConsumer(temp).receive();
c.close();
}

The sendRequest() method would somehow start a new transaction and send the request message. The question is when the temporary destination should be deleted. According to the JMS spec, the temporary destination should be invalidated as soon as the connection is closed. However, a transaction is still in progress, so the JMS provider will throw an exception if the temporary destination is deleted when c.close() is called. Can't we just ignore the temporary destination? After all, it will be deleted some time, e.g. when the physical JMS connection is closed. However, since the connection is pooled, it may take a long time before the physical JMS connection is closed. During that time temporary destinations just keep piling up. This approach will exhaust the JMS server.

The temporary destination can be deleted safely when the ManagedConnection is returned to the pool. And that's why it's important to know the state transitions.

How to detect state transitions

What hints does the application server give the ManagedConnection about the state transitions? Let's take a look at the ManagedConnection interface:

public interface ManagedConnection {
Object getConnection(Subject subject, ConnectionRequestInfo connectionRequestInfo) throws ResourceException;
void destroy() throws ResourceException;
void cleanup() throws ResourceException;
void associateConnection(Object object) throws ResourceException;
void addConnectionEventListener(ConnectionEventListener connectionEventListener);
void removeConnectionEventListener(ConnectionEventListener connectionEventListener);
XAResource getXAResource() throws ResourceException;
LocalTransaction getLocalTransaction() throws ResourceException;
ManagedConnectionMetaData getMetaData() throws ResourceException;
void setLogWriter(PrintWriter printWriter) throws ResourceException;
PrintWriter getLogWriter() throws ResourceException;
}

The getConnection() method tells the ManagedConnection to create a new connection handle, so after that method the ManagedConnection knows that it is not in the pooled state.

The destroy() method destroys the ManagedConnection so it signals a state transition to "non-existent".

Doesn't the cleanup() method indicate that a connection is no longer used and will be returned to the pool? It depends on the application server when exactly this method is called. Most application servers will call cleanup() immediately when the application calls Connection.close(). At that moment the connection may still be enlisted in a transaction, and may be reused as we've seen in the examples above.

As it turns out, there are two states that the ManagedConnection needs to keep track of so that it can detect a state transition from "in-use" to "pooled".  The two diagrams keep track of whether the application has access to the ManagedConnection through a connection handle. In other words, it keeps track of whether the ManagedConnection has any outstanding connection handles. See the figure below.

The second state is a transactional state: it keeps track of whether the ManagedConnection is enlisted in a transaction. See the figure below:

Let's look at a few examples that illustrate this concept:

@Resource EJBContext ctx;
@Resource(name="jms/cf") ConnectionFactory cf;
@Resource(name = "jmx/q1") Queue q;

public void ex1() {
ctx.getUserTransaction().begin();
Connection c = cf.createConnection();
Session s = c.createSession(false, Session.AUTO_ACKNOWLEDGE);
s.createProducer(q).send(s.createTextMessage("Hello world"));
c.close();
ctx.getUserTransaction().commit();
}

In this example the getConnection() method is called upon cf.createConnection().createSession(): the ManagedConnection is in use. At the same time the connection is enlisted in the transaction; this is done through the XAResource obtained through ManagedConnection.getXAResource(). When c.close() is called, the ManagedConnection is no longer accessible to the application: all the connection handles are closed. The application server may and probably will call ManagedConnection.cleanup(). However, the connection is still enlisted in the transaction, so the connection is not returned to the pool yet. That happens when getUserTransaction().commit() is called. While the connection is still enlisted, the resource adapter should not try to delete any objects such as temporary destinations in the example mentioned above.

In the following example the enlistment happens a little later. See the inline comments for the state transitions.

public void ex2() {
Connection c = cf.createConnection();
Session s = c.createSession(false, Session.AUTO_ACKNOWLEDGE); // Accessible, but not enlisted
ctx.getUserTransaction().begin(); // Accessible and enlisted
s.createProducer(q).send(s.createTextMessage("Hello world"));
c.close(); // Inaccessible and enlisted
ctx.getUserTransaction().commit(); // Inaccessible and not enlisted
// Return to pool
}

There is also a possible transition from "Inaccessible and enlisted" back to "Accessible and enlisted":

public void ex3() {
Connection c = cf.createConnection();
Session s = c.createSession(false, Session.AUTO_ACKNOWLEDGE); // Accessible, but not enlisted
ctx.getUserTransaction().begin(); // Accessible and enlisted
s.createProducer(q).send(s.createTextMessage("Hello world"));
c.close(); // Inaccessible and enlisted

c = cf.createConnection();
s = c.createSession(false, Session.AUTO_ACKNOWLEDGE); // Accessible and enlisted
s.createProducer(q).send(s.createTextMessage("Hello world"));
c.close(); // Inaccessible and enlisted
ctx.getUserTransaction().commit(); // Inaccessible and not enlisted

// Return to pool
}

The following example shows that there can be multiple connection handles:

public void ex4() {
Connection c = cf.createConnection();
Session s = c.createSession(false, Session.AUTO_ACKNOWLEDGE); // Accessible, but not enlisted
ctx.getUserTransaction().begin(); // Accessible and enlisted
s.createProducer(q).send(s.createTextMessage("Hello world"));

Connection c2 = cf.createConnection();
Session s2 = c.createSession(false, Session.AUTO_ACKNOWLEDGE);// Accessible and enlisted
s2.createProducer(q).send(s2.createTextMessage("Hello world"));
c2.close(); // Accessible and enlisted
ctx.getUserTransaction().commit(); // Accessible and not enlisted
c.close(); // Inaccessible and not enlisted
// Return to pool
}

How to monitor transaction enlistment

To monitor enlistment and the commit() or rollback() method on the XAResource it is possible to write a wrapper around the XAResource of the underlying provider. There are some drawbacks associated with that, and it is better to monitor the transaction through a javax.transaction.Synchronization object. I've described this in more detail in my blog of July 23, 2006: J2EE JCA Resource Adapters: The problem with XAResource wrappers .

Conclusion

By keeping track of both the number of outstanding connection handles given out to the application and the enlistment in a transaction, the ManagedConnection can figure out when it is returned to the pool so that it can undertake necessary actions such as destruction of objects indirectly created by the application.

13 comments:

Ludovic Orban said...

Frank,
I learned something new again with your example about myBusinessMethod() and otherMethod(), thanks !
I just think you should have elaborated a little bit more on how the connection gets reused as it requires close interaction between the connection/session pool and the TM. Anyway, I will manage to implement it and solve the quick pool depletion problem I was running into.
You also forgot to mention that this is not always the case that connections/sessions must be kept out of the queue until after commit: some resources support transaction interleaving thus allowing the connection to be released as soon as close is called.

Ludovic Orban said...

Hi Frank,
I'd be happy to discuss this subject with you in more details. Please drop me an email (if you're interrested) at ludovic.orban _AT_ gmail.com as it's more convenient than posting comments.
Thanks for your feedback !

Michael Bartmann said...

Hi Frank,

you wrote "(actually, it is the JMS Session -- but let's not go into that right now)". This is exactly where some of my problems grasping JMS/JCA lie. As the XASession normaly is the object you get the XARessource from, I'd expect JMS sessions to be the objects in the pool, but all examples go like
createConnection().createSession(..), which implies that the connections are the pooled "physical" objects.

Another thing is that connection.close() is (as of API spec) supposed to close the session, too.
And the contract of JMS Session requires that any outstanding work must be rolled back when closing transacted sessions. Why is JCA allowed to "break" this contract, even if it surely is "the right thing to do" when using pooling?
I am missing some statement in the spec like "not rolling back is allowed (if not required!) in the RA case" or ".. in the JTA/EJB case".

N.B. Your Blog and RA insights in particular are very valuable, if not frustrating :-) , as you solve some problems, which I did not even have before (or realized if have). The only RA experience I have is from implementing an FTP client as outbound RA under JCA 1.0.
Now I am planing to encapsulate our JMS implementation as RA in JCA 1.5 and many new questions arise, as the FTP RA was non-transacted :-)

Frank Kieviet said...

Hi Michael,
Here's what happens:

1 QueueConnectionFactory f = ctx.lookup(...);

2 QueueConnection c = f.createQueueConnection();

3 QueueSession s = c.createQueueSession();



The factory in 1 is an object provided by the RA. The connection in 2 is an
object provided by the RA that acts like a factory object for sessions; it does not represent a physical jms connection/session at this point.



When 3 is executed, the connection object from 2 calls into the connection
manager and gets a session-connection pair from the pool. The connection and
session in this pair are "physical" jms objects. The session object
that is returned to the caller (variable s in 3), is a wrapper provided by the
RA.


Now let's close the connection:

4 c.close()

Recall that the variable c is a wrapper provided by the RA. It will return the
real jms connection/session pair back to the pool and it will mark the session
that the application sees (variable s) as closed, so that all operations that
the application performs on s will result in an exception.



What will happen to outstanding work when the application calls close()? This is where the behavior is different from a standalone jms client from
application code in an appserver. In an appserver the container has control
over the transaction; unless the transaction is marked for rollback, it will
not rollback the messages in JMS, but commit them.



The reason for the difference in behavior is the following: when doing using
JMS in CMT, the connection has got to be closed within the applicaiton code (=
within the transaction) to avoid connection leaks. If that would imply that the
data in JMS would be rolled back, that would mean that you simply could never
use JMS because data would never be committed.



I think you can find this behavior mandated in the EJB spec or J2EE spec
(it's indeed not in the JMS spec).



Implementing a spec-compliant RA for JMS is an expensive undertaking, and I
recommend to reuse open source as much as possible. The JMSJCA implementation
that I have worked on will be open sourced soon; perhaps you could reuse it.
JMSJCA has a number of abstractions that makes it especially easy to deal with
differences in JMS implementations.



Thank you for your feedback on this blog. Plz let me know how I can improve it.



Also, if I can further clarify things, let me know; I'll be happy to lend
a helping hand.

Michael Bartmann said...

Hi Frank,
and thank you very much for your
comprehensive and unbelievably fast
answer.
I must admit that the fact that the
appserver (read: JCA pool) provides you
with objects which do not exactly behave as their api doc promisses still
bugs me. The fact that these objects
are wrappers, which _can_ (technically)
change the behaviour of their delegate, does not excuse that the wrappes do _themselvse_ formally implement "QueueSession".
A similar thing kept me wondering for a while: the jca spec lets the jca container enlist its XARessource to the tm. Where does the application know from, whether the factories bound in jndi are providing "auto-enlisting" ressources, or ressources which have to be enlistes "manually". The behaviour of a ressource should not depend upon the fact that is is a jca/pooled ressource. So the jca spec might regulate the internal contracts between managed ressources and the jca container, but not the relation between application (session bean) and ejb container/jta.
Perhaps I am missing something here, or I do not understand the "whole picture". Or am I too nitpicking about the role of specs and apis in generall?

Thanks again,
Michael

Frank Kieviet said...

Hi Michael,


Thanks for your reply! This looks interesting. Not that I'm trying to sell you anything, but did you consider using an open source JMS server? The Sun JMS server was released into open source recently (JMQ or Open MQ). The JMS server that is bundled with Java CAPS (called STCMS) may also be open sourced at some point; this server has different performance characteristics compared with JMQ / Open MQ. If and when I'm not sure.




The JMSJCA source will likely be released under the CDDL license.


I'm not sure if you're running your code in an application server. In case you're not, here's something else that you may be interested in: I'll likely add a connection manager to JMSJCA so that it can be used outside of an application server while still providing connection pooling, support for transaction managers, and with a solution for closing connections in the middle of a transaction. Similarly there will be some classes that make it possible to leverage the inbound part of the connector outside of an application server.


Frank 


Michael Bartmann said...

Hi Frank,

(this time it's me with a tardy response...)

All this sounds like JMSJCA might fit well to most of our requirements.
Is there some planned schedule or timeline to release your JMSJCA as open source?

Kind regards,
Michael

Frank Kieviet said...

Hi Michael,


JMSJCA is being open sourced as part of JBI; the binaries are already available as part of the JMS binding component. I expect that the source will be there before mid January.  Will that work for you?


Frank




Frank Kieviet said...

Announcement: jmsjca is now available as an open source project on java.net: http://jmsjca.dev.java.net

jenny said...

hi,

I am developing a distributed transactions framework using jboss and spring.I want to access the API's also.so can we have custom commit and rollback in jboss.

Frank Kieviet said...

Re Jenny: sorry, I don't know.

Enrique said...

Hi Frank,

Thank you for your article, it's being really useful. I also checked the presentation at javaone.

I just wanted to make sure: the transitions from enlisted-not enlisted are made by calls to start() and commit()/rollback(), performed by the TM to the XAResource associated with the ManagedConnection?

Thank you

Kind Regards

Frank Kieviet said...

Re Enrique,

Indeed, the transition from enlisted to not-enlisted can be made by listening on the progress of the transaction. This can be done by listening in on the XAResource, but it can also be done by registering a javax.transaction.Synchronization object.

Frank