SeeBeyond was acquired by Sun a year ago. What changed?

It's been one year now since Sun Microsystems acquired SeeBeyond. What did it mean for SeeBeyond employees like me? What did and will it mean for customers?

What didn't change. First of all, Sun didn't come in and turn the place upside down. Instead it left it pretty much untouched. There were no major reorganizations. Nobody got fired. We didn't change the way we develop software. We didn't change the plans of the products that we were working on.

Culture shock. The old-SeeBeyond was a company of secrecy and need-to-know-only. But the Sun culture is one of openness and transparency. For the first time ever, employees at all levels now had some insight in plans and directions. We could find out what other groups within Sun are doing. We were invited to participate and to share our plans. Even our openness to the outside world changed.  For example, this blog would have been unthinkable a little bit over a year ago.

Integrating products and what that means to customers: there's some overlap between SeeBeyond's product offering and Sun's. We're trying to integrate both offerings as much as possible. That means that in some cases we'll invest less in products that have a better counterpart in Sun. It surely makes the release of a product a lot more complicated: we now need to make sure that all the parts that we depend on and are produced by other groups within Sun are all ready at the same time and work together properly. But for customers it means a better product offering. And it also means a wider product offering because customers now get easier access to products that SeeBeyond didn't offer. Big wins for customers.

Information overload. The interdependencies with other groups within Sun requires us to keep track of many developments. What are the release plans of the Glassfish team? What is the road map of the Message Queue? What is the Tools group up to? What groups are working on NetBeans? And so on. Conference calls several times a week. Wikis and internal sites by the hundreds. At times I get a distinct feeling of information overload and wish I could ignore everything.

Opportunities for SeeBeyond employees.  SeeBeyond has definitely become a more interesting place to work since it became part of Sun. Also a place with more opportunities: smart people, cool products and a good environment means more opportunities. Last week I talked with a long-time Sun employee and he mentioned career paths within Sun. "Career path" is a word I had not heard for many years.

Changes to come and what it means to customers: Sun's new approach to software is that of open source and radically different revenue models. The old SeeBeyond had a revenue model based on license fees, and a sales model in which the first contact with the customer was through an RFP. That will change. Software will be downloadable by anybody and can be used by anybody free of charge. That should draw developers to try out our software. The first contact with customers will be right there. Through more open communication with the end-user, we'll be able to build products that better meet customers' requirements. Since the developers we are targeting have the freedom to choose, we'll also be forced to change and improve our products quite a bit compared with previous versions.

"When Connection.close() should not close", or the J2EE JCA ManagedConnection life cycle

One of the things I've struggled with most when I was developing the JMSJCA Resource Adapter at SeeBeyond, was the life cycle of the ManagedConnection. I wasted numerous hours going into fruitless directions simply because I did not fully grasp the intricacies of the ManagedConnection life cycle. If you're involved in developing Resource Adapters, you're likely to stumble onto the same problems, so read on!

What do you mean, close() should not close?

Let's take a look at how you might use a JMS Connection in an EJB:

@Resource(name="jms/cf") ConnectionFactory cf;
@Resource(name = "jmx/q1") Queue q;

@TransactionAttribute(TransactionAttributeType.REQUIRED)
public void myBusinessMethod() throws Exception {
 Connection c = cf.createConnection();
 Session s = c.createSession(false, Session.AUTO_ACKNOWLEDGE);
 s.createProducer(q).send(s.createTextMessage("Hello world"));
 c.close();
}

Should c.close() really close the connection?

Keep in mind that since J2EE 1.4, JMS providers interface with the application server using JCA. JMS connections should be pooled so that repeated execution of the statement fact.createConnection() is cheap. So the answer is no, the connection should not really be closed.

Should c.close() return the connection to the pool then, so that another EJB could use it? Remember that there is a transaction in progress, so the container should hold on to the connection until the transaction is completed. Only then should the container return the connection to the pool.

Are you appreciating yet the complexities that the application server has to deal with? If not, let's add the following to the method above:

@TransactionAttribute(TransactionAttributeType.REQUIRED)
public void myBusinessMethod() throws Exception {
 ConnectionFactory fact = (ConnectionFactory) new InitialContext().lookup("jms/cf");
 Connection c = fact.createConnection();
 Session s = c.createSession(false, Session.AUTO_ACKNOWLEDGE);
 s.createProducer(s.createQueue("q")).send(s.createTextMessage("Hello world"));
 c.close();

 otherMethod();
}


public void otherMethod() throws Exception {
 ConnectionFactory fact = (ConnectionFactory) new InitialContext().lookup("jms/cf");
 Connection c = fact.createConnection();
 Session s = c.createSession(false, Session.AUTO_ACKNOWLEDGE);
 s.createProducer(s.createQueue("q")).send(s.createTextMessage("Goodbye world"));
 c.close();
}

When myBusinessMethod() is called, which in turn calls otherMethod(), how many connections are created? Good application servers like the Java CAPS Integration Server, the Sun Java System Application Server, and Glassfish use only one connection in this example. The connection that is used in otherMethod() is the same connection that was created in myBusinessMethod().

Let's look in more detail at what is happening under the covers, and why that is important.

Alphabet soup

Before we begin, let me reiterate some of the terms and abbreviations used. JCA stands for the Java Connector Architecture. It provides the interfacing between applications running in an application server, and external systems such as CRM packages, but also systems such as JMS.

A Resource Adapter is a set of classes that implement the JCA. The central interface for outbound communications is the ManagedConnection. An application, e.g. an EJB, never gets access to a a ManagedConnection directly. Instead, it gets a connection handle. The handle in the above example is the JMS Connection (actually, it is the JMS Session -- but let's not go into that right now). This Connection object is not the JMS Connection that is implemented by the JMS provider, but is a wrapper around such a connection. The wrapper is implemented by the Resource Adapter.

A ManagedConnection holds the physical connection to the external system, e.g. the JMS Connection and Session. Because the physical connection and hence the ManagedConnection is expensive to create, the application server tries to pool the ManagedConnections.

State transitions, and why they are important

We've already seen that a ManagedConnection can be in an idle state or pooled state and it can be in use. It would be very nice if the application server would tell the ManagedConnection when these state transitions happen. But it doesn't. Why would the ManagedConnection care to know when it is being used or when it is being returned to the pool? Let's look at an example in JMS. Let's change the example a little bit:

@TransactionAttribute(TransactionAttributeType.REQUIRED)
public void usingTemp() throws Exception {
 ConnectionFactory fact = (ConnectionFactory) new InitialContext().lookup("jms/cf");
 Connection c = fact.createConnection();
 Session s = c.createSession(false, Session.AUTO_ACKNOWLEDGE);
 Queue temp = s.createTemporaryQueue();
 Message request = s.createTextMessage("541897-9841");
 request.setJMSReplyTo(temp);
 sendRequest(request);
 Message reply = s.createConsumer(temp).receive();
 c.close();
} 

The sendRequest() method would somehow start a new transaction and send the request message. The question is when the temporary destination should be deleted. According to the JMS spec, the temporary destination should be invalidated as soon as the connection is closed. However, a transaction is still in progress, so the JMS provider will throw an exception if the temporary destination is deleted when c.close() is called. Can't we just ignore the temporary destination? After all, it will be deleted some time, e.g. when the physical JMS connection is closed. However, since the connection is pooled, it may take a long time before the physical JMS connection is closed. During that time temporary destinations just keep piling up. This approach will exhaust the JMS server.

The temporary destination can be deleted safely when the ManagedConnection is returned to the pool. And that's why it's important to know the state transitions.

How to detect state transitions

What hints does the application server give the ManagedConnection about the state transitions? Let's take a look at the ManagedConnection interface:

public interface ManagedConnection {
 Object getConnection(Subject subject, ConnectionRequestInfo connectionRequestInfo) throws ResourceException;
 void destroy() throws ResourceException;
 void cleanup() throws ResourceException;
 void associateConnection(Object object) throws ResourceException;
 void addConnectionEventListener(ConnectionEventListener connectionEventListener);
 void removeConnectionEventListener(ConnectionEventListener connectionEventListener);
 XAResource getXAResource() throws ResourceException;
 LocalTransaction getLocalTransaction() throws ResourceException;
 ManagedConnectionMetaData getMetaData() throws ResourceException;
 void setLogWriter(PrintWriter printWriter) throws ResourceException;
 PrintWriter getLogWriter() throws ResourceException;
}

The getConnection() method tells the ManagedConnection to create a new connection handle, so after that method the ManagedConnection knows that it is not in the pooled state.

The destroy() method destroys the ManagedConnection so it signals a state transition to "non-existent".

Doesn't the cleanup() method indicate that a connection is no longer used and will be returned to the pool? It depends on the application server when exactly this method is called. Most application servers will call cleanup() immediately when the application calls Connection.close(). At that moment the connection may still be enlisted in a transaction, and may be reused as we've seen in the examples above.

As it turns out, there are two states that the ManagedConnection needs to keep track of so that it can detect a state transition from "in-use" to "pooled".  The two diagrams keep track of whether the application has access to the ManagedConnection through a connection handle. In other words, it keeps track of whether the ManagedConnection has any outstanding connection handles. See the figure below.

The second state is a transactional state: it keeps track of whether the ManagedConnection is enlisted in a transaction. See the figure below:

Let's look at a few examples that illustrate this concept:

@Resource EJBContext ctx;
@Resource(name="jms/cf") ConnectionFactory cf;
@Resource(name = "jmx/q1") Queue q;

public void ex1() {
 ctx.getUserTransaction().begin();
 Connection c = cf.createConnection();
 Session s = c.createSession(false, Session.AUTO_ACKNOWLEDGE);
 s.createProducer(q).send(s.createTextMessage("Hello world"));
 c.close();
 ctx.getUserTransaction().commit();
}

In this example the getConnection() method is called upon cf.createConnection().createSession(): the ManagedConnection is in use. At the same time the connection is enlisted in the transaction; this is done through the XAResource obtained through ManagedConnection.getXAResource(). When c.close() is called, the ManagedConnection is no longer accessible to the application: all the connection handles are closed. The application server may and probably will call ManagedConnection.cleanup(). However, the connection is still enlisted in the transaction, so the connection is not returned to the pool yet. That happens when getUserTransaction().commit() is called. While the connection is still enlisted, the resource adapter should not try to delete any objects such as temporary destinations in the example mentioned above.

In the following example the enlistment happens a little later. See the inline comments for the state transitions.

public void ex2() {
 Connection c = cf.createConnection();
 Session s = c.createSession(false, Session.AUTO_ACKNOWLEDGE); // Accessible, but not enlisted
 ctx.getUserTransaction().begin(); // Accessible and enlisted
 s.createProducer(q).send(s.createTextMessage("Hello world"));
 c.close(); // Inaccessible and enlisted
 ctx.getUserTransaction().commit(); // Inaccessible and not enlisted
 // Return to pool
}

There is also a possible transition from "Inaccessible and enlisted" back to "Accessible and enlisted":

public void ex3() {
 Connection c = cf.createConnection();
 Session s = c.createSession(false, Session.AUTO_ACKNOWLEDGE); // Accessible, but not enlisted
 ctx.getUserTransaction().begin(); // Accessible and enlisted
 s.createProducer(q).send(s.createTextMessage("Hello world"));
 c.close(); // Inaccessible and enlisted
 
 c = cf.createConnection();
 s = c.createSession(false, Session.AUTO_ACKNOWLEDGE); // Accessible and enlisted
 s.createProducer(q).send(s.createTextMessage("Hello world"));
 c.close(); // Inaccessible and enlisted
 ctx.getUserTransaction().commit(); // Inaccessible and not enlisted

 // Return to pool
}

The following example shows that there can be multiple connection handles:

public void ex4() {
 Connection c = cf.createConnection();
 Session s = c.createSession(false, Session.AUTO_ACKNOWLEDGE); // Accessible, but not enlisted
 ctx.getUserTransaction().begin(); // Accessible and enlisted
 s.createProducer(q).send(s.createTextMessage("Hello world"));
 
 Connection c2 = cf.createConnection();
 Session s2 = c.createSession(false, Session.AUTO_ACKNOWLEDGE);// Accessible and enlisted
 s2.createProducer(q).send(s2.createTextMessage("Hello world"));
 c2.close(); // Accessible and enlisted
 ctx.getUserTransaction().commit(); // Accessible and not enlisted
 c.close(); // Inaccessible and not enlisted
 // Return to pool
}

How to monitor transaction enlistment

To monitor enlistment and the commit() or rollback() method on the XAResource it is possible to write a wrapper around the XAResource of the underlying provider. There are some drawbacks associated with that, and it is better to monitor the transaction through a javax.transaction.Synchronization object. I've described this in more detail in my blog of July 23, 2006: J2EE JCA Resource Adapters: The problem with XAResource wrappers .

Conclusion

By keeping track of both the number of outstanding connection handles given out to the application and the enlistment in a transaction, the ManagedConnection can figure out when it is returned to the pool so that it can undertake necessary actions such as destruction of objects indirectly created by the application.

JMS request/reply from an EJB

Sometimes something that appears trivial turns out to be a lot more involved. Doing JMS request/reply from an EJB is such a case. Let's take a look...

Request/reply

JMS can be used as a form of inter process communication. JMS is meant for asynchronous calls, but it can also used for synchronous calls. In that case, the first process sends a message to a queue or topic, and waits until another process has replied to the message. The reply is sent to a different queue or topic. Advantages of using JMS for interprocess communication is that the two processes are completely decoupled. The two processes only need to agree on what is in the message.

Synchronous communication in JMS is also called request/reply.

The JMS api provides two helper classes to make request/reply easier. They are the QueueRequestor and TopicRequestor classes. Here is an example of how they can be used:

QueueConnection c = (QueueConnectionFactory) (new
InitialContext().lookup("qcf"));
QueueSession s = c.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
Queue q = (Queue) (new InitialContext().lookup("requestqueue"));
QueueRequestor r = new QueueRequestor(s, q);
Message reply = r.request(s.createTextMessage("1233442342"));
c.close();

Looks simple, right? However, when this code is used in a typical EJB, it will likely not work. Why not? Is there something wrong with the QueeuRequestor? Well, yes, yet that is not the problem. Let's look at the implementation of the QueueRequestor:

public class QueueRequestor {
 QueueSession session; 
 Queue queue; 
 TemporaryQueue tempQueue;
 QueueSender sender;
 QueueReceiver receiver;

public QueueRequestor(QueueSession session, Queue queue) throws JMSException {
  this.session = session;
  this.queue = queue;
  tempQueue = session.createTemporaryQueue();
  sender = session.createSender(queue);
  receiver = session.createReceiver(tempQueue);
}

public Message request(Message message) throws JMSException {
  message.setJMSReplyTo(tempQueue);
  sender.send(message);
  return receiver.receive();
}

public void close() throws JMSException {
  session.close();
  tempQueue.delete();
}
}

So we can rewrite the above code sample as follows:

QueueConnection c = (QueueConnectionFactory) (new InitialContext().lookup("qcf"));
QueueSession s = c.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
Queue q = (Queue) (new
InitialContext().lookup("requestqueue"));
Message request = s.createTextMessage("1233442342");
request.setJMSReplyTo(s.createTemporaryQueue();
s.createSender(q).send(m);
Message reply = s.createReceiver(q).receive();
c.close();

There's nothing obviously wrong with that, so why does this typically not work when this code is in an EJB?

Transactions

Before we look at the explanation of why the above code doesn't work in an EJB, let's review some basics about application servers and JMS servers. A JMS server is typically a stand alone process. Applications can communicate with this server using a client runtime. The client runtime implements the JMS API as defined in the JMS specification. If you're building a stand-alone application, you use the client runtime directly. Example, when you're using the JMS server shipped with Java CAPS you would be using com.stc.stcjms.jar to talk to the server process (stcms.exe). The client runtime implemented in com.stc.jms.stcjms.jar implements the JMS api and communicates with the server (stcms.exe) over tcp/ip sockets.

However, when your code lives in an EJB, you're not using the JMS client runtime directly. For example, when you obtain a connection factory from JNDI in an EJB, you will not get the connection factory object that belongs to the JMS provider itself, but you get a factory that is implemented by a resource adapter. As of J2EE 1.4, the interfacing between the application server and JMS is done through a Resource Adapter. In the example for Java CAPS, you would obtain a connection factory implemented by the JMSJCA resource adapter instead of the com.stc.jms.client.STCQueueConnectionFactory that belongs to the STCMS client runtime.

The Resource Adapter acts like a wrapper around the objects that are implemented in the JMS client runtime. One of the responsibilities of the Resource Adapter is to make sure that JMS acts like a transactional resource. This means that JMS connection (technically, the JMS session) is enlisted with the transaction manager of the applicaiton server. In fact, under the covers the Resource Adapter really uses an XASession rather than an AUTO_ACKNOWLEDGE session. What effect did it have to specify the arguments to

c.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);

In fact, these arguments are ignored, so they have no effect whatsoever.

Use of JMS in a transaction

Let's say that we have a bean managed EJB (BMT) with the following code:

EJBContext ctx = ...;
ctx.getUserTransaction().begin();
QueueConnection c = (QueueConnectionFactory) (new
InitialContext().lookup("qcf"));
QueueSession s = c.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
Queue q = (Queue) (new InitialContext().lookup("requestqueue"));
Message request = s.createTextMessage("1233442342");
request.setJMSReplyTo(s.createTemporaryQueue();
s.createSender(q).send(m);
Message reply = s.createReceiver(q).receive();
c.close();
ctx.getUserTransaction().commit();

The message will be sent to the queue only when the statement getUserTransaction().commit() is executed. Before this statement, the message will not be visible anywhere.  So when the receive() method is executed, the message was really not sent to the queue. The other process never gets the request, and hence, a reply will never be received. Since no timeout is specified, the application simply hangs.

Now that we know that, the solution is pretty simple: just commit the transaction before receiving the reply.

ctx.getUserTransaction().begin();
QueueConnection c = (QueueConnectionFactory) (new InitialContext().lookup("qcf"));
QueueSession s = c.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
Queue q = (Queue) (new InitialContext().lookup("requestqueue"));
Message request = s.createTextMessage("1233442342");
request.setJMSReplyTo(s.createTemporaryQueue();
s.createSender(q).send(m);
ctx.getUserTransaction().commit();

ctx.getUserTransaction().begin();
Message reply = s.createReceiver(q).receive();
c.close();
ctx.getUserTransaction().commit();

It depends on the business logic if you can simply split up the transaction in two. Perhaps something else happens in the transaction, for example you could update a database in the same transaction. In that case, you may not want to commit the transaction yet! How can we deal with that situation?

The solution would be to suspend the transaction, create a new transaction, send the request, commit the new transaction, unsuspend the first transaction, and continue. Sounds complicated? It is, but it can be simplified using an EJB: simply add a new business method to the EJB, make sure that the method's trans-attribute is set to NotSupported.
Next, instantiate a new EJB and call that method. By calling that method, the transaction will automatically be suspended.

Note that it is crucial to instantiate a new EJB: if you just call the new business method from the same EJB instance, the application server will not intercept the method and will not suspend the transaction.

When the EJB is CMT

Pretty much the same goes if the EJB is not BMT but CMT. If a transaction was started, the same deadlock occurs: the send() is never committed until the transaction is committed, so the receive() method will never receive a reply. In that case, add a new business method and make sure that the trans-attribute is set to RequiresNew. The application server will then suspend the main transaction and start a new transaction automatically. As with BMT, make sure that you call the new business method on a new instance of the EJB rather than on the same instance.

When there is no transaction

What happens if there is no transaction when you call send()? Well, that depends on the implementation. The specification is not clear on what should happen when an XA resource is used outside of a transaction. The JMS CTS implies that the behavior should be that of auto-commit (or AUTO_ACKNOWLEDGE in JMS parlance). CTS stands for Compliance Test Suite. The CTS for JMS consists of a few thousand tests. Every implementation that claims that  be compliant with the JMS spec must pass these tests. Hence, most implementations, including the one that ships with Java CAPS will exhibit AUTO_ACKNOWLEDGE behavior. In that case, the above code will work and the QueueRequestor can be used.

What about the QueueRequestor?

Can't we use the QueueRequestor with transactions? The simple answer is no. Why did the Resource Adapter not take care of this? The problem is that the QueueRequestor is implemented by the JMS api, and not by the JMS provider or Resource Adapter. Perhaps this is another example why it is generally a bad idea to provide implementation classes in API-s?

There's another problem with the QueueRequestor class: there is no way to specify a timeout value. You would probably prefer a timeout exception rather than having a hanging application.

Conclusion

Knowing what happens under the covers explains why request/reply cannot work in a single transaction. Simply post the request in a separate transaction.