Frank Kieviet's Engineering Notebook: April 2010

In the past two years I've managed the OpenESB open source project. This project is hosted on java.net. In this post I outline a number of tips and tricks I've learned while managing this project on java.net. This is useful information for my successor, but it also may be useful to other people who are managing projects on java.net.

Web presence

An important factor in the success of an open source project is its web presence: it will be the first point of contact between a potential new user and the project.

New visitors will want to see answers to these four items immediately:

What is it?
License?
Getting started
Downloads

It is important that the links to these pages are static, i.e. that they do not change. That is because a new visitor may enter the site through a link on a different site or through a search engine. For this very reason, it is especially important to keep a single downloads page rather than a page per version, so that users do end up immediately on the latest version, rather than a previous (old) version.

Looking at the front page of OpenESB, these four items are immediately apparent, and the links are permanent. Two more items are important: a visitor will judge a project for it being

Alive
- Active mailing list
- Recent/frequent announcements
Professional
- Well organized, thought through
- Documentation easily found and comprehensive
Inviting
- New users should feel welcome to not just use the product, but also to participate in the community.

You be the judge how well these goals have been met. Let's look at the mechanics of java.net now.

java.net project web site

How java.net processes HTML files

The HTML files for a project's web site are stored in the project's VCS system in a directory called www. There's a big twist here: although the HTML files are ordinary HTML files, i.e. with an <HTML>, <HEAD> and <BODY> tag, the java.net web server reads the HTML files and does a number of substitutions on the file before it writes the resulting file to the client.

Conceptually, there is a java.net template HTML file which has its own <HEAD> element, its own <BODY> element with static content, and some placeholders in these two elements for:

some elements of <HEAD> element of the user defined HTML file, e.g. <TITLE>
the contents of the project_tools.html file
the contents of the <BODY> element from the user defined HTML file

As such, the project owner has to perform special tricks to have the page displayed in other ways than the standard java.net way.

In OpenESB we have several java.net projects. All of them should have the same look and feel, i.e. the OpenESB "brand" and a common navigation. The same goes for the OpenESB wiki.

How OpenESB organizes HTML files

Each project has the same project_tools.html file. This file takes care of the following:

It includes custom CSS files. These files override the java.net styles.
It hides the standard java.net navigation bar. For admins, the navbar display can be turned on: the state is maintained in a cookie. The value in the cookie can be toggled by loading the admin.html file in OpenESB: https://open-esb.dev.java.net/admin.html.
Loads the menu. The menu is defined in a separate JavaScript file.
Displays the search box
Sets an event handler that is invoked when the page's loading is complete. The event handler manipulates the layout of the page and invokes Google Analytics.

As a result, the other HTML files in OpenESB or any of its sub projects don't have to bother with the common look and feel, the menu, etc. All of that is taken care of by project_tools.html, which as mentioned should be duplicated to all the OpenESB projects.

Some or if you will, many, HTML files in OpenESB have another common style element: a right hand side bar that displays common advertising for downloading GlassFish ESB, the wealth of components, and the a partner highlight. This right side bar is taken care of a separate JavaScript file. Files that should display this right hand side bar should explicitly be formatted to do so: it should contain a table in which the right column loads the JavaScript of the right hand side bar.

Previewing HTML

The fact that the look and feel of an HTML file is now defined in project_tools.html makes it difficult to see what a file will actually look like when deployed (i.e. checked in into the project's VCS) on java.net. There is a workaround: I created a directory that when deployed on a web server, emulates the java.net environment. To edit a file with the ability to preview it without checking in the file, follow this procedure:

Load the file in your browser through java.net's web server, e.g. https://open-esb.dev.java.net/Downloads.html.
Save this file to the emulation directory, e.g. Tomcat/webapps/ROOT/Downloads.html
Now load the file into your browser through your local web server, e.g. http://localhost:9080/Downloads.html. This should look identical to what was loaded from java.net.
Locate the text in the file to be edited. Change the text and save the file. Reload the file in your browser. Repeat this step until the file looks OK.
Do a diff using a tool like Araxis Merge, or Beyond Compare between the local file and the file that was checked in into VCS, e.g. diff Tomcat/webapps/ROOT/Downloads.html and open-esb/www/Downloads.html.
Merge the changes into the VCS file, i.e. open-esb/www/Downloads.html. Save and check in into VCS.

Here is the directory for Tomcat. If you use it for projects other than OpenESB, you can remove many of the files. Make sure to leave branding and css directories in tact. If you intend to reuse project_tools.html, don't forget to change the Google Analytics account ID.

SSL

Unfortunately, java.net uses SSL for all its files. This is unfortunate for two reasons:

Files do not get cached in the browser: every time a user opens his browser and goes to OpenESB, all the images, style sheets, etc are reloaded again.
To avoid security warnings in Internet Explorer, files that are included in an HTML file also should come through SSL. This rule is violated on the OpenESB main page (of all places!) where it loads a dynamic list of news item from a non-https server.

There's no workaround for this. Collabnet promised a long time ago that SSL would be removed but this still has not happened.

Changing the menu

The menu is defined in a json like format in a file called menu.js. This file is duplicated in the wiki. See below. Make sure to use absolute URLs because the file is used from different projects, i.e. different roots.

Wiki (not on java.net)

OpenESB is not using the java.net wiki infrastructure. I don't recall why this decision was made. New projects should probably evaluate the java.net wiki infrastructure before considering setting up a wiki elsewhere.

The OpenESB wiki is hosted on a machine hosted at Sun/Oracle, not at Collab net. It is collocated with other wikis, e.g. GlassFish, UpdateCenter, etc. The wiki engine is JSP wiki. The templates have been adapted to look and feel like the java.net OpenESB web site.

Since java.net has / used to have a lot of downtime, and to avoid having this downtime impact the wiki, the menu.js file is duplicated on the wiki. That java.net uses ssl is another reason to duplicate the file. For the same reason, the icons displayed in the menu are also duplicated on the wiki server.

Users are managed separately from java.net. User management / restrictions are necessary to avoid spam. Spam has been a been big problem on the OpenESB wiki and the other wikis collocated with it. Cleaning up spam can be done by directly manipulating the text files that make up the wiki page.

JSPWiki security is configured to use JAAS. This fact is important if the wiki needs to be setup on a different GlassFish server. In that case make sure to update the server.policy file of GlassFish. The policy file specifies that only "validated" users can edit pages. Users are stored in a user database and a group database. The latter specifies group membership, of which the "validated" group is important. Both files are in the etc directory of the wiki directory.

To add a user, the group database needs to be updated. This can be done through the web interface, http://wiki.open-esb.java.net/EditGroup.jsp?group=Validated or can be done by editing the group file directly.

The raw data files and user management files of the wiki can be accessed through SSH.

Since the user management of the wiki is separate from java.net, users can choose a different userid on the wiki than they do on java.net. Allowing this is a mistake in retrospect: editing the wiki requires a SCA (Sun Contributor Agreement) or employment in Sun/Oracle, and having to manage/audit two sets of users instead of one is a waste of time. Again, in retrospect.

java.net user management

Anybody on java.net can request developer privileges. When that happens, an email is sent by java.net to the project owner. An automated process is setup that replies automatically to the user list asking the applicant for his motivation, background, etc. This reply is specified in role-approval.policy in the www directory.

For some reason, many people ask for developer privileges. They are not known, have not contributed or communicated before, and are never heard from again.

Consequently, this automated process sounds nice, but is useless in practice. A better process is that someone first communicates on the mailing list, proves he/she is serious and able to contribute, after which the project owner grants privileges. Before privileges are granted, a signed SCA needs to be on file. This process is documented on the OpenESB web site.

The link to the membership management page is displayed in the java.net nav bar. Recall that the nav bar is hidden. To toggle display of the nav bar, load https://open-esb.dev.java.net/admin.html. Alternatively, jump to the page directly: https://open-esb.dev.java.net/servlets/ProjectMemberList.

When a user is granted commit privileges to a project, he/she also automatically gets commit privileges to the code repositories of the sub projects. E.g. granting commit privileges to open-esb also gives commit privileges to open-jbi-components, a sub project of open-esb. This is only the case if the VCS of the child project is of the same type as the parent project. Since open-jbi-components and open-esb both use CVS, open-esb committers have access to open-jbi-components. However, if the VCS system is different, e.g. openesb-builds uses SVN, separate access to this project needs to be granted next to open-esb.

java.net email management

Email lists are managed by an email list manager running in java.net. This email list manager is only partially integrated in java.net. Users can click on a button on the java.net site to subscribe to an email list. What effectively happens is that it will register the email address that is associated with that user at that moment in the list server. Similarly, a user can unsubscribe through a button, which will remove the email list associated with that user at that moment. Where this knowledge comes in handy? There is confusion if the user changes his email address associated with his java.net id between subscribing and unsubscribing. Sometimes users get so confused that they'll need help unsubscribing from the list. After all the last thing you want is repeated posts of "how do I unsubscribe from this list?".

Here is another task with respect to email: spam. Protecting the list to spam is very important. Therefore lists should be setup as "discuss" or "moderated". A setting of "discuss" means that only "subscribed" or "allowed" posters can post. If someone posts a message who is not in the "subscribed" or "allowed" list, the message will be forwarded to the list owner(s) for review. Replying to that message will allow the message to be posted. To avoid getting too many of these "review messages", frequent posters can be added to the "allowed" list. The "allowed" list is accessible from the email management page.

The link to the email list management page is displayed in the java.net nav bar. Recall that the nav bar is hidden. To toggle display of the nav bar, load https://open-esb.dev.java.net/admin.html. Alternatively, jump to the page directly: https://open-esb.dev.java.net/servlets/ProjectMailingListList.

Nabble is a view on the mailing list. OpenESB has integrated it on the web site. Note that posters are identified by their "from" email address only, but posters on Nabble somehow do not need to be in the "allowed" of "subscribed" list. This is fortunate, but puzzling from a technical point of view.

This brings me to one more email list related tasks: once in a while, someone posts an email that he/she wants to revoke. For example, someone could accidentally send a private email to the list, or include confidential (customer information for example) information in an email.

Of course there's no such thing as undoing the sending of an email. Subscribed users will have received the email by the time that the sender notices the mistake. However, there is an option to remove the email from Nabble and from the java.net mail browser. Note that there are two places where the email needs to be removed: both Nabble and java.net. Removing the mail there at least makes the mistake less visible and will make it harder for the mistaken email to be picked up by search engines.

java.net automation

Some of the management tasks on java.net can be automated. Kohsuke has developed an extensive set of tools that provide a Java api to java.net web interface: https://javanettasks.dev.java.net/.

I've written a few tools to make community management easier. One such tool is used to make an inventory of all the users in the community with commit privileges. Everybody in this list should either be a Sun/Oracle employee, or should have signed an SCA. The tool loads and parses the list of signatories (from https://sca.dev.java.net/CA_signatories.htm) and compares this list with the one extracted from java.net. Discrepancies are people who are currently or have been Sun/Oracle employees. Since it is difficult to track who is no longer an employee (especially with the layoffs that used to happen every now and then), the tool can email all "discrepancies" and ask them to either submit an SCA or to verify employment by clicking a link pointing to an internal server. These tools are available on https://open-esb-build.dev.java.net/svn/open-esb-build/trunk/community-mgt/.

It's been a few months since Oracle acquired Sun. At the time there were a lot of questions about what Oracle would do with OpenESB. On Feb 15, I posted a plan to the users mailing list. What was the plan? And how is the plan coming together? How is OpenESB doing a few months after the acquisition? What will the future hold for OpenESB?

First, let's try to understand Oracle's perspective. Oracle already has an integration product: the SOA Suite. Through the BEA acquisition it got another one, and through the Sun acquisition it got yet two more: CAPS and OpenESB. Of course there's no sense in keeping high levels of investments in all these products; it makes much more sense to focus on one product. What is this strategic product? Oracle is very clear about that: Oracle's strategic SOA middleware platform is the Oracle SOA Suite. Consequently, Oracle has reduced the level of investment in OpenESB.

That was the bad news. Now for the good news. Oracle could have just pulled out from OpenESB and let it fall into a black hole. Or worse, it could have taken the site down, removed the downloads, or put other obstacles in place. It did not. It did quite the contrary.

Although Sun was the main sponsor behind OpenESB, Oracle recognized that a lot of people made investments into OpenESB, either in the community or into their own company by using OpenESB, and with a feeling of responsibility and fairness, Oracle decided to help the community to find a way to stand on its own, so that all these investments would bear fruit for as long a time as possible.

Did you invest in OpenESB? Let's look at what will change for you depending on the kind of investment you made.

What will change for you

What will change for you if you bought GlassFish ESB support? Nothing will change: you are still fully supported. The same Sun SOA Support department is still available to help on issues. Should patches be created to address issues, the full Sun SOA Sustaining department is still there to create patches.

Are you using GlassFish ESB but did not buy support? Unlike Sun, Oracle is no longer trying to sell GlassFish ESB licenses to new customers. Instead, you may rely on community support, or commercial support provided by one of the OpenESB community partners. The user mailing list is nowadays less frequented by Sun/Oracle engineers, but other community members have stepped up and are keeping the mailing list responsive. More about that below.

Are you investing in OpenESB by contributing code or other artifacts, or are considering to do so? You will find that it has become easier to contribute. I've put together a new governance document that gives greater freedom to community contributors. Oracle can still exert influence, but that is intended to keep the peace in the community should that be necessary. Overall, you will find it easier to propose and implement new changes, to commit code, and last but not least, to influence the roadmap. This brings us to the future of OpenESB.

The future of OpenESB

The future of OpenESB revolves around OpenESB becoming an open source community that can stand on its own, i.e. without Sun or Oracle as the single major sponsor. That transformation will of course not happen overnight, and Oracle is committed to help with this transformation. For instance, Oracle will do periodic builds and post these on the downloads site. Another commitment is that Oracle will merge patches that it makes for customers, back into the open source repository.

What did Oracle do so far? Next to the governance document I already mentioned, the sources of GlassFish ESB v2.2 can now be found in the open source repository. The HL7 BC and the WLM SE are also back in the open source repository, and the binaries can be downloaded from the downloads page. The process of an automatic periodic build has incurred some delays due to technical issues, but is well on its way.

OpenESB today

How is OpenESB doing today? Let's look at the users mailing list. From the Users mailing list on Markmail, it can be seen that the list activity has declined a bit, but is not much below the activity of that of a year ago.

What we also can learn from Markmail is that community members who are not on the Oracle payroll are stepping up. For example, here are the most active posters for March:

Another metric that we can look at is the number of users. In GlassFish ESB v2.1 we introduced a feature in which NetBeans checks for updates upon startup. By looking at the traffic to the updates-server, we can estimate how many users there are of GlassFish ESB. I defined the number of users as the number of NetBeans installations that ping the server at least three times in a time window greater than five days. As can be seen, the number of users is going up.

In terms of a roadmap there are no concrete proposals from the community yet, but several members have expressed interest in continuing with Fuji.

More good news: a few new committers have joined the project and are contributing code.

Frank Kieviet's Engineering Notebook

Wednesday, April 28, 2010

Google TV Ads

Monday, April 26, 2010

Working with java.net: tips and tricks for project owners

Web presence

java.net project web site

How java.net processes HTML files

How OpenESB organizes HTML files

Previewing HTML

SSL

Changing the menu

Wiki (not on java.net)

java.net user management

java.net email management

java.net automation

Wednesday, April 21, 2010

OpenESB under Oracle

What will change for you

The future of OpenESB

OpenESB today

Moving on… next stop: Google

About Me

Links

Most popular...

Followers

Search This Blog

Blog Archive