In the past two years I've managed the OpenESB open source project. This project is hosted on java.net. In this post I outline a number of tips and tricks I've learned while managing this project on java.net. This is useful information for my successor, but it also may be useful to other people who are managing projects on java.net.
Web presence
An important factor in the success of an open source project is its web presence: it will be the first point of contact between a potential new user and the project.
New visitors will want to see answers to these four items immediately:
- What is it?
- License?
- Getting started
- Downloads
It is important that the links to these pages are static, i.e. that they do not change. That is because a new visitor may enter the site through a link on a different site or through a search engine. For this very reason, it is especially important to keep a single downloads page rather than a page per version, so that users do end up immediately on the latest version, rather than a previous (old) version.
Looking at the front page of OpenESB, these four items are immediately apparent, and the links are permanent. Two more items are important: a visitor will judge a project for it being
- Alive
- Active mailing list
- Recent/frequent announcements
- Professional
- Well organized, thought through
- Documentation easily found and comprehensive
- Inviting
- New users should feel welcome to not just use the product, but also to participate in the community.
You be the judge how well these goals have been met. Let's look at the mechanics of java.net now.
java.net project web site
How java.net processes HTML files
The HTML files for a project's web site are stored in the project's VCS system in a directory called www. There's a big twist here: although the HTML files are ordinary HTML files, i.e. with an <HTML>, <HEAD> and <BODY> tag, the java.net web server reads the HTML files and does a number of substitutions on the file before it writes the resulting file to the client.
Conceptually, there is a java.net template HTML file which has its own <HEAD> element, its own <BODY> element with static content, and some placeholders in these two elements for:
- some elements of <HEAD> element of the user defined HTML file, e.g. <TITLE>
- the contents of the project_tools.html file
- the contents of the <BODY> element from the user defined HTML file
As such, the project owner has to perform special tricks to have the page displayed in other ways than the standard java.net way.
In OpenESB we have several java.net projects. All of them should have the same look and feel, i.e. the OpenESB "brand" and a common navigation. The same goes for the OpenESB wiki.
How OpenESB organizes HTML files
Each project has the same project_tools.html file. This file takes care of the following:
- It includes custom CSS files. These files override the java.net styles.
- It hides the standard java.net navigation bar. For admins, the navbar display can be turned on: the state is maintained in a cookie. The value in the cookie can be toggled by loading the admin.html file in OpenESB: https://open-esb.dev.java.net/admin.html.
- Loads the menu. The menu is defined in a separate JavaScript file.
- Displays the search box
- Sets an event handler that is invoked when the page's loading is complete. The event handler manipulates the layout of the page and invokes Google Analytics.
As a result, the other HTML files in OpenESB or any of its sub projects don't have to bother with the common look and feel, the menu, etc. All of that is taken care of by project_tools.html, which as mentioned should be duplicated to all the OpenESB projects.
Some or if you will, many, HTML files in OpenESB have another common style element: a right hand side bar that displays common advertising for downloading GlassFish ESB, the wealth of components, and the a partner highlight. This right side bar is taken care of a separate JavaScript file. Files that should display this right hand side bar should explicitly be formatted to do so: it should contain a table in which the right column loads the JavaScript of the right hand side bar.
Previewing HTML
The fact that the look and feel of an HTML file is now defined in project_tools.html makes it difficult to see what a file will actually look like when deployed (i.e. checked in into the project's VCS) on java.net. There is a workaround: I created a directory that when deployed on a web server, emulates the java.net environment. To edit a file with the ability to preview it without checking in the file, follow this procedure:
- Load the file in your browser through java.net's web server, e.g. https://open-esb.dev.java.net/Downloads.html.
- Save this file to the emulation directory, e.g. Tomcat/webapps/ROOT/Downloads.html
- Now load the file into your browser through your local web server, e.g. http://localhost:9080/Downloads.html. This should look identical to what was loaded from java.net.
- Locate the text in the file to be edited. Change the text and save the file. Reload the file in your browser. Repeat this step until the file looks OK.
- Do a diff using a tool like Araxis Merge, or Beyond Compare between the local file and the file that was checked in into VCS, e.g. diff Tomcat/webapps/ROOT/Downloads.html and open-esb/www/Downloads.html.
- Merge the changes into the VCS file, i.e. open-esb/www/Downloads.html. Save and check in into VCS.
Here is the directory for Tomcat. If you use it for projects other than OpenESB, you can remove many of the files. Make sure to leave branding and css directories in tact. If you intend to reuse project_tools.html, don't forget to change the Google Analytics account ID.
SSL
Unfortunately, java.net uses SSL for all its files. This is unfortunate for two reasons:
- Files do not get cached in the browser: every time a user opens his browser and goes to OpenESB, all the images, style sheets, etc are reloaded again.
- To avoid security warnings in Internet Explorer, files that are included in an HTML file also should come through SSL. This rule is violated on the OpenESB main page (of all places!) where it loads a dynamic list of news item from a non-https server.
There's no workaround for this. Collabnet promised a long time ago that SSL would be removed but this still has not happened.
Changing the menu
The menu is defined in a json like format in a file called menu.js. This file is duplicated in the wiki. See below. Make sure to use absolute URLs because the file is used from different projects, i.e. different roots.
Wiki (not on java.net)
OpenESB is not using the java.net wiki infrastructure. I don't recall why this decision was made. New projects should probably evaluate the java.net wiki infrastructure before considering setting up a wiki elsewhere.
The OpenESB wiki is hosted on a machine hosted at Sun/Oracle, not at Collab net. It is collocated with other wikis, e.g. GlassFish, UpdateCenter, etc. The wiki engine is JSP wiki. The templates have been adapted to look and feel like the java.net OpenESB web site.
Since java.net has / used to have a lot of downtime, and to avoid having this downtime impact the wiki, the menu.js file is duplicated on the wiki. That java.net uses ssl is another reason to duplicate the file. For the same reason, the icons displayed in the menu are also duplicated on the wiki server.
Users are managed separately from java.net. User management / restrictions are necessary to avoid spam. Spam has been a been big problem on the OpenESB wiki and the other wikis collocated with it. Cleaning up spam can be done by directly manipulating the text files that make up the wiki page.
JSPWiki security is configured to use JAAS. This fact is important if the wiki needs to be setup on a different GlassFish server. In that case make sure to update the server.policy file of GlassFish. The policy file specifies that only "validated" users can edit pages. Users are stored in a user database and a group database. The latter specifies group membership, of which the "validated" group is important. Both files are in the etc directory of the wiki directory.
To add a user, the group database needs to be updated. This can be done through the web interface, http://wiki.open-esb.java.net/EditGroup.jsp?group=Validated or can be done by editing the group file directly.
The raw data files and user management files of the wiki can be accessed through SSH.
Since the user management of the wiki is separate from java.net, users can choose a different userid on the wiki than they do on java.net. Allowing this is a mistake in retrospect: editing the wiki requires a SCA (Sun Contributor Agreement) or employment in Sun/Oracle, and having to manage/audit two sets of users instead of one is a waste of time. Again, in retrospect.
java.net user management
Anybody on java.net can request developer privileges. When that happens, an email is sent by java.net to the project owner. An automated process is setup that replies automatically to the user list asking the applicant for his motivation, background, etc. This reply is specified in role-approval.policy in the www directory.
For some reason, many people ask for developer privileges. They are not known, have not contributed or communicated before, and are never heard from again.
Consequently, this automated process sounds nice, but is useless in practice. A better process is that someone first communicates on the mailing list, proves he/she is serious and able to contribute, after which the project owner grants privileges. Before privileges are granted, a signed SCA needs to be on file. This process is documented on the OpenESB web site.
The link to the membership management page is displayed in the java.net nav bar. Recall that the nav bar is hidden. To toggle display of the nav bar, load https://open-esb.dev.java.net/admin.html. Alternatively, jump to the page directly: https://open-esb.dev.java.net/servlets/ProjectMemberList.
When a user is granted commit privileges to a project, he/she also automatically gets commit privileges to the code repositories of the sub projects. E.g. granting commit privileges to open-esb also gives commit privileges to open-jbi-components, a sub project of open-esb. This is only the case if the VCS of the child project is of the same type as the parent project. Since open-jbi-components and open-esb both use CVS, open-esb committers have access to open-jbi-components. However, if the VCS system is different, e.g. openesb-builds uses SVN, separate access to this project needs to be granted next to open-esb.
java.net email management
Email lists are managed by an email list manager running in java.net. This email list manager is only partially integrated in java.net. Users can click on a button on the java.net site to subscribe to an email list. What effectively happens is that it will register the email address that is associated with that user at that moment in the list server. Similarly, a user can unsubscribe through a button, which will remove the email list associated with that user at that moment. Where this knowledge comes in handy? There is confusion if the user changes his email address associated with his java.net id between subscribing and unsubscribing. Sometimes users get so confused that they'll need help unsubscribing from the list. After all the last thing you want is repeated posts of "how do I unsubscribe from this list?".
Here is another task with respect to email: spam. Protecting the list to spam is very important. Therefore lists should be setup as "discuss" or "moderated". A setting of "discuss" means that only "subscribed" or "allowed" posters can post. If someone posts a message who is not in the "subscribed" or "allowed" list, the message will be forwarded to the list owner(s) for review. Replying to that message will allow the message to be posted. To avoid getting too many of these "review messages", frequent posters can be added to the "allowed" list. The "allowed" list is accessible from the email management page.
The link to the email list management page is displayed in the java.net nav bar. Recall that the nav bar is hidden. To toggle display of the nav bar, load https://open-esb.dev.java.net/admin.html. Alternatively, jump to the page directly: https://open-esb.dev.java.net/servlets/ProjectMailingListList.
Nabble is a view on the mailing list. OpenESB has integrated it on the web site. Note that posters are identified by their "from" email address only, but posters on Nabble somehow do not need to be in the "allowed" of "subscribed" list. This is fortunate, but puzzling from a technical point of view.
This brings me to one more email list related tasks: once in a while, someone posts an email that he/she wants to revoke. For example, someone could accidentally send a private email to the list, or include confidential (customer information for example) information in an email.
Of course there's no such thing as undoing the sending of an email. Subscribed users will have received the email by the time that the sender notices the mistake. However, there is an option to remove the email from Nabble and from the java.net mail browser. Note that there are two places where the email needs to be removed: both Nabble and java.net. Removing the mail there at least makes the mistake less visible and will make it harder for the mistaken email to be picked up by search engines.
java.net automation
Some of the management tasks on java.net can be automated. Kohsuke has developed an extensive set of tools that provide a Java api to java.net web interface: https://javanettasks.dev.java.net/.
I've written a few tools to make community management easier. One such tool is used to make an inventory of all the users in the community with commit privileges. Everybody in this list should either be a Sun/Oracle employee, or should have signed an SCA. The tool loads and parses the list of signatories (from https://sca.dev.java.net/CA_signatories.htm) and compares this list with the one extracted from java.net. Discrepancies are people who are currently or have been Sun/Oracle employees. Since it is difficult to track who is no longer an employee (especially with the layoffs that used to happen every now and then), the tool can email all "discrepancies" and ask them to either submit an SCA or to verify employment by clicking a link pointing to an internal server. These tools are available on https://open-esb-build.dev.java.net/svn/open-esb-build/trunk/community-mgt/.
