Friday, February 26, 2016

8 things you may not know about Google Tag (Assistant) Recordings

Many people know Google Tag Assistant and use it as a tool to quickly look at their Google Analytics (and other) tags on their website. A few months ago we launched a new product: Google Tag Recordings which we integrated with Google Tag Assistant.

The idea behind Tag Recordings is that you as a GA user can record a journey through your website and then look at how GA would process the GA hits. This lets you validate that your site works correctly with GA. If there's an issue, you can troubleshoot it on the spot, make changes --for instance to the GA configuration-- and instantly see the effect of your changes.

Here is a 2 minute demo:

Here are 8 things you perhaps didn't know about Google Tag (Assistant) Recordings:

1. It’s a service!

Although integrated with Tag Assistant, Tag Recordings is a standalone service. It lives at this URL: It being a separate service instead of javascript in the browser, lets it use some of the same GA parsing and processing code that GA normally uses.

To use Tag Recordings, you need to have a recording of a journey through your website. The recording is in the form of a HAR. The easiest way to record is to use Google Tag Assistant: just hit the record button. But you can also use the record functionality in the Chrome Devtools. You can also use Firefox or IE or construct your own HAR.

2. Dropped hits? Check hit parse errors!

If you make an error in --for example-- the number format of an event sent from your web page, the hit will be rejected when it arrives at because the hit cannot be parsed. Example:
  ga('send', 'event', 'cat', 'act', 'lab', 5.2);
This fails to parse in the GA server because the last parameter must be an integer. The parse error does not make it back to your web browser, so you don't know that your hit will never make it into any reports.

While Tag Assistant does not show the error (it only does a small set of validations in the browser itself), Tag Recordings however does show the parse error: it uses the same code that the Google Analytics server uses.

3. No hits? Find the filter that is to blame

If you see no hits at all in a GA View, a filter may be to blame. Views often have lots of different filters, making it difficult to quickly see which filter is to blame. In Tag Recordings you can see if a hit is dropped for a View, and if so, which filter is responsible.

4. Mutating filters

Filters may change attributes of a hit. Also View settings may mutate hits, for example for query parameter stripping. Tag Recordings shows exactly how a hit changes after it is sent.

5. Find session breaks

Going from one domain to another domain may cause GA to start a new session. This of course will have a big impact on your GA reports. Because Tag Recordings looks at all the hits in a recording, it can figure out where GA would create a new boundary. See for an example the video above or this example report.

6. Verify your goals, filters, etc

When you create a new goal or filter, Google Analytics allows you to preview how the goal or filter will work using hits that were received and processed previously. For this to work the View already must have data, obviously. How the actual filter or goal will work, can only be seen after a day or so when new hits have come in and have been processed.

A powerful feature of Tag Recordings is that it operates on the current configuration of your property and view. So you can get instant feedback on what happens to hits if you change the configuration. That allows you to instantly validate changes to filters, goals, etc.

7. Pretend to be elsewhere

Once you have a recording, you can pretend that the recording was made from a different specific IP address, or you can pretend that the IP address belongs to a different geo. This makes for a quick approach to verify that your IP-specific or geo-specific filters work as you would expect them to work.

8. Troubleshoot GCLID problems

If you click on an ad in Google Search and you have setup autotagging, the GCLID needs to be propagated from all the way to the the ads landing page. If you have redirects in between, for instance because all traffic for mobile devices is redirected to a different sub domain, the GCLID may be mangled or dropped. This results in seeing fewer sessions in your GA report than ad clicks.
Tag Recordings reports all redirects and warns you if a GCLID gets dropped or mangled in a redirect. Example report

A demo site for Tag Recordings

If you want to try out Tag Recordings yourself, you can record a journey through a sample site at This site will send various types of hits and show several different problems.
Before you go there, first create a new GA property. Next, enter the UA-string on the site so that the hits will be sent to this new property. This allows Tag Recordings to use the configuration of your property and display all the information associated with the hits. It also allows you to make changes to the GA configuration of the property so that you can see what happens due to the configuration change.

More information

Saturday, June 29, 2013

Cleaning out my Google Reader Starred Items

Tomorrow is the last day of Google Reader. I went through all my "starred items". Here are some that were worth saving. Enjoy!

Mrs. O'Malley arrived in Boston from Ireland, and in no time at all her bean soup made her the talk of New England society. At a party celebrating the sale of her recipe to a fancy Charles Street restaurant, an old matron approached Mrs. O'Malley and said, "My dear girl, what is the secret of your soup?"

Mrs. O'Malley said, "The secret o' me soup is that I use but two hundred thirty-nine beans to make it."

The woman said, "Why only two hundred thirty-nine?"

Mrs. O'Malley said, "Because one more would make it too farty."

At the Olympic Games, Rhoda meets a man carrying an eight-foot-long metal stick.
"Excuse me," says Rhoda to the man. "Are you a pole vaulter?"
"No," says the man, "I'm German, but how did you know my name is Walter?"

What happened to the butcher when he backed up into the meat grinder?
He got a little behind in his orders.

What's the difference between roast beef and pea soup?
Anyone can roast beef.

"So, what do you do for a living?"
"I'm a cop."
"No, the regular kind."

*The caliber of your weapon is not as important as shot placement.*
*The reliable Beretta Jetfire .22 Short pistol is a personal favorite of mine and I am never without it. It saved my life a few years ago when attacked by a Grizzly while hiking in the mountains with a family member. I was able to escape, walking at a brisk pace, after I shot my brother-in-law in the knee.*

This little Italian boy and this little Jewish boy lived about a block apart in the neighborhood and basically grew up together. The Jewish boy was the son of a jeweler and the Italian boy was the son of a hitman. Oddly enough, they had the same birthday. Well, for their 12th birthday, the little Jewish boy gets a Rolex watch and the little Italian boy gets a .22 Baretta.

The next day they are out on the street corner comparing their presents and neither is happy so they switch gifts with each other. The little Italian boy goes home to show his father and his father is NOT pleased!

"What're you, nuts? Lemme tell you something, you idiot!! Some day you're gonna meet a nice girl, you're gonna wanna settle down and get married. You'll have a few kids, all that stuff. THEN one day, you're gonna come home and find your wife in bed with another man. What the heck ya gonna do??? Look at your watch and say, 'Hey, how long you gonna be?'

Four brothers left home for college, and they became successful doctors and lawyers and prospered. Some years later, they chatted after having dinner together. They discussed the gifts they were able to give their elderly mother who lived far away in another city.
The first said, "I had a big house built for Mama."
The second said, " I had a hundred thousand dollar theater built in the house."
The third said, "I had my Mercedes dealer deliver an SL600 to her."
The fourth said, "You know how Mama loved reading the Bible and you know she can't read anymore because she can't see very well. I met this preacher who told me about a parrot that can recite the entire Bible. It took twenty preachers 12 years to teach him. I had to pledge to contribute $100,000 a year for twenty years to the church, but it was worth it. Mama just has to name the chapter and verse and the parrot will recite it."

The other brothers were impressed. After the holidays Mom sent out her Thank You notes.

She wrote: "Milton , the house you built is so huge I live in only one room, but I have to clean the whole house. Thanks anyway."

"Marvin, I am too old to travel. I stay home, I have my groceries delivered, so never use the Mercedes. The thought was good. Thanks."

"Michael, you gave me an expensive theater with Dolby sound, it could hold 50 people, but all of my friends are dead, I've lost my hearing and I'm nearly blind. I'll never use it. Thank you for the gesture just the same."

"Dearest Melvin, you were the only son to have the good sense to give a little thought to your gift. The chicken was delicious. Thank you."

Luv Ya, Mama

A public union employee, a tea party activist, and a CEO are sitting at a table with a plate of a dozen cookies in the middle of it.
The CEO takes 11 cookies, turns to the tea partier and says, "Watch out for that union guy. He wants a piece of your cookie."

Dear Rubik's Cube,


Sincerely, Colorblind

Dear Icebergs,

Sorry to hear about the global warming. Karma's a bitch.

Sincerely, The Titanic

A couple preparing for a religious conversion meets with the orthodox rabbi for their final session.
The rabbi asks if they have any final questions.
The man asks, "Is it true that men and women don't dance together?"
"Yes," says the rabbi, "For modesty reasons, men and women dance separately."
"So I can't dance with my own wife?"
"Well, okay," says the man, "but what about sex?"
"Fine," says the rabbi. "A mitzvah within the marriage!"
"What about different positions?" the man asks.
"No problem," says the rabbi.
"Woman on top?" the man asks.
"Why not?" replies the rabbi.
"How about doggie-style?"
"Of course!"
"Well, what about standing up?"
"NO!" says the rabbi....
"Why Not???" asks the man.
"Could lead to dancing!"

A couple of biologists had twins. One they called John and the other control.

A donkey had an IQ of 186. He had no friends at all though. Even in the animal kingdom, nobody likes a smart-ass.

A key ring is a handy little gadget that allows you to lose all your keys at once.

Due to a water shortage in Ireland, Dublin swimming pools have announced they are closing lanes 7 and 8.

A termite walks into a barroom and asks, "Is the bartender here?"

A gorilla walks into a bar.
He orders a beer.
The bartender says, "That'll be $10. You know, we don't get many gorillas coming in here."
The gorilla says, "At $10 a beer, it's not hard to understand."

Man driving down road.
Woman driving up same road.
They pass each other.
Woman yells out window, "PIG!"
Man yells out window, "BITCH!"
Man rounds next curve.
Man crashes into a HUGE PIG in middle of road and dies.

A member of Parliament to Disraeli: "Sir, you will either die on the gallows or of some unspeakable disease."
"That depends, Sir," said Disraeli, "whether I embrace your policies or your mistress."

An Australian travel writer touring North America was checking out of the Spokane Hilton, and as he paid his bill said to the manager, "By the way, what's with the Indian chief sitting in the lobby? He's been there ever since I arrived."

"Oh that's 'Big Chief Forget-me Not'," said the manager. "The hotel is built on an Indian reservation, and part of the agreement is to allow the chief free use of the premises for the rest of his life. He is known as 'Big Chief Forget-me Not' because of his phenomenal memory. He is 92 and can remember the slightest detail of his life."

The travel writer took this in, and as he was waiting for his cab decided to put the chief's memory to the test.

"'ello, mate!" said the Aussie, receiving only a slight nod in return. "What did you have for breakfast on your 21st birthday?"

"Eggs," was the chief's instant reply, without even looking up, and indeed the Aussie was impressed.

He went off on his travel writing itinerary, right across to the east coast and back, telling others of Big Chief Forget-me Not's great memory. (One local noted to him that 'How' was a more appropriate greeting for an Indian chief than ''ello mate.') On his return to the Spokane Hilton six months later, he was surprised to see 'Big Chief Forget-me Not' still sitting in the lobby, fully occupied with whittling away on a stick.

"How?" said the Aussie.

"Scrambled," said the Chief.

"Doctor, Doctor! You've gotta help me! my husband thinks he's Moses!"

"Tell him to stop taking the Tablets."

I was in the restaurant yesterday when I suddenly realized I desperately needed to let off gas.. The music was really, really loud, so I timed it with the beat of the music.

After a couple of songs, I started to feel lots better. I finished my coffee, and noticed that everybody was staring at me...

Then I suddenly remembered that I was listening to my iPod.

Doctor: "I'm stumped. I guess we're just going to have to wait for the autopsy results."

Dear Diary,

Last Sunday in Church I told my wife, "I just let out a silent fart; what do you think I should do?"

She replied, "Get a new battery in your hearing aid."

Teacher: Maria, go the map and find North America.

Maria: Here it is.

Teacher: Correct. Now class, who discovered America?

Class: Maria

Two little kids are in a hospital, lying on gurneys next to each other outside the operating room.

The first kid leans over and asks, "What are you in here for?"

The second kid says, "I'm in here to get my tonsils out and I'm a little nervous."

The first kid says, "You've got nothing to worry about. I had that done when I was four. They put you to sleep, and when you wake up they give you lots of Jell-O and ice cream. It's a breeze."

The second kid then asks, "What are you here for?"

The first kid says," A circumcision."

The second kid says, "Whoa, Good luck buddy! I had that done when I was born. Couldn't walk for a year."

A wife was making a breakfast of fried eggs for her husband.

Suddenly, her husband burst into the kitchen. 'Careful,' he said, 'CAREFUL! Put in some more butter! Oh my gosh! You're cooking too many at once. TOO MANY! Turn them! TURN THEM NOW! We need more butter. Oh my gosh! WHERE are we going to get MORE BUTTER? They're going to STICK! Careful. CAREFUL! I said be CAREFUL! You NEVER listen to me when you're cooking! Never! Turn them! Hurry up! Are you CRAZY? Have you LOST your mind? Don't forget to salt them. You know you always forget to salt them Use the! salt. USE THE SALT! THE SALT!'

The wife stared at him. 'What in the world is wrong with you? You think I don't know how to fry a couple of eggs?'

The husband calmly replied, 'I just wanted to show you what it feels like when I'm driving.'

A young suitor was being led through the voluminous pages of the old family album by his girl's proud father. After seeing scores of members of the clan, the young man was finally shown the picture of a solid-looking old gentleman.

"This," said the father proudly, "is the founder of the family."

"What did he do?" asked the young man.

"He founded the family," the older man said again.

"I mean, sir," the suitor floundered, "what did he do to distinguish himself?"

"He was the founder of the family," the father rasped in exasperation.

"I understand that, sir," the suitor sighed. "I just wondered what the old gentleman did during the day."

Q: Did you hear about the dyslexic who tried to commit suicide?
A: He threw himself behind an oncoming train.

Last night at the Pub somebody told a joke disparaging Italians, by purporting them as stupid.
I was really offended and shouted, "Hey you! I'm Italian and I don't like you telling those Italian jokes!"
Then I pulled out a razor and everybody was quite concerned, until they realized I couldn't find a place to plug it in.

I was so depressed last night thinking about the economy, wars, jobs, my savings, Social Security, retirement funds, etc., I called the Suicide Lifeline. I got a call center in Pakistan , and when I told them I was suicidal, they got all excited, and asked if I could drive a truck.

Charles Dickens goes into a bar: "I'd like a martini, please."
Bartender: "Certainly, sir. Olive or twist?"

"As good as this Pub is," said the Scotsman, "I still prefer the pubs back home. In Glasgow, there's a wee place called McTavish's. The landlord goes out of his way for the locals. When you buy four drinks, he'll buy the fifth drink."

"Well, Angus," said the Englishman, "at my local spot in London, the Red Lion, the barman will buy you your third drink after you buy the first two."

"Ahhh, dat's nothin'," said the Irishman, "back home in my favorite pub, the moment you set foot in the place, they'll buy you a drink, then another, all the drinks you like, actually. Then, when you've had enough drinks, they'll take you upstairs and see dat you gets laid, all on the house!"

The Englishman and Scotsman were suspicious of the claims. The Irishman swore every word was true.

"Did this actually happen to you?"

"Well, not meself, personally, no," admitted the Irishman, "but it did happen to me sister quite a few times."

This guy is in the emergency room after a car accident. When he wakes up, the surgeon is at his bedside and says, "I have some good news and some bad news."

The patient says, "Give me the bad news first."

"OK," says the surgeon, "We had to amputate both legs."

"What's the good news?"

The surgeon says, "I really like your shoes; I'll give you $300 for 'em."

A young American tourist goes on a guided tour of a creepy old castle. At the end of the tour the guide asks her how she enjoyed it. She admits to being a bit worried about seeing a ghost in some of the dark cobwebby rooms and passages.

"Don't worry" says the guide, "I've never seen a ghost all the time I've been here."

"How long is that?" asks the girl.

"About three hundred years."

The devout cowboy lost his favorite Bible while he was mending fences out on the range.

Three weeks later a cow walked up carrying the Bible in it's mouth.

The cowboy couldn't believe his eyes.

He took the book out of the cow's mouth, raised his eyes heavenward and exclaimed, "It's a miracle!"

"Not really," said the cow.

"Your name was written inside the cover."

"Anyone with 'needs' to be prayed over, come forward, to the front at the altar," the Preacher says.

Leroy gets in line, and when it's his turn, the preacher asks, "Leroy, what do you want me to pray about for you?"

Leroy replies, "Preacher, I need you to pray for help with my hearing."

The preacher puts one finger in Leroy's ear, and he places the other hand on top of Leroy's head and prays and prays and prays, he prays a blue streak for Leroy. The whole congregation joined in with enthusiasm.

After a few minutes, the Preacher removes his hands, stands back and asks, "Leroy, how is your hearing now?"

Leroy says, "I don't know, Reverend, it ain't 'til next Wednesday."

The pastor asked if anyone in the congregation would like to express praise for answered prayers. Suzie Smith stood and walked to the podium.

She said, "I have a praise. Two months ago, my husband, Tom, had a terrible bicycle wreck and his scrotum was completely crushed. The pain was excruciating and the doctors didn't know if they could help him."

You could hear a muffled gasp from the men in the congregation as they imagine the pain that poor Tom must have experienced.

"Tom was unable to hold me or the children," she went on, "and every move caused him terrible pain. We prayed as the doctors performed a delicate operation, and it turned out they were able to piece together the crushed remnants of Tom's scrotum, and wrap wire around it to hold it in place."

Again, the men in the congregation cringed and squirmed uncomfortably as they imagined the horrible surgery performed on Tom.

"Now," she announced in a quivering voice, "thank the Lord, Tom is out of the hospital and the doctors say that with time, his scrotum should recover completely."

All the men sighed with unified relief. The pastor rose and tentatively asked if anyone else had something to say.

A man stood up and walked slowly to the podium.

He said, "I'm Tom Smith."

The entire congregation held its breath.

"I just want to tell my wife the word is sternum."

When weeding, the best way to make sure you are removing a weed and not a valuable plant is to pull on it. If it comes out of the ground easily, it is a valuable plant.

I lost the pub quiz last night by 1 point.

The last question was "where do most women have curly hair?"

Apparently the correct answer is Africa...

We must believe in luck. For how else can we explain the success of those we don't like.

Darryl and Harold were in a mental institution. The place had an unusual annual contest, picking two of the best patients and giving them two questions. If they got them correct, they were deemed cured and free to go.

Darryl was called into the doctor s office first and asked if he understood that he'd be free if he answered the questions correctly. Darryl said "Yes" and the doctor proceeded. "Darryl, what would happen if I poked out one of your eyes?"

Darryl said, "I'd be half blind."

"That's correct. What if I poked out both eyes?"

"I d be completely blind." The doctor stood up, shook Darryl s hand, and told him he was free to go.

On Darryl's way out, as the doctor filled out the paperwork, Darryl mentioned the exam to Harold, who was seated in the waiting room. He told him what questions were going to be asked and gave him the answers.

So Harold went into the doctor's office when he was called. The doctor went thru the formalities and then asked, "What would happen if I cut off one of your ears?" Remembering what Darryl had told him, he answered, "I'd be half blind."

The doctor looked a little puzzled, but went on. "What if I cut off the other ear?"

"I'd be completely blind," Harold answered."

"Harold, can you explain how you'd be blind?"

"My hat would fall down over my eyes."

In 1986, Peter Davies was on holiday in Kenya after graduating from Northwestern University.

On a hike through the bush, he came across a young bull elephant standing with one leg raised in the air. The elephant seemed distressed, so Peter approached it very carefully.

He got down on one knee and inspected the elephant's foot and found a large piece of wood deeply embedded in it. As carefully and as gently as he could, Peter worked the wood out with his hunting knife, after which the elephant gingerly put down its foot. The elephant turned to face the man, and with a rather curious look on its face, stared at him for several tense moments. Peter stood frozen, thinking of nothing else but being trampled. Eventually the elephant trumpeted loudly, turned, and walked away Peter never forgot that elephant or the events of that day.

Twenty years later, Peter was walking through the Chicago Zoo with his teenaged son. As they approached the elephant enclosure, one of the creatures turned and walked over to where Peter and his son Cameron were standing. The large bull elephant stared at Peter lifted its front foot off the ground and then put it down. The elephant did that several times then trumpeted loudly, all the while staring at the man.

Remembering the encounter in 1986, Peter couldn't help wondering if this was the same elephant. Peter summoned up his courage, climbed over the railing and made his way into the enclosure. He walked right up to the elephant and stared back in wonder. The elephant trumpeted again, wrapped its trunk around one of Peter legs and slammed his stupid ass against the railing, killing him instantly.

Probably wasn't the same elephant.

One winter morning a husband and wife in Denver were listening to the radio during breakfast. They heard the announcer say, "We are going to have 8 to 10 inches of snow today. You must park your car on the even-numbered side of the street, so the snowplows can get through."

So the good wife went out and moved her car.

A week later while they are eating breakfast again, the radio announcer said, "We are expecting 10 to 12 inches of snow today. You must park your car on the odd-numbered side of the street, so the snowplows can get through."

The good wife went out and moved her car again.

The next week they are again having breakfast, when the radio announcer says, "We are expecting 12 to 14 inches of snow today. You must park...." Then the electric power went out. The good wife was very upset, and with a worried look on her face she said, "Honey, I don't know what to do. Which side of the street do I need to park on so the snowplows can get through?"

With the love and understanding in his voice that all men who are married to blondes exhibit, the husband replied, "Why don't you just leave it in the garage this time?"

One day an old German Shepherd starts chasing rabbits and, before long, discovers that he's lost. Wandering about, he notices a panther heading rapidly in his direction with the intention of having lunch.

The old German Shepherd thinks, "Oh, oh! I'm in deep doo-doo now!" Noticing some bones on the ground close by, he immediately settles down to chew on the bones with his back to the approaching cat. Just as the panther is about to leap, the old German Shepherd exclaims loudly, "Boy, that was one delicious panther! I wonder if there are any more around here?"

Hearing this, the young panther halts his attack in mid-strike; a look of terror comes over him, and he slinks away into the trees. "Whew!" says the panther, "That was close! That old German Shepherd nearly had me!"

Meanwhile, a squirrel watching the whole scene from a nearby tree figures he can put this knowledge to good use and trade it for protection from the panther. So, off he goes. The squirrel soon catches up with the panther, spills the beans, and strikes a deal for himself with the panther.

The young panther is furious at being made a fool of and says, "Here, squirrel, hop on my back and see what's going to happen to that conniving canine!"

Now, the old German Shepherd sees the panther coming with the squirrel on his back and thinks, "What am I going to do now?" but instead of running, the dog sits down with his back to his attackers, pretending he hasn't seen them yet, and just when they get close enough to hear, the old German Shepherd says... "Where's that squirrel? I sent him off an hour ago to bring me another panther!"

Moral of this story...

Don't mess with the old dogs... Age and skill will always overcome youth and treachery! BS and brilliance come only with age and experience.

Gracie Allen's Classic Recipe for Roast Beef:

1 large roast of beef

1 small roast of beef

Take the two roasts and put them in the oven.

When the little one burns, the big one is done.

A man was driving down the road and ran out of gas. Just at that moment, a bee flew in his window.

The bee said, 'What seems to be the problem?'

'I'm out of gas,' the man replied.

The bee told the man to wait right there and flew away. Minutes later, the man watched as an entire swarm of bees flew to his car and into his gas tank. After a few minutes, the bees flew out.

'Try it now,' said one bee.

The man turned the ignition key and the car started right up. 'Wow!' the man exclaimed, 'what did you put in my gas tank'?

The bee answered, 'Bee Pee.'

When I was young my intent was to go to medical school, but I didn't pass the entrance exam. One of the questions was:

"Rearrange the letters P N E S I to spell out an important part of human body that is more useful when erect."

Those who spelled SPINE became Doctors...

The rest of us ended up working for the government.

Wednesday, March 14, 2012 became Oracle, and then…

When Oracle took over Sun, the idea was that employee's blogs would survive the acquisition. They did, sort of. Right now, a lot of pictures on my blog at Oracle have disappeared.

Time to finally move my blog to blogger where I have full control over it, and where I don't have to worry that some day it will disappear altogether. There are a few entries on here (about memory leaks) that still draw a few hundred hits per day.

Missing pictures:

bytespersecond cmstates-access cmstates-enlist gcleakservlet gcsimpleservlet gflifecycle hat1 hat2 hat3 hat4 messagespersecond simpleservlet speakingatjavaone2008 writespersecond


Saturday, July 31, 2010

GlassFish Security

Pact Publishing was kind enough to send me a copy of the book "GlassFish Security" that was released very recently.


It's tough to find the time to read a book cover to cover. In fact, it's been a while since I've read a book from beginning to end. Typically I'm only interested in a few chapters which I then read. Later, when the need arises, I may get back to other chapter. It's like treating a book like an encyclopedia or a dictionary. I bet that most people read technical books that way nowadays. So it's important that a book lends itself to be read that way.

Security is a very broad field with many many different topics, e.g. user authentication and authorization in web applications, integration with an external security server, web services security, and so on. Fortunately, very few people will have to deal with all these aspects at the same time. GlassFish Security covers many if not all of those aspects, and that's another reason why it should be possible to read this book like an encyclopedia.

I went back in time and tried to remember all the times that I have had anything to do with security in GlassFish and checked what the book had to say about it. For instance, to start simple, there's the issue of configuring realms and users. There's the time I tried to get JSPWiki to work on GlassFish using declarative authentication and authorization, something that goes through server.policy. More advanced, there's the time we tried to integrate GlassFish with AccessManager. On all these topics, the book delivers. There are many other topics (e.g. integration with OpenSSO, securing JMX, etc), all described in detail without wasting much space on repetition or endless code or XML listings. The book would have saved me a lot of time had I had it at the time when I needed it.

If you're using GlassFish, this book belongs on your bookshelf!

Wednesday, April 28, 2010

Google TV Ads

On Tuesday I went to the Google office to talk about what I might be working on. It's likely going to be Google TV Ads. What is that? Here is a 3 minute video that explains it.

Monday, April 26, 2010

Working with tips and tricks for project owners

In the past two years I've managed the OpenESB open source project. This project is hosted on In this post I outline a number of tips and tricks I've learned while managing this project on This is useful information for my successor, but it also may be useful to other people who are managing projects on

Web presence

An important factor in the success of an open source project is its web presence: it will be the first point of contact between a potential new user and the project.

New visitors will want to see answers to these four items immediately:

  • What is it?
  • License?
  • Getting started
  • Downloads

It is important that the links to these pages are static, i.e. that they do not change. That is because a new visitor may enter the site through a link on a different site or through a search engine. For this very reason, it is especially important to keep a single downloads page rather than a page per version, so that users do end up immediately on the latest version, rather than a previous (old) version.

Looking at the front page of OpenESB, these four items are immediately apparent, and the links are permanent. Two more items are important: a visitor will judge a project for it being

  • Alive
    • Active mailing list
    • Recent/frequent announcements
  • Professional
    • Well organized, thought through
    • Documentation easily found and comprehensive
  • Inviting
    • New users should feel welcome to not just use the product, but also to participate in the community.

You be the judge how well these goals have been met. Let's look at the mechanics of now.

openesbsite project web site

How processes HTML files

The HTML files for a project's web site are stored in the project's VCS system in a directory called www. There's a big twist here: although the HTML files are ordinary HTML files, i.e. with an <HTML>, <HEAD> and <BODY> tag, the web server reads the HTML files and does a number of substitutions on the file before it writes the resulting file to the client.

Conceptually, there is a template HTML file which has its own <HEAD> element, its own <BODY> element with static content, and some placeholders in these two elements for:

  • some elements of <HEAD> element of the user defined HTML file, e.g. <TITLE>
  • the contents of the project_tools.html file
  • the contents of the <BODY> element from the user defined HTML file

As such, the project owner has to perform special tricks to have the page displayed in other ways than the standard way.

In OpenESB we have several projects. All of them should have the same look and feel, i.e. the OpenESB "brand" and a common navigation. The same goes for the OpenESB wiki.

How OpenESB organizes HTML files

Each project has the same project_tools.html file. This file takes care of the following:

  • It includes custom CSS files. These files override the styles.
  • It hides the standard navigation bar. For admins, the navbar display can be turned on: the state is maintained in a cookie. The value in the cookie can be toggled by loading the admin.html file in OpenESB:
  • Loads the menu. The menu is defined in a separate JavaScript file.
  • Displays the search box
  • Sets an event handler that is invoked when the page's loading is complete. The event handler manipulates the layout of the page and invokes Google Analytics.

As a result, the other HTML files in OpenESB or any of its sub projects don't have to bother with the common look and feel, the menu, etc. All of that is taken care of by project_tools.html, which as mentioned should be duplicated to all the OpenESB projects.

Some or if you will, many, HTML files in OpenESB have another common style element: a right hand side bar that displays common advertising for downloading GlassFish ESB, the wealth of components, and the a partner highlight. This right side bar is taken care of a separate JavaScript file. Files that should display this right hand side bar should explicitly be formatted to do so: it should contain a table in which the right column loads the JavaScript of the right hand side bar.

Previewing HTML

The fact that the look and feel of an HTML file is now defined in project_tools.html makes it difficult to see what a file will actually look like when deployed (i.e. checked in into the project's VCS) on There is a workaround: I created a directory that when deployed on a web server, emulates the environment. To edit a file with the ability to preview it without checking in the file, follow this procedure:

  1. Load the file in your browser through's web server, e.g.
  2. Save this file to the emulation directory, e.g. Tomcat/webapps/ROOT/Downloads.html
  3. Now load the file into your browser through your local web server, e.g. http://localhost:9080/Downloads.html. This should look identical to what was loaded from
  4. Locate the text in the file to be edited. Change the text and save the file. Reload the file in your browser. Repeat this step until the file looks OK.
  5. Do a diff using a tool like Araxis Merge, or Beyond Compare between the local file and the file that was checked in into VCS, e.g. diff Tomcat/webapps/ROOT/Downloads.html and open-esb/www/Downloads.html.
  6. Merge the changes into the VCS file, i.e. open-esb/www/Downloads.html. Save and check in into VCS.

Here is the directory for Tomcat. If you use it for projects other than OpenESB, you can remove many of the files. Make sure to leave branding and css directories in tact. If you intend to reuse project_tools.html, don't forget to change the Google Analytics account ID.


Unfortunately, uses SSL for all its files. This is unfortunate for two reasons:

  • Files do not get cached in the browser: every time a user opens his browser and goes to OpenESB, all the images, style sheets, etc are reloaded again.
  • To avoid security warnings in Internet Explorer, files that are included in an HTML file also should come through SSL. This rule is violated on the OpenESB main page (of all places!) where it loads a dynamic list of news item from a non-https server.

There's no workaround for this. Collabnet promised a long time ago that SSL would be removed but this still has not happened.

Changing the menu

The menu is defined in a json like format in a file called menu.js. This file is duplicated in the wiki. See below. Make sure to use absolute URLs because the file is used from different projects, i.e. different roots.

Wiki (not on

OpenESB is not using the wiki infrastructure. I don't recall why this decision was made. New projects should probably evaluate the wiki infrastructure before considering setting up a wiki elsewhere.

The OpenESB wiki is hosted on a machine hosted at Sun/Oracle, not at Collab net. It is collocated with other wikis, e.g. GlassFish, UpdateCenter, etc. The wiki engine is JSP wiki. The templates have been adapted to look and feel like the OpenESB web site.

Since has / used to have a lot of downtime, and to avoid having this downtime impact the wiki, the menu.js file is duplicated on the wiki. That uses ssl is another reason to duplicate the file. For the same reason, the icons displayed in the menu are also duplicated on the wiki server.

Users are managed separately from User management / restrictions are necessary to avoid spam. Spam has been a been big problem on the OpenESB wiki and the other wikis collocated with it. Cleaning up spam can be done by directly manipulating the text files that make up the wiki page.

JSPWiki security is configured to use JAAS. This fact is important if the wiki needs to be setup on a different GlassFish server. In that case make sure to update the server.policy file of GlassFish. The policy file specifies that only "validated" users can edit pages. Users are stored in a user database and a group database. The latter specifies group membership, of which the "validated" group is important. Both files are in the etc directory of the wiki directory.

To add a user, the group database needs to be updated. This can be done through the web interface, or can be done by editing the group file directly.

The raw data files and user management files of the wiki can be accessed through SSH.

Since the user management of the wiki is separate from, users can choose a different userid on the wiki than they do on Allowing this is a mistake in retrospect: editing the wiki requires a SCA (Sun Contributor Agreement) or employment in Sun/Oracle, and having to manage/audit two sets of users instead of one is a waste of time. Again, in retrospect.

openesbwiki user management

Anybody on can request developer privileges. When that happens, an email is sent by to the project owner. An automated process is setup that replies automatically to the user list asking the applicant for his motivation, background, etc. This reply is specified in role-approval.policy in the www directory.

For some reason, many people ask for developer privileges. They are not known, have not contributed or communicated before, and are never heard from again.

Consequently, this automated process sounds nice, but is useless in practice. A better process is that someone first communicates on the mailing list, proves he/she is serious and able to contribute, after which the project owner grants privileges. Before privileges are granted, a signed SCA needs to be on file. This process is documented on the OpenESB web site.

The link to the membership management page is displayed in the nav bar. Recall that the nav bar is hidden. To toggle display of the nav bar, load Alternatively, jump to the page directly:

When a user is granted commit privileges to a project, he/she also automatically gets commit privileges to the code repositories of the sub projects. E.g. granting commit privileges to open-esb also gives commit privileges to open-jbi-components, a sub project of open-esb. This is only the case if the VCS of the child project is of the same type as the parent project. Since open-jbi-components and open-esb both use CVS, open-esb committers have access to open-jbi-components. However, if the VCS system is different, e.g. openesb-builds uses SVN, separate access to this project needs to be granted next to open-esb. email management

Email lists are managed by an email list manager running in This email list manager is only partially integrated in Users can click on a button on the site to subscribe to an email list. What effectively happens is that it will register the email address that is associated with that user at that moment in the list server. Similarly, a user can unsubscribe through a button, which will remove the email list associated with that user at that moment. Where this knowledge comes in handy? There is confusion if the user changes his email address associated with his id between subscribing and unsubscribing. Sometimes users get so confused that they'll need help unsubscribing from the list. After all the last thing you want is repeated posts of "how do I unsubscribe from this list?".

Here is another task with respect to email: spam. Protecting the list to spam is very important. Therefore lists should be setup as "discuss" or "moderated". A setting of "discuss" means that only "subscribed" or "allowed" posters can post. If someone posts a message who is not in the "subscribed" or "allowed" list, the message will be forwarded to the list owner(s) for review. Replying to that message will allow the message to be posted. To avoid getting too many of these "review messages", frequent posters can be added to the "allowed" list. The "allowed" list is accessible from the email management page.

The link to the email list management page is displayed in the nav bar. Recall that the nav bar is hidden. To toggle display of the nav bar, load Alternatively, jump to the page directly:

Nabble is a view on the mailing list. OpenESB has integrated it on the web site. Note that posters are identified by their "from" email address only, but posters on Nabble somehow do not need to be in the "allowed" of "subscribed" list. This is fortunate, but puzzling from a technical point of view.


This brings me to one more email list related tasks: once in a while, someone posts an email that he/she wants to revoke. For example, someone could accidentally send a private email to the list, or include confidential (customer information for example) information in an email.

Of course there's no such thing as undoing the sending of an email. Subscribed users will have received the email by the time that the sender notices the mistake. However, there is an option to remove the email from Nabble and from the mail browser. Note that there are two places where the email needs to be removed: both Nabble and Removing the mail there at least makes the mistake less visible and will make it harder for the mistaken email to be picked up by search engines. automation

Some of the management tasks on can be automated. Kohsuke has developed an extensive set of tools that provide a Java api to web interface:

I've written a few tools to make community management easier. One such tool is used to make an inventory of all the users in the community with commit privileges. Everybody in this list should either be a Sun/Oracle employee, or should have signed an SCA. The tool loads and parses the list of signatories (from and compares this list with the one extracted from Discrepancies are people who are currently or have been Sun/Oracle employees. Since it is difficult to track who is no longer an employee (especially with the layoffs that used to happen every now and then), the tool can email all "discrepancies" and ask them to either submit an SCA or to verify employment by clicking a link pointing to an internal server. These tools are available on

Wednesday, April 21, 2010

OpenESB under Oracle

It's been a few months since Oracle acquired Sun. At the time there were a lot of questions about what Oracle would do with OpenESB. On Feb 15, I posted a plan to the users mailing list. What was the plan? And how is the plan coming together? How is OpenESB doing a few months after the acquisition? What will the future hold for OpenESB?

First, let's try to understand Oracle's perspective. Oracle already has an integration product: the SOA Suite. Through the BEA acquisition it got another one, and through the Sun acquisition it got yet two more: CAPS and OpenESB. Of course there's no sense in keeping high levels of investments in all these products; it makes much more sense to focus on one product. What is this strategic product? Oracle is very clear about that: Oracle's strategic SOA middleware platform is the Oracle SOA Suite. Consequently, Oracle has reduced the level of investment in OpenESB.

That was the bad news. Now for the good news. Oracle could have just pulled out from OpenESB and let it fall into a black hole. Or worse, it could have taken the site down, removed the downloads, or put other obstacles in place. It did not. It did quite the contrary.

Although Sun was the main sponsor behind OpenESB, Oracle recognized that a lot of  people made investments into OpenESB, either in the community or into their own company by using OpenESB, and with a feeling of responsibility and fairness, Oracle decided to help the community to find a way to stand on its own, so that all these investments would bear fruit for as long a time as possible.

Did you invest in OpenESB? Let's look at what will change for you depending on the kind of investment you made.

What will change for you

What will change for you if you bought GlassFish ESB support? Nothing will change: you are still fully supported. The same Sun SOA Support department is still available to help on issues. Should patches be created to address issues, the full Sun SOA Sustaining department is still there to create patches.

Are you using GlassFish ESB but did not buy support? Unlike Sun, Oracle is no longer trying to sell GlassFish ESB licenses to new customers. Instead, you may rely on community support, or commercial support provided by one of the OpenESB community partners. The user mailing list is nowadays less frequented by Sun/Oracle engineers, but other community members have stepped up and are keeping the mailing list responsive. More about that below.

Are you investing in OpenESB by contributing code or other artifacts, or are considering to do so? You will find that it has become easier to contribute. I've put together a new governance document that gives greater freedom to community contributors. Oracle can still exert influence, but that is intended to keep the peace in the community should that be necessary. Overall, you will find it easier to propose and implement new changes, to commit code, and last but not least, to influence the roadmap. This brings us to the future of OpenESB.

The future of OpenESB

The future of OpenESB revolves around OpenESB becoming an open source community that can stand on its own, i.e. without Sun or Oracle as the single major sponsor. That transformation will of course not happen overnight, and Oracle is committed to help with this transformation. For instance, Oracle will do periodic builds and post these on the downloads site. Another commitment is that Oracle will merge patches that it makes for customers, back into the open source repository.

What did Oracle do so far? Next to the governance document I already mentioned, the sources of GlassFish ESB v2.2 can now be found in the open source repository. The HL7 BC and the WLM SE are also back in the open source repository, and the binaries can be downloaded from the downloads page. The process of an automatic periodic build has incurred some delays due to technical issues, but is well on its way.

OpenESB today

How is OpenESB doing today? Let's look at the users mailing list. From the Users mailing list on Markmail, it can be seen that the list activity has declined a bit, but is not much below the activity of that of a year ago.


What we also can learn from Markmail is that community members who are not on the Oracle payroll are stepping up. For example, here are the most active posters for March:


Another metric that we can look at is the number of users. In GlassFish ESB v2.1 we introduced a feature in which NetBeans checks for updates upon startup. By looking at the traffic to the updates-server, we can estimate how many users there are of GlassFish ESB. I defined the number of users as the number of NetBeans installations that ping the server at least three times in a time window greater than five days. As can be seen, the number of users is going up.


In terms of a roadmap there are no concrete proposals from the community yet, but several members have expressed interest in continuing with Fuji.

More good news: a few new committers have joined the project and are contributing code.