Tuesday, March 06, 2007

Response to a Jolt Award critic

This started out as a comment on someone else's blog, but "just growed".

Original anti-Jolt rant here. You'll want to go there first or this will make no sense.

My response:

Mr. McGovern,

I wanted to wait until the judging process was over for the year before responding to your blog post. I had to laugh at the general tone of your post, especially the George Bush graphic. If only we judges had a fraction of the mystery and power you seem to ascribe to "they" (i.e., us)! Not to mention the unanimity that you imply; the judges are without question the most cantankerously independent herd of cats I've ever encountered professionally.

As for the lack of reader input on nominations, you may have a point there. This is why the Reader's Choice Awards were instituted. I personally regret that they seem to have gone by the wayside, I was one of the judges advocating that readers' nominations be considered in the Jolt mainstream and was only partly mollified by Reader's Choice.

"Folks such as Martin Fowler, Scott Ambler, Dave Thomas, Uncle Bob and other agilists serve on panels that help choose the winning products"...well, I'm flattered to be a 'folks such as' THAT august group. Unfortunately you need to check your facts, only one of those people is a Jolt judge and he regularly has to recuse himself because he's a prolific author in the space.

The process is necessarily subjective; a popularity contest among the readership would be "fairer" but much less useful to people actually interested in choosing tools. You can take or leave my personal credibility, but when people like Mike Riley or Andrew Binstock or Larry O'Brien weigh in on a tool, you'd do well to listen if you care about your organization's bottom line. It's not as if divine fiat has decreed "The Jolt Awards are the most important honor in software development". The Jolts matter as long as they provide value to the community, and not a heartbeat longer. But guess what? The industry still cares about them. Must not be as corrupt and scandalous as you posit. Or perhaps we've successfully pulled the wool over the eyes of the notoriously pliant and ignorant software-development industry for seventeen years straight. (And if you believe that one...let me show you the mail I get from readers when I goof in print. Ain't much gets past THAT crowd, friend.)

If you think you can do better, more power to you, start your own awards. (You're an industry "thought leader", it says so right there on your blog.) But you'll find that, no matter how much you protest your absolute integrity, no one is going to pay you the slightest scrap of attention. We earned our credibility, Mr. McGovern, and we earn it again every single year.

We don't publish much about the process, for three reasons: We don't want people gaming the system, we can't stand getting jawboned by sore losers (which happens often enough to disgust all of us), and we want the judges to feel absolutely free to express their opinions without worrying about who's looking over their shoulder.

But to offer a little insight, here's how the finalist selection process works: Vendors submit their nominations. As the list has grown, we've started requiring them to provide specific information (rather than just a URL to their product page) to justify why their product should receive the award. The judges then discuss the information, applying their knowledge of the industry to decide what is likely to be the most significant of the nominees, and finally vote by secret ballot. Frequently judges have experience with many of the products and can report on them; sometimes we have to go with our guts and try to predict what is of potential significance, even if it isn't hugely popular. Believe me, the issue of what constitutes a Jolt-worthy product is hotly contested -- every year.

We apply a sizable list of criteria. Is a product truly new, or just a bug-fix point revision of last year's? Is there widespread adoption? If not...should there be? Does a product have 'buzz'? Is it so worthy that we should put the weight of the Jolts behind creating some? And on, and on.

Finally, Mr. McGovern, I caution you to carefully consider the use of words like 'scandal'. It feels good when you rip on The Man, but forgive me if I take that a little personally, because The Man is me and some people who have earned my respect the hard way. If you have evidence of chicanery, the judges would sure like to know, because we want those people out of the process. I'm proud to be a member of this team, and I stand by them.


Blogger Scott W, Ambler said...

I wanted to add a few comments:
1. Some good information about the Jolt Awards can be found at the public site, http://www.joltawards.com/.

2. As far as OSS products go, I personally go out of my way each year to prod the OSS community to nominate themselves, a "huge" effort of filling in a fairly straightforward form. I also go out of my way to motivate people that are currently working with great tools to contact the appropriate vendors and prod them to nominate themselves. Sometimes it's like pulling teeth. Luckily, as a judge, I'm in a position to nominate one product into each category and I often choose to do so for OSS products, products from small firms that might not be able to afford the paltry nomination fee (which is used to cover the costs of running the awards each year), and worthy books that don't seem to be getting sufficient support from their publisher.

3. The discussions surrounding what it means to be Jolt-worthy, when should a judge recuse from a category (I'm particularly fervent about this issue), and what should be nominated for a Hall-Of-Fame award are always interesting each year. I can't begin to estimate how much effort goes into just those discussions.

4. The judges put in a lot of effort each year for no compensation other than licenses to a bunch of good software which they're not allowed to use for commercial work and some sort of gift which is invariably a shirt, coffee mug, or similarly priced trinket. None of us are in it for the money, which is exactly zero.

5. Readership awards are interesting but unfortunately easy to game. If you go back and look at previous Jolt award winners, particularly those who won productivity awards, you'll see that there are a lot of really great products that have won but that very likely would never have won any sort of popularity contest. In short, I think the Jolt Awards do a pretty good job of it.

6. If you know of any good products, I highly suggest that you make the vendor/team responsible for them aware of the Jolt Awards. Although the 2006 Jolts will be awarded at Software Development (www.sdexpo.com) in a couple of weeks, there's always next year.

- Scott
Practice Leader Agile Development
IBM Rational

11:00 AM  

Post a Comment

<< Home