Watson Wins a Test Match

I have written here several times before about IBM’s ambitious project to build a computer system that can play — and win — on the popular TV game show, Jeopardy.  A test match, pitting the system, called Watson, against two human Jeopardy champions, Ken Jennings and Brad Rutter, is scheduled to be aired February 14-16.

A blog on the Discover magazine site now has a report that, in a trial run for the actual competition, Watson bested both of his human opponents.

Today, IBM rolled out its Jeopardy-playing computer, a whiz machine named Watson that was four years in the works. … Bad news, humans: In today’s exhibition of about 15 questions, Watson tallied $4,400, compared to $3,400 for Jennings and $1,200 Rutter.

This test was apparently put on primarily as a demonstration for the media, so it is not clear how representative it was of an actual Jeopardy match.  It also involved only 15 questions, while a real match involves 30 clues in each of two rounds.  There are other complications with the three “Daily Doubles”, in which the player has the clue to himself, and makes a wager on getting the right answer.  The “Final Jeopardy” round of the game has one clue; players must make a wager knowing the category of the clue, but not the clue itself. Even with an entirely human cast, there are sometimes surprises.

One of the interesting aspects of the reported test was that Watson was apparently pretty good at knowing what it knew.  It buzzes in to answer only about half the time, but is correct 85-95% of the time when it does.

Ken Jennings, who has won more Jeopardy matches than anyone, apparently was his usual cool and somewhat philosophical self.

And setting aside his own pride for a moment, Jennings says it’s worth noting that humans built the thing. Whoever wins, we win.

It should certainly make for some interesting television.

Update Thursday, January 13, 22:05 EST

The Wired site has a post on the Epicenter blog about this practice match.  It gives a few more technical details about the Watson system.  Its database contains about 200 million pages of content, and the hardware requirements probably mean it won’t be available on your phone in time for Christmas.

The system is powered by 10 racks of IBM POWER 750 servers running Linux, and uses 15 terabytes of RAM, 2,880 processor cores and can operate at 80 teraflops.

There is also an article on this practice match at PhysOrg.com.  It mentions an interesting analogy someone suggested to Ken Jennings.

Jennings said someone suggested his challenge was like the legend of John Henry, the 19th-century laborer who beat a steam drill in a contest but died in the effort.

There is a bit of irony here, since Jennings is himself a software engineer.  But I’m reminded of a parody of the song about John Henry, by the Smothers Brothers back in the 1960s, or at least the end of it:

Well, John Henry said to the Captain,
“Captain, I ain’t no fool,
Before I die with my hammer in my hand,
I’m gonna get me a steam drill tool, Lord, Lord,
Get me a steam drill tool.”

This should be fun to watch.

Update Friday, 14 January, 12:10 EDT

The Engadget blog now has a report of the test match posted, complete with video.

One Response to Watson Wins a Test Match

  1. […] to air February 14-16,  is continuing to attract a good deal of attention, especially since the trial match that took place last week.   The PBS science program, Nova, will air a special program about […]

%d bloggers like this: