Playing Against Watson

In the course of my postings about IBM’s Watson project, to build a computer system that can compete on Jeopardy!, I’ve mentioned Stephen Baker’s blog a couple of times.  Mr. Baker is the author of a new book on the experiment, Final Jeopardy.  He has a new blog post about his experience in playing an informal test match against Watson last summer, and links to an article he wrote for the Huffington Post describing the experience.

He makes some good observations about the relative strengths and weaknesses of Watson compared to a human contestant.  On the plus side, Watson does extremely well on clues that are linked to specific facts, like names or dates.  This is not surprising, because those facts give Watson an “anchor” for its evaluation of potential answers.  (In another post, Baker has observed, I believe correctly, that Watson would be almost unbeatable if Jeopardy! had multiple choice questions.)   Watson also does not panic, or suffer momentary mental blanks.  (It does, however, crash.)  It does least well, relatively, with the Daily Doubles and Final Jeopardy (the last round in the game), because these tend to have more complicated clues, and also have the additional complication of wagering.  Baker also suggests that humans may have an edge in clues with few words, since they can probably finish reading more quickly than Watson can finish its evaluation.  (Note, however, that all contestants are prevented from “buzzing in” to answer until the show’s host, Alex Trebek, has finished reading the clue aloud.)

Of course, human contestants, at least, also have categories in which they are relatively strong or weak.   I would probably do reasonably well in science-related categories; on the other hand, Ken Jennings, who has won more Jeopardy! matches than anyone, and who will be playing against Watson in the challenge match, said on one show that he had a list of a few hundred favorite movies.  I am pretty sure I have never seen half that number of movies in my whole life, never mind counting favorites.

All of this conversation about Watson is undoubtedly music to IBM’s ears; getting publicity for this new technology is obviously one of their key objectives.  Nonetheless, it seems that they really have made some important advances in processing free-form questions.

Incidentally, Stephen Baker’s blog also has a post with a “Jeopardy! Party Kit” that you can download (it’s a PDF).  Obviously, this is intended as publicity for the book, but it’s kind of fun.

2 Responses to Playing Against Watson

  1. Rich, Thanks for writing this post. One point about your expertise in science: No matter how much you know, you have to beat the other players to the buzz. This was one of the frustrating things I found while playing Watson. Sometimes I knew more, but it knew quicker.

    • Rich says:

      Thanks for your comment. I agree; the “buzzing in” process is a very important part of the game, which people who watch it casually often seem to miss. Even so, it’s not uncommon to see a human contestant having trouble with getting the buzz in.

      I meant the science v. movies example to illustrate that human players, at least, are also subject to the chance (dis)advantage of the clue categories they face.

      I hadn’t thought of it before, but presumably Watson is, or could be made,less subject to the effects of category selection, since in principle the system is agnostic with respect to the areas it “knows” about.

%d bloggers like this: