More on Watson

I have written here a couple of times about IBM’s project to build a computer system that can be a successful contestant on the TV quiz show, Jeopardy!.   A test of the new system, called Watson, is scheduled to be broadcast next month; Watson will be playing against Ken Jennings and Brad Rutter, two of the long-running show’s most successful champions.

The IT World site has a new article that gives some additional information about the Watson system.  As has been rumored before, the system is designed to use large-scale parallel processing.

When it comes tackling a challenge as tough as answering a human question, the best computational approach may be to break the job down into multiple parts and run them all in parallel, IBM is betting.

The software component that manages the search for an answer is an IBM Research development called DeepQA.  This process is considerably more complex than an Internet search; the program’s clues are often based on word play, hints, and employ elliptical language, so that many possible solution paths have to be explored.

Sometimes the question is an obvious one, and a query to a specific database will do the trick. Most times, however, the question will kick off five or 10 searches across different data sources, each an interpretation of what the question might be.

…

The process is iterative. A set of results may require a new set of searches to be undertaken. “So, now you might have hundreds of processes, each generating additional candidate answers. Imagine that fan-out,” Ferrucci said. An end-result may have 10,000 sets of possible questions and their corresponding answers.

The hardware used in the system is designed for massively parallel operation.

Watson itself is composed of two racks of IBM Power7 System servers, or about 2,500 processor cores, all acting in harmony in a clustered configuration.

The Jeopardy! challenge, like IBM’s Deep Blue chess playing computer, of course has considerable publicity value.  But IBM also thinks that the lessons learned in building Watson will have important applications in other areas of natural language processing.

Ultimately, IBM plans to use this software to build commercial systems that could answer specific questions in selected fields, such as health care, tech support, and the legal field.

It will be most interesting to see how successful Watson is; I am looking forward to watching this match,