Ken Jennings v. Lore

December 16, 2012

In the course of writing this blog, I’ve referred to articles from quite a few different publications.  Until now, though, I have not referenced Parade magazine — the color supplement that comes in the advertising package with the Sunday Washington Post, and other papers.  It is not, frankly, a publication that I expected to be citing.  But this week, Parade has an article by Ken Jennings, the Jeopardy! game show champion†, addressing, and debunking, some hoary chestnuts of folk wisdom, the kind I refer to as “lore”, that parents often tell their children, without necessarily wondering whether or not they are true.  As Jennings puts it:

That’s the dirty secret of parenting: It’s a big game of Telephone, stretching back through the centuries and delivering garbled, though well-intentioned, medieval bromides to the present.

[“Telephone” is the American name for the game called “Chinese Whispers” in the UK.]

I suspect most readers will have heard most of these precepts at one time or another:

  1. “Stay away from the poinsettia! The leaves are poisonous.”
  2.  “No swimming for an hour after lunch. You’ll cramp up.”
  3. “When you start shaving, the hair will grow in thicker.”
  4. “Don’t eat snow—it’ll make you sick!”
  5. “Drink eight 8-ounce glasses of water a day.”
  6. “It’s too dark in here. You’ll hurt your eyes.”
  7. “You are a special little snowflake.”
  8. “You need hydrogen peroxide on that.”
  9. “Take off the Band-Aid to let your cut air out.”
  10. “Don’t cross your eyes—they’ll get stuck like that!”
  11. “No soda! The sugar makes you hyper.”
  12. “Don’t wake a sleepwalker.”
  13. “Most of your body heat escapes through your head!”
  14. “You’re not fat. You’re just big-boned.”
  15. “If you pick up a baby bird, its mommy will reject it.”

Some of these, such as numbers 3 and 7, are just more or less harmless nonsense.  Others — number 12, on sleepwalking, is an example — embody basically correct conclusions for the wrong reasons.  (In this, they resemble the frequently given advice to get into a car in a lightning storm.)  Others are just nonsense from top to bottom.

For example, I have heard many people express their belief in number 5, the idea that one needs to drink eight 8-ounce glasses of water every day.  As Jennings points out, many of these people have lost sight of the considerable amount of water that we take in every day in the form of food.   I’ve also heard the advice, mentioned in the article, that liquids like coffee or beer, don’t count, because the caffeine or alcohol acts as a diuretic.  At some level, this is true: if you drink a quart of straight whisky at one sitting, you probably will get a bit dehydrated, among other things.  On the other hand, the effect does have something to do with relative amounts: if I put one teaspoon of whisky, or coffee, into ten gallons of water, I am quite confident that you can drink as much of the resulting mixture as you want with no risk of dehydration.

One might argue that none of these adages is especially pernicious, so little harm is done.  But getting people to behave rationally, even once in a while, seems to be hard.  Reinforcement of irrational thinking is hardly constructive.

As Kin Hubbard said, “Tain’t what a man don’t know that hurts him; it’s what he knows that just ain’t so. ”


† Ken Jennings is a champion of the TV game show, Jeopardy!, who won more consecutive games (74) than any other player.  He was also one of the two human players involved in the Jeopardy! challenge match with IBM’s Watson computer system.

Elementary Watson

January 6, 2012

Early last year, I wrote several times about IBM’s Watson system, and its victory against two human champions in the popular TV game show, Jeopardy!.  IBM has also announced some initiatives to use Watson’s technology in medical diagnosis.

In December 2011, at the USENIX Large Installation System Administration Conference [LISA] in Boston, MA, Michael Perrone, Manager of Multi-Core Computing at IBM’s Thomas J. Watson Research Center, gave an entertaining talk on Watson’s methods, and how they were developed in the context of the Jeopardy! game.  He presents some statistics and examples to show how the game is difficult, compared to something like chess, and how what Watson does is different from a search engine, like Google.  Inference from natural language is tricky.  For example, what do these three things have in common: shirts, TV remotes, and elevators?  Well, they all have buttons.

Mr. Perrone also gives examples of some of the questions Watson got wrong in its early incarnations (it also made a couple of bloopers in the match against human champions).  For example, in the category “New York Times Headlines”, the clue was:

An exclamation point was warranted for the end of this! in 1918.

Most people would probably realize the correct answer is “What is World War I?” — Watson came up with a silly but eminently logical answer: “What is a sentence?” The talk also includes some information about the hardware used to implement Watson for Jeopardy!.

The talk has now been made available on YouTube; it runs about one hour and twenty minutes.  The slides from the talk [PDF] are also available.

Dr. Watson Will See You Now

September 18, 2011

Today’s edition of the Washington Post has an interesting article,by Martin Ford, on the potential application of IBM’s Jeopardy!-winning Watson technology in medicine.   You will probably recall that medical diagnosis was mentioned prominently by IBM as a potential practical use of Watson’s technology. On September 12, IBM and Wellpoint, the largest medical benefits company in the US by membership, announced an agreement for the joint development of medical applications.

The article points out some of the areas in which Watson could provide real assistance to medical personnel, based on its ability to process huge amounts of information in unstructured, natural language documents.

Watson could churn through millions of case histories to learn what diagnosis is likely to be correct and what treatment would be the most effective. The system could almost instantly process medical textbooks, electronic medical records and the latest published research, illuminating obscure links among studies in seemingly unrelated specialties. Watson could someday be a standard diagnostic tool. Its ability to make sense of a universe of data would be far beyond that of any person or team of experienced physicians.

Watson’s ability to process enormous amounts of information will come in handy; it has been estimated that the overall body of medical knowledge doubles in size about every five years.  A more efficient, and less error prone, method of assimilating the flood of new information might help check the ongoing escalation of health care costs.  Combined with the growing  use of electronic medical records, the system could help spot unusual conditions, or rare but serious drug interactions.

Mr. Ford also points out that, if Watson can establish a track record as a diagnostician, it might help alleviate a projected shortage of physicians, especially in primary care. Some medical practices already use physician’s assistants to help them care for more patients; a system like Watson might make this a lot more common.

The article also touches on the threat that systems like Watson might pose to some sectors of the economy.  I’ve mentioned before the idea, suggested by the late computer science and AI pioneer, Joseph Weizenbaum, that intelligent systems might devalue routine mental labor, just as the Industrial Revolution devalued routine physical labor.  (I wrote about a proposal to use Watson in marketing not long ago.)   As with other disruptive technologies (genetic engineering comes to mind), there is, in some sense, no turning back. The knowledge of how to build Watson can’t be unlearned; the genie can’t be put back in the bottle.  To the extent that humans possess intelligence beyond that of machines, we will need to use it to make wise use of our discoveries.

[Martin Ford is the author of The Lights in the Tunnel: Automation, Accelerating Technology and the Economy of the Future; he also writes the Econfuture blog.]

Watson’s Encore

September 3, 2011

Back in the beginning of this year, I wrote several times about IBM’s Watson, a massively parallel computing system designed to compete on Jeopardy!,  the popular TV game show.  A televised test match was held in February, pitting Watson against two of the long-running shows most successful champions, Ken Jennings and Brad Rutter.   Watson won handily, raising some interesting questions in the process; the whole experiment was fascinating to watch.  It also prompted some thought-provoking discussion of different approaches to answering unstructured queries.

This past week, IBM and Sony Pictures, the producers of Jeopardy!, jointly announced that there would be an “encore presentation” (that is, a re-run) of the Watson match this month.  The three half-hour shows are slated to air on September 12, 13, and 14, in the regular local Jeopardy! time slots.  (The Jeopardy! site has some short video ads promoting the match.)   I’m looking forward to watching it again.

More Stuff on Watson

February 18, 2011

As I guess one might expect, from the outcome of the Jeopardy! IBM challenge, in which the computer system Watson scored a convincing win over former Jeopardy! champions Ken Jennings and Brad Rutter, there has been a good deal of subsequent conversation about the result, what it means, and so on.  There are a couple of items, in particular, that I wanted to mention here, since I think they are more informed and insightful than much of the commentary that I’ve seen.

The first, which I’ve mentioned before, is Stephen Baker”s Final Jeopardy blog, on which he has a number of interesting and amusing posts about the match.  He has always made insightful comments on the project; and, of course, he is the author of the book, Final Jeopardy: Man vs. Machine, which I am in the process of reading, and will review here when I’m done.  He notes a couple of questions about Watson posted on Twitter, one of which was “What is love?”.  It seems to me unreasonable to expect a machine to come up with the answer to a question that humans have been trying to answer, never entirely successfully, for millenia.  As Baker has often been at pains to remind us, it is tempting to use anthropomorphic language to describe what Watson does, but it really is better to avoid it, lest we disappear down the rabbit hole in discussions of the meaning of meaning.

Baker also mentions an article by Ben Zimmer, language columnist for the New York Times, who writes in The Atlantic about Watson’s natural language processing ability.  He refers to Ken Jennings’s quip at the end of the match (I quoted it here.).   As he points out, although Watson’s Natural Language Processing abilities are very impressive, it sees words as symbols, and doesn’t “get” their cultural context.

All of this is to say that while Ken and Brad lost the battle, Team Carbon is still winning the language war against Team Silicon. The “war” metaphor, incidentally, had been playing out for weeks, stoked by IBM and Jeopardy! to build public interest in the tournament.  …  IBM knew from the Kasparov vs. Deep Blue days that we’re all suckers for the “man vs. machine” trope, going back to John Henry’s mythical race against the steam-powered hammer.

The last item I want to share is one from the site.  TED describes itself this way:

TED is a small nonprofit devoted to Ideas Worth Spreading. It started out (in 1984) as a conference bringing together people from three worlds: Technology, Entertainment, Design. Since then its scope has become ever broader.

It also runs the site, which features the “TED talks”, some of the best presentations from TED’s semi-annual conferences, on a wide variety of subjects.   It awards the annual TED Prize, for “One Idea to Change the World”, and sponsors an Open Translation Project to make the talks available in a wide variety of languages.

On Thursday morning, February 17, TED sponsored a panel discussion, hosted at IBM, on Watson’s success, and what he should do next if he wants an honest job.  (In this context, I find the anthropomorphic language impossible to resist.)  The discussion moderator was Stephen Baker, and the other participants were IBM’s principal scientist on Watson, Dr. David Ferrucci, IBM Fellow Kerrie Holley, and Prof. Herbert Chase from the Columbia University Medical Center, one of the participants in a test of using Watson as a medical diagnostic assistant.  The video presentation, which runs slightly more than half an hour, is archived here.

There were a couple of interesting things that came out in the discussion.  The first was the announcement that IBM has partnered with Nuance Communications to endow Watson with speech processing and recognition capabilities.  That will obviously make it more usable in a wider variety of contexts.

One claim, which has been circulated in various discussions on the Internet, is that Watson had an unfair advantage because it could press the Jeopardy! buzzer faster. As both Mr. Baker and Dr. Ferrucci point out, though, human contestants have a compensating advantage, because they can see the clue as it is read by Alex Trebek, and anticipate his finishing it.  (Contestants are not allowed to “buzz in” before that point.)  So it seems to me that neither Watson nor that human contestant has an advantage, on balance.

Another highlight, at least for me, was a question asked by Stephen Baker, which he says is asked all the time, “Does Watson think?”   Dr. Ferrucci’s answer to this question, besides being somewhat humorous, really pointed out how silly this question can be; he asked “Can submarines swim?”   The point, of course, is that in both cases, the answer to the question depends almost entirely on how we define the activity, a point I tried to make in a previous post.  Humans, of course, have a conscious experience of thinking, which Watson doesn’t have (although, as I also said before, I don’t believe any of us is sufficiently introspective to understand our thought processes completely, and certainly not all the way down to the “hardware”).  In any case, Watson’s real value is its ability to process a huge amount of data, expressed in natural language, and present a set of alternative answers, with its estimate of the likelihood of their being correct.  As Prof. Chase pointed out, this is just what is wanted by a physician doing differential diagnosis.

I do encourage you to have a look at the video.  Mr. Baker does a great job at moderating the discussion, and the other panelists all have interesting insights to contribute.

Watson Cleans Up

February 17, 2011

“I for one welcome our new computer overlords” — Ken Jennings

The contest between IBM’s Watson computer and human champions on Jeopardy! is over, and Watson won, big time.  In the three-day match, Watson ended up with a total of $77,147, Ken Jennings had $24,000, and Brad Rutter finished in third place with $21,000.

(These are not the actual prize amounts.  Watson will receive a $1 million prize, which IBM will donate to charity.  Jennings will receive $300,000, and Rutter $200,000.  Both human contestants will donate half their prize money to charity.)

Despite its very impressive overall performance, Watson did make some funny mistakes.   For example, in a category “US Cities”, the clue given was “Its largest airport is named for a World War II hero; its second largest, for a World War II battle.”  Both Jennings and Rutter got the correct answer, Chicago; Watson answered “What is Toronto?”  As Dr.Chris Welty of IBM explained in an article at Ars Technica, the error is not so silly as it appears, given the way Jeopardy! categories work.

“If you change the question to ‘This US City’s largest airport…’, Watson gets the right answer,” Welty said during a panel at Rensselaer Polytechnic Institute’s Experimental Media and Performing Arts Center. Welty pointed out that though categories in Jeopardy seem like they will have a set type of answers, they almost never do, and Watson was taught not to assume they would.

Still, this is real step forward in the quest for more intelligent machines.  One of the promising applications for Watson’s technology is to provide assistance in medical diagnosis.  A report from the Associated Press, via Yahoo!, says that two hospitals, the Columbia University Medical Center, and the University of Maryland School of Medicine, have signed up to test the technology, once it is adapted to work in a medical environment.

Update Thursday, 17 February, 18:05 EST

The New York Times also has an article on the match.

More on Watson’s Methods

February 6, 2011

Yesterday, I wrote about an article in the Wall Street Journal, by Yale computer science Professor David Gelernter, on the approach used by Watson, IBM’s computer system that plays Jeopardy!.   The Journal also has an excerpt from Stephen Baker’s new book, Final Jeopardy, which discusses some of the background of the project, and its early trials.  (I mentioned Mr. Baker’s book, and his blog, in an earlier post.)   As he reports, Watson is by no means infallible; the system sometimes misses an allusion that a person would catch, and sometimes it just comes up with wacky answers.  His account of some of Watson’s history and “growing pains” is also interesting.

The New York Times also has an article, by the novelist Richard Powers, on Watson.  Mr. Powers observes that, although the upcoming match between Watson and Jeopardy! champions Ken Jennings and Brad Rutter is undoubtedly a stunt designed to capture the public’s interest (it certainly seems to have captured the media’s), it is nonetheless work at the forefront of Artificial Intelligence research.

Open-domain question answering has long been one of the great holy grails of artificial intelligence. It is considerably harder to formalize than chess. It goes well beyond what search engines like Google do when they comb data for keywords. Google can give you 300,000 page matches for a search of the terms “greyhound,” “origin” and “African country,” which you can then comb through at your leisure to find what you need.

As I discussed in the post yesterday, Watson uses a massively parallel approach, on both the hardware and software levels, to try to come up with an answer.  The system decides whether to “buzz in” to answer based on a statistical measure of confidence that its answer is right.  Mr. Powers says that some people may discount this as a sort of gimmick.

This raises the question of whether Watson is really answering questions at all or is just noticing statistical correlations in vast amounts of data.

As he goes on to suggest, the question that is really raised is. what do we mean by really answering questions?   I don’t believe any of us is introspective enough to determine how our decision making process works, certainly not down to the “hardware” level.  Similar sorts of objections have been raised about the Turing test, proposed by the English mathematician Alan Turing as an operational test of a machine’s intelligence.  I have always found these objections to be vague in the extreme; if intelligence in playing Jeopardy! or the Turing test has some quality that transcends the ability to answer convincingly, I have never seen that quality described.

Regardless of whether Watson is a total flop, or wipes the floor with Ken and Brad, I think the project that built it can teach us a good deal about the problem of interpreting natural language.


%d bloggers like this: