Watson on Jeopardy

[Meadbh]

Did anyone watch Jeopardy this evening? Normally I would never spend time watching a game show, but I was intrigued by the scientific experiment. "Watson" is the name of a supercomputer that IBM has been working on for several years. IBM approached the Jeopardy team with the proposal in 2007; they wanted to build a computer that could search knowledge, judge the probability of a correct answer, and make a decision to volunteer an answer based on reaching a critical confidence level of being right. That's mostly data mining and reporting, but the neat thing is the interpretation of language cues, which are sometimes pretty complex in Jeopardy clues.

So Watson is ready, and is competing with the two top human Jeopardy champions of all time: the guy who kept the crown for 74 shows, and the guy who won the most money. Part 1 was tonight. Parts 2 and 3 are being broadcast tomorrow and the day after.

Watson (represented on the set by an avatar) gets the questions as text files. It started off in the lead. But some of the questions really threw it off and at the end of tonight's round it was in a dead heat with a human. IT was "sweating" too! Viewers could see Watson's top ranked answers and what I saw suggests to me that the algorithms on popular culture are the strongest - perhaps reflecting the age group of the engineers designing Watson! I've also heard a radio piece that suggested that Watson has demonstrated the ability to learn from the answers of the other Jeopardy players.

I got annoyed that Alex Trebek was referring to Watson as "he"; Watson's "voice" is male but that was an arbitrary human decision.

Anyhow, I find artificial intelligence fascinating.

 

[WhoDaresWins]

I watch Jeopardy most evenings, but I made a point of viewing last night. I think that language and AI are interesting, not that I know much about these fields. I saw a CMU professor interviewed last week on a Pittsburgh TV station program "Night Talk" as he and some of his grad students also worked on the Watson project. I am rooting for one of the humans to win. It seems that Watson might lose ground on nuance but is very fast with the answers. I am presuming that since Watson would most often be a male given name, the host is saying "he". I believe Thomas Watson was the founder of IBM if I am not mistaken.
I plan to tune in again tonight.

 

[clifp]

Darn I've been hearing about this match for months and seen several programs about it, NOVA, Newshour etc. and I missed it...

I couldn't find the Day 1 anywhere online. Anybody found a link for it.

 

[Rustward]

I got annoyed that Alex Trebek was referring to Watson as "he"; Watson's "voice" is male but that was an arbitrary human decision.

Anyhow, I find artificial intelligence fascinating.

Could it be that this project is named after the real Watson who was a "he"?

 

[ERD50]

Anybody found a link for it.

I missed also, should get to see it today. Would love a link to the 1st episode.

I agree with OP, the understanding of the clue is the hard thing. Words can have such twisted meanings. Trying to figure out what they are getting at isn't a very direct kind of thing. I think the 'response' is the easy part (for a computer' - figuring out the clue is the hard part.

I was watching when Ken Jennings was on his streak. One thing I noticed is he seemed to 'size up' the clue, and sometimes buzz in as soon as he could, even if he hadn't figured out the response yet. Buzz in, then figure it out. And I could sorta, kinda do it on occasion if I tried (and got lucky!) Sometimes, you can get a pretty good feeling, just skimming the terms quickly that "I know this!", but it takes another second or two to actually formulate and phrase the 'response' correctly.

The computer can't do that, it needs fully understand the clue, evaluate responses and weigh them, but the faster the computer is the less that human advantage is. The language recognition part is just amazing though.

-ERD50

 

[grumpy]

I often watch Jeopardy and was watching last night when Watson was competing. Very interesting. One thing I noticed was that Watson seemed to beat the humans on buzzing in, not that the humans didn't know the "questions", just that they didn't get the chance to answer. It seemed that the human reaction time couldn't compete with the machine's.

I wonder what the outcome would be if the contestants were allowed to buzz in BEFORE the question reading ended. I suspect that the humans would sometimes be able to make an intuitive leap to the correct question before Watson could complete "his" algorithm process. As it is, Watson may already be well into his "thought" process before the reading of the question ends.

I wonder if the programmers were embarrassed when one of the humans gave the wrong answer and then Watson buzzed in and gave the same wrong answer. An obvious oversight in their logic.

 

[ERD50]

I wonder if the programmers were embarrassed when one of the humans gave the wrong answer and then Watson buzzed in and gave the same wrong answer. An obvious oversight in their logic.

I laughed when I read about that. That's a mistake that only a bad player would make. Does it mean the programmers never considered their machine wouldn't buzz in first, or that it couldn't learn from others? Says something about the programmers.

I also wonder if the timing of the buzzer is 'fair'. A machine has some advantage here. I never really understood exactly how that works on Jeopardy. Do they have to wait for Alex to stop talking to buzz in? Is there a light or something? I know they get locked out for a few seconds or something if they buzz in too early. A human might make that reaction mistake, but a machine wouldn't.

-ERD50

 

[Nodak]

Last week there was an extremely interesting NOVA program on PBS about the design and programming of Watson. If you can find a re-run of it it's well worth watching.

 

[walkinwood]

I thought that the computer was going to "listen" or "read" to the clue. It is still impressive, but would be so much better if it got the clue in the same way the other players did.

Thanks for the heads up. Will record it tonight.

 

[glippy]

Me and my fellow nerd friends are having a party on Wednesday for the final episode. I've followed advancements in Artificial Intelligence and Machine Learning for years and what IBM has achieved with Watson is nothing short of a breakthrough.

We've got a pool going, I'm the only one betting on the machine.

I lot of good can come out of the advancements IBM is making here. Imagine a program with a google like, text-input interface where you can type in a natural language question and get the answer you're looking for 9 out of 10 times. - Or the help it could provide to medical doctors in making a diagnosis.

The change from simply a human putting in a thousands of custom rules to the computer being able to figure out, on its own, what rules to follow in order to succeed makes the "intelligence" and "knowledge" of the computer grow exponentially faster than if people had to sit there and punch in every single rule.

@ERD50
Does it mean the programmers never considered their machine wouldn't buzz in first, or that it couldn't learn from others? Says something about the programmers.

They considered the scenario, and it happened in practice sessions. But Watson doesn't have a microphone or voice recognition software. If he did, they could avoid this problem. Not sure why they didn't bother. Probably a matter of too much to do and not enough time to do it in.

 

[ziggy29]

I lot of good can come out of the advancements IBM is making here. Imagine a program with a google like, text-input interface where you can type in a natural language question and get the answer you're looking for 9 out of 10 times. - Or the help it could provide to medical doctors in making a diagnosis.

Or a machine that can "think" more cheaply and efficiently than human labor, and thus throwing even more people out of work.

I'm not advocating against this; I'm certainly no Luddite. I'm just saying that technology has a high social price tag to it when it comes to making labor unnecessary. And too many of the new jobs that develop this technology that's throwing our people out of work are being created in places like China and India.

 

[glippy]

@ziggy29

I agree. The more adept robots and artificial intelligence gets, the more we'll have to think of an alternative to an economy where we sell our labor.

 

[HFWR]

@ziggy29

I agree. The more adept robots and artificial intelligence gets, the more we'll have to think of an alternative to an economy where we sell our labor.

I'm working on that alternative as we speak...

 

[Midpack]

Last week there was an extremely interesting NOVA program on PBS about the design and programming of Watson. If you can find a re-run of it it's well worth watching.

I don't watch Jeopardy, but I caught the NOVA episode and was so intrigued that I DVRd the Mon, Tue & Wed episodes. Watched the first last night and it was fascinating IMO. Watson got off to an incredible start and it looked "he" was going to embarass the two champs, but the first episode ended with all three in the money. I am looking forward to the next two episodes and I agree you will better appreciate the AI challenge if you see the NOVA episode. Arming Watson with lots of answers turned out to be the easy part, and some of the earlier versions were amusing .

 

[braumeister]

I also wonder if the timing of the buzzer is 'fair'. A machine has some advantage here. I never really understood exactly how that works on Jeopardy. Do they have to wait for Alex to stop talking to buzz in? Is there a light or something? I know they get locked out for a few seconds or something if they buzz in too early. A human might make that reaction mistake, but a machine wouldn't.

I've taken the in-person audition twice, and believe me there is a real art to using the buzzer. It becomes activated as soon as Alex stops talking, but there is a person backstage listening to him, and the backstage person activates all the contestants' buzzers by pressing another buzzer.

If a contestant buzzes too soon, he is locked out for a brief time (they were a little unclear on exactly how long, but I'm pretty sure it's not over a second).

You definitely want to be the first, because you'll still have a split second to keep trying to come up with the answer, and that was Ken Jennings' real strength. He's a virtuoso on the buzzer.

 

[Arnie]

I am rooting for the humans. Can you spell skynet?

 

[clifp]

I laughed when I read about that. That's a mistake that only a bad player would make. Does it mean the programmers never considered their machine wouldn't buzz in first, or that it couldn't learn from others? Says something about the programmers.

I also wonder if the timing of the buzzer is 'fair'. A machine has some advantage here. I never really understood exactly how that works on Jeopardy. Do they have to wait for Alex to stop talking to buzz in? Is there a light or something? I know they get locked out for a few seconds or something if they buzz in too early. A human might make that reaction mistake, but a machine wouldn't.

-ERD50

There is a red light behind Alex that turns of when he has finished reading the question. At the point you can buzz in, if you do so before the light goes off you are penalized.

Evidently Watson is feed the text of the question (which is some what cheating since voice recognition is to me part of AI) but doesn't hear the other contestants answers.

 

[clifp]

I am rooting for the humans. Can you spell skynet?

I for one welcome our new silicon-based overlords.
You will be assimilated resistance is futile.

 

[BigNick]

I wonder if the programmers were embarrassed when one of the humans gave the wrong answer and then Watson buzzed in and gave the same wrong answer. An obvious oversight in their logic.

I laughed when I read about that. That's a mistake that only a bad player would make. Does it mean the programmers never considered their machine wouldn't buzz in first, or that it couldn't learn from others? Says something about the programmers.

I find it slightly strange to see a comment "Says something about the programmers". Are you seriously imagining that they overlooked this? We're dealing here with some of the finest minds in artificial intelligence, with a very large budget. This is not just a bunch of college freshmen on a 100-line Pascal assignment.

There's a much simpler explanation for this error. Watson receives the questions by text input, not by listening to what others say. (I presume that the question is pre-typed, and someone presses "Enter" when Alex stops or starts speaking.) Listening to the voices of the contestants or host is not in the system's specification. And AFAIK the operators don't (or don't have time to) input what the other contestants are saying.

 

[GregLee]

There's a much simpler explanation for this error. Watson receives the questions by text input, not by listening to what others say. (I presume that the question is pre-typed, and someone presses "Enter" when Alex stops or starts speaking.) Listening to the voices of the contestants or host is not in the system's specification. And AFAIK the operators don't (or don't have time to) input what the other contestants are saying.

Yes, and it's not exactly an "error". AI is often construed as an enterprise to understand human intelligence, so when the machine makes exactly the same error as the human, that's a triumph. It shows that the machine reasons like the human.

 

[clifp]

Yes, and it's not exactly an "error". AI is often construed as an enterprise to understand human intelligence, so when the machine makes exactly the same error as the human, that's a triumph. It shows that the machine reasons like the human.

I agree. I often find myself thinking the same incorrect answer as a contestant. Jeopardy questions are deliberately tricky.

 

[clifp]

I just watched Watson on Jeopardy, my god it killed Jennings and Brad's clock.

Only to blow it big time on a pretty easy final Jeopardy question.

 

[ERD50]

There is a red light behind Alex that turns of when he has finished reading the question. At the point you can buzz in, if you do so before the light goes off you are penalized.

Thanks, I recall trying to find that out a while back (probably when Ken Jennings was on his streak), and I never found anything specific.

I find it slightly strange to see a comment "Says something about the programmers". Are you seriously imagining that they overlooked this? We're dealing here with some of the finest minds in artificial intelligence, with a very large budget. This is not just a bunch of college freshmen on a 100-line Pascal assignment.

I was assuming someone could text Watson the [-]answers[/-] 'responses' from the other contestants - why not? If he's getting the text of the 'clue', why not the 'responses'? It is totally do-able, and it would add value. Edison famously knew the value of mistakes (" I've found 10,000 ways that don't work"). Why wouldn't they do this? What's wrong about thinking they should?

Of course the programmers are brilliant. I didn't mean to imply they weren't. What I was wondering was, were they too 'cocky' to think they would need this input from the others? That could say something about them.


Now I'm curious as to the timing of this text of the clue. It doesn't seem fair to send it all at once at the start, of course a computer can 'read' (but not comprehend) the words almost instantaneously. Sure, the humans see the words all at once (and I assume they ignore the spoken words), but it takes some time to read them, and then comprehend them. Maybe they should text the words at the rate of human reading? Maybe they do?

Voice recognition at this speed would be the next very impressive step. I saw the show today, and Watson was amazing. Interesting that he missed the one about the period of art the stolen paintings were from - that really was a very twisted wording - plenty of ways to misinterpret and think of the museum name, or the artist (which is what Watson did), or something. Was surprised it missed the missing art from Baghdad though - I recall reading about that, and the clue was fairly straight wording.

I just watched Watson on Jeopardy, my god it killed Jennings and Brad's clock.

Only to blow it big time on a pretty easy final Jeopardy question.

Heh, heh - I missed it too, and I'll blame it on being from Chicago-land. I kept thinking "Orchard Airport" (the original name and source of the ORD assignment). Couldn't think of any WWII heroes named "Orchard", so I moved on to Milwaukee, pretty sure that Mitchell was a WWII guy (ooops, wrong again - WWI). One of the reasons you've never seen me on Jeopardy!

-ERD50

 

[clifp]

Heh, heh - I missed it too, and I'll blame it on being from Chicago-land. I kept thinking "Orchard Airport" (the original name and source of the ORD assignment). Couldn't think of any WWII heroes named "Orchard", so I moved on to Milwaukee, pretty sure that Mitchell was a WWII guy (ooops, wrong again - WWI). One of the reasons you've never seen me on Jeopardy!

-ERD50

Hey I got it immediately O'Hare and Midway. I always wondered about the destination. NYC was possibility since JFK was WWII hero (before a president) but LaGuardia is just a mayor.

Reading the IBM blog I guess Watson didn't understand that Toronto isn't a USA city.

 

[clifp]

One more Watson thing if you haven't read Ken Jenning's blog on the Washington Post.
Do so it is laugh out loud funny.

Q.
The crowd
Was there anybody in the crowd cheering for you or were they all Watson supporters?

A.
Ken Jennings :
It was an all-IBM crowd: programmers, executives. Stockholders all! They wanted human blood. It was gladiatorial out there. The stage had a big Watson logo on it too. This was definitely an away game for humanity.

Q.
Admin note
Can the Washington Post software engineers reverse this so we readers provide the answers and Jennings provides the questions? It seems appropriate.
A.
Ken Jennings :
What is "Congratulations, you could be a drive time DJ!" Every radio personality in America has had this idea every time they try to interview me. In practice, it doesn't work so hot.