Wednesday, February 16, 2011

When will you be able to ask Watson?

Watson is a computer currently playing the quiz Jeopardy. You can see a video of this here


One useful question is how long this sort of supercomputer like power will take to be seen in everyday life? It has 3000 cores running and answers in less than three seconds. On one machine would take 2 hours to answer a question according to here
So how long to get a desktop computer to be able to answer at this speed? I propose that half the speed improvement in Watson like programs will be from Moore's law and half from algorithm improvements. To go from 7200 seconds to say 2 seconds will require about 13 halvings in speed. I predict this will occur in the time it take processors to improve 7 doublings in computation. You can buy your own supercomputer now on the cloud and depending on how long you are willing to wait you could get an answer to a Watson like answer for costs of around a dollar (if you had the data and software Watson uses).

One comparison here is with chess. "It has been estimated that doubling the computer speed gains approximately fifty to seventy Elo points in playing strength (Levy & Newborn 1991:192)."

If doubling in processing power happens ever two years that would imply about a 30 point increase a year. The actual improvement (in computer v computer games) is described as 'With a 40-point annual improvement due to hardware upgrades, and a 30-point annual improvement due to software upgrades'. Implying improved algorithms are responsible for half the improvement.

This argument is summed up in this explanation as to why a fairly normal computer Deep Fritz in 2002 was an improvement over the supercomputer Deep Blue in 1997 'Deep Fritz has improved considerably over Deep Blue. Despite Deep Fritz having available only about 1.3% as much brute force computation, it plays chess at about the same level because of its superior pattern-recognition-based pruning algorithm'

I claim there is an analogy to Moore's law that says something like.

Once computers get good enough at a task to have a flashy TV challenge from then on algorithms will cause half the improvement in that task

So if I am right I think Watson like computation instead of talking over 25 years to move from twelve refrigerators to something we can use everyday should take less than 10 years. I would take an even money bet that in five years there will be a service you can use from your phone that is a lot closer to Watson than to a search engine we see today.

*update A friend just made this bet with me
"20 euro bet that January 2015 there is a search engine that with a set of quiz questions from one gameshow program (the weakest link, who want to be a millionaire, university challenge) and it has to get 7/10 right". The effects of learning by reading (and autonomous cars) are examined in this rather long but brilliant blogpost

15 comments:

  1. Totally not the point of your post, but... surely Watson's abilities are largely down to his education. If he has total textual recall of the millions of documents fed to him then the fact of his beating humans at a trivia contest is not much more impressive than if he beat blackjack players by counting cards.

    ReplyDelete
  2. Its worth looking at the sort of questions jeopardy asks. There things like 'clothing a young girl might wear on an operatic ship'

    It is the sort of thing you could answer. And almost all questions a person with wikipedia and a good ability to search could answer in under 5 minutes.

    Most of the questions are not relatively simple "name the capital of France" questions. and even if they were the ability to do direct general knowledge answers would still be an improvement to google.

    BTW answer is "what is a Pinafore?"

    ReplyDelete
  3. Actually i just googled "Capital of France" and it gave me the correct answer as did wolfram Alpha

    ReplyDelete
  4. I know what you mean, there just didn't seem to be enough of those in that clip. I mean, the ones that mention specific fictional characters, or sporting events, or song lyrics: find the piece of information that's missing from the statement (title of the book; location of the olympics) and it's not a big leap, algorithmically speaking. No?

    ReplyDelete
  5. Google also knows the county town of Offaly and the largest city in Franconia.

    ReplyDelete
  6. There are all sorts of claims that Watson like information retrieval will be used in Medicine and Law. I have trouble seeing how though.

    It can currently answer at about the level a searcher could in five minutes. Which makes sense for a search engine as 5 minutes saved with a fair chance of failure is a big deal in search. Also no one is going to die if it answer that the capital of France is Offaly.

    But I don't see it yet getting rid of the long and weird medical/law searches that take an expert days to do.

    ReplyDelete
  7. Hooray! The twenty minutes I wasted at work looking at this are now professionally justified!

    ReplyDelete
  8. I'm the friend.... Essentially I agree with most of what Dave is saying. The place where I disagree is that Watson is currently answering one question with its supercomputing power. While algorithms and computers will increase to allow such a program to run on a reasomnable server by 2015, I don't believe that they will be able to scale up to the load that such a service will expect, i.e. ansewering a million of these a second via a mobile app will not be feasible by Jan 1st 2015. However this is one of those bets that I hope I will lose ;)

    ReplyDelete
  9. I'll also be joining in on said bet. Pretty much agree with Mick's logic - while watson will run on a smaller hardware set by 2015, scaling 'a la' google will not be realistic within the same timeframe..

    ReplyDelete
  10. http://geekandpoke.typepad.com/geekandpoke/2011/03/post-watson-jeopardy.html

    ReplyDelete
  11. Ha! that is brilliant Julie

    ReplyDelete
  12. I'm game. Bring on the pints :P

    ReplyDelete
  13. Im confident that with the proper linguistic models, and some of the (semantic, etc) graph-based "knowledge engines" being developed by the likes of Google, Wolfram (and if Apple architect tie-ins with Siri), then it may be way less than 5 years.

    ReplyDelete
  14. Im confident that with the proper linguistic models, and some of the (semantic, etc) graph-based "knowledge engines" being developed by the likes of Google, Wolfram (and if Apple architect tie-ins with Siri), then it may be way less than 5 years.

    ReplyDelete
  15. Ok Owen you are signed up for the there will be a public search engine that gets 7/10 on a quiz show in 2015 side.

    Siri doesn't do this yet I have tried it.

    ReplyDelete