Crowdsourcing Taxonomy

I had an interesting debate on twitter with Panos Ipeirotis and Anand Kulkarni about what counts as a crowdsourcing application.

Wikipeidia defines crowdsourcing as
"Crowdsourcing is the act of outsourcing tasks, traditionally performed by an employee or contractor, to an undefined, large group of people or community (a crowd), through an open call."
And they are even so nice as to give a list of crowdsourcing projects


These seem to fall into a few areas
1. Microwork. MTurk, Freelancer.com, 99designs. You go on mturk you put in a task like "tell me if this sentence is in English" and an amount "1 cent per task" and then you let the people on mturk do the classifications for you.

I think this sort of work will be huge. There are talented people sitting around willing to do something interesting for cash in their spare time just waiting to be harnessed.

Many projects exist to crowdsource advertising ideas. for example IdeaBounty. These advertising ideas seem similar to other microwork projects.

2. Games with a purpose. Galaxy finder, Foldit etc. There is a task like "count the number of spiral galaxies" you put this out there as a 'fun' task and people do it for you free, gratis and for nothing.

This is similar to citizen science projects but explicitly involves the use of fun and game mechanics to get people to do the work.

3. Market mechanisms could be thought of as a form of collective intelligence. In a market the competing forces of supply and demand result on a value being placed on an item. So for example on ebay many people have decided the value of collectible baseball cards. Or on intrade many have decided the probability Obama is to be reelected.

Is market formation a form of crowdsourcing? I think prediction markets are great and should be used more but their placing accurate values on the probability of an event is a byproduct of their action rather than the intention of those operating on it. People would rather if the odds were wrong so they could make money being right.

For some reason I don't think a prediction market is a crowdsourcing application and none are listed on the wikipedia list.

As an aside I read a brilliant blog post two days ago (that I now cannot find) that pointed out that ebay is essentially a storage device. You sell things they go into the ebay and when you want them back you go on the ebay and find the product again and buy it. Your storage costs are usually just the postage and packing charges. This is similar to the argument that there are two ways to produce cars. One is to build factories and hire people and such the other is to send boat loads of cows to Tokyo where magically you get boats filled with cars back in return.

4. Microlending
Kiva and other sites allow you to lend money to business. Say a guy want to plough my field. 10 people give you 5 dollars each so you can buy a plough. The field gets ploughed. They get 6 dollars back after harvest. This again does not seem like crowdsourcing of jobs as you are lending capital not labour.

Artists using a street performer protocol to pay to get an album made is something similar. Kickstarter is one site that does this

5. Wikies, Forums, social search engines, message boards
People on Forums, give their advise and knowledge almost always for free. It is pretty amazing when you think about it. The fact that some very clever and skilled individual is willing to take their time to tell me the arguments to a Java function are wrong is amazing. The wikipedia page does not class these as crowdsourcing but again I cannot see why not.

6. Competitions, kaggle, Netflix, DARPA, Goldcorp and InnoCentive regularly have prizes where they put out a task such as 'improve on our recomendations' or 'make a car that drives itself' and people try to do the best they can at it. I think prizes as a way to encourage innovation are also going to be massively important in the future.

7. Response to events. The guardian set up a webpage to get people to read through all the mp expenses reports to find things that looked excessive and wikileaks have used crowdsourcing to examine the millions of leaked documents and highlight the most interesting.

8. Crime detection. Crowdsourcing has been used for illegal immigrant spotting in Texas and other similar uses.

9. Missing persons. After hurricane Katrina and Steve Fossett's loss crowdsourcing was used to locate missing people.

10. Politics. Oxfam Novib, Moveon.org and other such sites try to use crowdsourcing to create a community based political movement. If these are counted as crowdsourcing ventures I do not see why hobby sites that try promote brewing or knitting are not.

11. Art. Improv everywhere, mechanical olympics, Flash mobs are all listed as crowdsourcing projects.

12. Citizen science. Digitized versions of old weather or other science reports are made available to people to increase our knowledge of past events. Even the open data movement must have some basis in crowdsourcing. If people are not expected to analyse the data there is not much point making it availible to them.

13 Cartography. Open street map, Waze and other projects hope to use peoples GPS systems to build up a map of the environment. This can include real time traffic data.


Reading through wikipedias list of crowdsourcing projects these are the types of projects that occured to me. Most classes are probably wrong and need to be split or combined. Maybe the hierarchy needs to be different. Please comment or write a rebuttal to if you are interested in what projects count as crowdsourcing.

A failed Mturk translation test

The Mturk is Amazon's platform where you can put up jobs for people to do. These tend to be things like translation or image categorisation that humans are good at but computers are not.

According to Panos Ipeirotis' research mturk workers seem to be highly educated. The majority having degrees and about 10% having postgraduate degrees. Much mturk work is boring and probably does not use the skills these well educated workers have.

A few days ago google announced sponsorship for projects they think are carrying out good work. One winner was

"The Khan Academy will receive $2 million toward funding its work on the "make educational content available online for free" theme. The academy does just that, with a library of over 1,800 videos with lessons on math, science, finance, and history.

Bill Gates is a big fan of the Khan academy "This guy is amazing," he wrote. "It is awesome how much he has done with very little in the way of resources."
.

I am also a huge fan of the Khan academy I think that these videos and other online education videos such as MIT's online courses have amazing potential to transform the education of millions of people.

How far could googles 2 million grant stretch? Obviously it is up to Mr Khan to decide how to use his resources but I thought it would be interesting to see if the mturk could be used to translate one of his mathematics videos. The languages with over 100 million speakers are Mandarin, Spanish, English, Hindi, Bengali, Cantonese, Arabic,Portuguese, Russian and Japanese. 10 languages for 2000 videos would be 20,000 video translations. If each video cost 100 dollars to translate that would spend the 2 million dollars google donated. If this cost can be reduced more languages could be added.

A mathematics video is not something the average person can translate. However we know a large number of turkers have degrees and many are from India where they would likely have an understanding of English and Hindi or Bengali.

I tried an experiment to see if I could get one of Khan's videos translated into Hindi using the mturk at a cost of 5 dollars. Unfortunately I failed. The first person who accepted the task dictated what Khan said into text. Which is useful but not what I was looking for. The second person posted up another video on Solving linear inequalities in English not a translation of Khan's video.



This small experiment tells me that you need to be very clear on the mturk how you ask for a task to be completed. It also says that it might be worthwhile once you find someone who understands and completes the task to encourage them to translate other videos rather than rely on the vagaries of who happens to accept your mturk task.

This experiment ignored the problem of copyright. Khan owns his videos and it is unfair for someone to come along and copy him. I was not trying to steal any glory from Mr Khan with this experiment just to see if the mturk could be used to translate his videos.

Other people have successfuly used mturk to reduce the cost of translations. 'How I reduced translation costs of 200 articles from $9000 to $46' is an interesting article on one successful usage. This tells me that the problem was more likely with my unclear instructions than with the mturk platform. You can even monitor translations taking place in the mturk here so I still think this method would be cost effective. However my simple experiment failed.