16.2. Planned Crowd Work Examples#

Let’s now consider some examples of planned crowd work, meaning a system or task was intentionally created and given to a crowd to work on.

16.2.1. Crowd Work Platforms#

Some online platforms are specifically created for crowd work. For example:

  • Wikipedia: Is an online encyclopedia whose content is crowdsourced. Anyone can contribute, just go to an unlocked Wikipedia page and press the edit button. Institutions don’t get special permissions (e.g., it was a scandal when US congressional staff edited Wikipedia pages), and the expectation that editors do not have outside institutional support is intended to encourage more people to contribute.

  • Quora: An crowdsourced question and answer site.

  • Stack Overflow: A crowdsourced question and answer site specifically for programming questions.

  • Amazon Mechanical Turk: A site when you can pay for crowdsourcing small tasks (e.g., pay a small amount for each task, and then let a crowd of people choose to do the tasks and get paid).

  • Upwork: A site that lets people find and contract work with freelancers (generally larger and more specialized tasks than Amazon Mechanical Turk.

16.2.2. Example Crowd Work Tasks#

You probably already have some ideas of how crowds can work together on things like editing articles on a site like Wikipedia or answer questions on a site like Quora, but let’s look at some other examples of how crowds can work together.

Fold-It is a game that lets players attempt to fold proteins. At the time, researchers were having trouble getting computers to do this task for complex proteins, so they made a game for humans to try it. Researchers analyzed the best players results for their research and were able to publish scientific discoveries based on the contributions of players.

Game screen with some text boxes and buttons around the side, and in the middle a twisty blob of protein pieces that have been folded.

Fig. 16.1 Screenshot of the fold-it game.#

A research study demonstrated the power of crowd work by having Mechanical Turk workers build off of the work done by previous workers. To demonstrate, they wrote a note with intentionally bad and almost undecipherable handwriting:

A four line long handwritten note. The handwriting is very messy with letters overlapping each other and letters not fully drawn or scribbled.

Fig. 16.2 A note written with intentionally bad handwriting.#

Turkers (the people who do Mechanical Turk tasks) were then given the handwritten note and after the first few attempts at deciphering it, Turkers were either a previous attempt at deciphering the note, or asked to vote on which interpretations were improvements. They were instructed to leave parentheses around sections they weren’t sure about. Here is a selection of subsequent attempts at interpreting the note (from the paper):

version 1:

  • You (?) (?) (?) (work). (?) (?) (?) work (not) (time). I (?) (?) a few grammatical mistakes. Overall your writing style is a bit too (phoney). You do (?) have good (points), but they got lost amidst the (writing). (signature)

version 4:

  • You (misspelled) (several) (words). (?) (?) (?) work next (time). I also notice a few grammatical mistakes. …

version 5:

  • You (misspelled) (several) (words). (Plan?) (spellcheck) (your) work next time. I also notice a few grammatical mistakes. Overall your writing style is a bit too phoney. You do make some good (points), but they got lost amidst the (writing). (signature)

version 6:

  • You (misspelled) (several) (words). Please spellcheck your work next time. I also notice a few grammatical mistakes. Overall your writing style is a bit too phoney . You do make some good (points), but they got lost amidst the (writing). (signature)