The "Dumbness of Crowds"

Community. Wisdom of Crowds. Collective Intelligence. The new emphasis on net-enabled collaboration is all goodness and light until somebody gets an eye I poked out. Is it merely a coincidence that Apple, run by (as James Gosling put it) "a dictator with good taste" leads the way in tech design, while risk-averse companies using design-by-committee (or consensus) are churning out bland, me-too, incremental tweaks to existing products? And if that's true about companies, why do we think consensus will work on an even larger scale with "users" in Web 2.0?

Jaron Lanier, in his controversial Edge essay Digital Maoism, has a great quote:

"In the last year or two the trend has been to remove the scent of people..."

All geeks-and-personal-hygiene-jokes aside, we need the smell. And the most frustrating part for me is how the "Wisdom of Crowds" idea has been twisted and abused to mean virtually the opposite of what New Yorker columnist James Surowiecki says in the book of the same name. He opened a talk at ETech telling us that while ants become smarter as the number of collaborators increases, humans become dumber. In what is potentially the most misleading book/idea title in the history of the world, the "Crowds" in "The Wisdom of Crowds" was never meant to mean "mobs", "groups acting as one", "committees", "consensus" or even "high collaboration".

By "crowd,", I think he meant "more people", sure, but he also defined a big ol' set of constraints for how much togetherness people can have before the results became dumber. And it turns out, not that much. By "crowd", he was referring to a collection of individuals. Individuals whose independent knowledge (and "independent" is a key word in what makes the crowd "smart") is aggregated in some way, not smushed into one amorphous Consensus Result.

Web 2.0 and putting the Community in Control
One of the high-profile concepts of the Web 2.0 meme is community. Giving community the control. Letting the community make decisions. Trusting the community. And--if you're a lucky bubble-2.0er--letting the community do all the work while you collect the money. But this idea of consensus-community is not at all what I've heard Tim O'Reilly talk about when he uses the phrase, "harnessing collective intelligence" or when he describes Web 2.0 as something whose value to users grows with the number of users.

What's the difference between Collective Intelligence and Dumbness of Crowds? A few examples:

"Collective intelligence" is a pile of people writing Amazon book reviews.

"Dumbness of Crowds" is a pile of people collaborating on a wiki to collectively author a book.
(Not that there aren't exceptions, but that's just what they are--rare exceptions for things like reference books. I'm extremely skeptical that a group will produce even a remotely decent novel, for example. Most fiction suffers even with just two authors.)

"Collective Intelligence" is all the photos on Flickr, taken by individuals on their own, and the new ideas created from that pool of photos (and the API).

"Dumbness of Crowds" is expecting a group of people to create and edit a photo together.

"Collective Intelligence" is about getting input and ideas from many different people and perspectives.

"Dumbness of Crowds" is blindly averaging the input of many different people, and expecting a breakthrough.
(It's not always the averaging that's the problem it's the blindly part)

"Collective Intelligence" is about the community on Threadless, voting and discussing t-shirts designed by individuals.

"Dumbness of Crowds" would be expecting the Threadless community to actually design the t-shirts together as a group.

Art isn't made by committee.

Great design isn't made by consensus.

True wisdom isn't captured from a crowd.

At least not when the crowd is acting as a single entity. Clearly there IS wisdom in the many as long as you don't "poison" the crowd by forcing them to agree (voting doesn't mean agreeing). According to Surowiecki, even just sharing too much of your own specialized knowledge with others in the group is enough to taint the wisdom and dumb-down the group.

It's the sharp edges, gaps, and differences in individual knowledge that make the wisdom of crowds work, yet the trendy (and misinterpreted) vision of Web 2.0 is just the opposite--get us all collborating and communicating and conversing all together as one big happy collborating, communicating, conversing thing until our individual differences become superficial.

Imagine a community--let's say the Dog Lover's Society--that through a genetic breakthrough is given the chance to design the perfect dog. Everyone gets to contribute. Everyone's idea counts. The dog will be the perfect reflection of the wisdom of the dog-loving crowd. What will they come up with?

I see two options:
1) A non-descript Generic Dog--the average of every possible dog attribute. It would look something like the abstract DOG used in pre-school books where you teach two-year olds to "point to the DOGGY"
OR
2) Frankendog--a hideous patchwork of dog parts that were never meant to go together. It would look something like a Star Trek transporter accident.

Of course most of what I've been dissing is the popular, rampant misinterpretation of Wisdom of Crowds, not what Surowiecki actually meant. Read the book and you'll see just how significant and powerful the aggregation of individual knowledge really is, and how in the right circumstances with the right constraints, the wisdom found in that group CAN be smarter than the smartest individual in the group. But he never says the group itself becomes smarter when they work together to produce a result as a group.

20Q - a perfect example of the real Wisdom of Crowds

If you're one of the twelve people who haven't yet played with the 20Q "toy", you have no idea how scarily well this thing "guesses" what you're thinking about. The creepy thing isn't necessarily that it figures out you were thinking of thermometer, bra, microscope, painting, mp3 player, or lightbulb (all things it guessed correctly for me yesterday). The really creepy thing is how it got there from the questions it asked. Although the program clearly changes questions based on your answer to the previous questions, it doesn't change them nearly as much as you'd think it would need to.

If I didn't know better, I'd swear it's using voice recognition to cheat. But no, it turns out there's a perfect explanation for its supernatural accuracy. The creator harnessed collective intelligence. Hundreds of thousands of people "taught" the program over a period of years, by playing software versions of his game. The program uses a neural net, and learned. It learned so well, in fact, that it learned a few dumb things. For example, way too many people think a dolphin is a fish, so even if you say "yes" when asked by the device if what you're thinking of is a fish, it can still figure out you mean dolphin from other cues. (Apparently it also thinks human beings may not actually be animals, based on the "collective intelligence" of those who've played the online version.)

Finally, while I disagree with much of what Jaron talks about in his essay, I know how damn smart this guy is (we were on a panel together a long time ago at the Junos in Canada). A few of my favorite quotes:
"Meanwhile, an individual best achieves optimal stupidity on those rare occasions when one is both given substantial powers and insulated from the results of his or her actions.
"If the above criteria have any merit, then there is an unfortunate convergence. The setup for the most stupid collective is also the setup for the most stupid individuals.
"Every authentic example of collective intelligence that I am aware of also shows how that collective was guided or inspired by well-meaning individuals. These people focused the collective and in some cases also corrected for some of the common hive mind failure modes."

No matter what, I believe that in our quest to exploit the "We" in Web, we must not sacrifice the "I" in Internet.