« Blow your mind in Z-space | Main | We joined the 9rules family »

Listening to users considered harmful?

Listeningtousers

Is "listening to users" really the most important way to keep them happy and -- if we're lucky -- passionate? Is giving users what they ask for the best way to help them kick ass? Or should you create or modify a product based solely on what you believe in... even if it doesn't match what users tell you?

Last weekend I attended the sold-out Parelli Natural Horsemanship conference. I was surrounded by 2000 passionate fans (at least 75% of the people were wearing at least one Parelli-branded shirt, jacket, or hat). The conference was amazing (more on that in another post), but the real reason I went was to interview the founder/visionary Pat Parelli, for the Creating Passionate Users book. His hugely successful, multi-million dollar company is one of the few we've found that does virtually everything on our "reverse-engineering passion" checklists, without having first waited for the fans to do it themselves.

In the equestrian world (total annual impact of the horse industry on the US economy is $112 billion [yes, that's with a "b"]), Pat Parelli has so greatly outstripped the "horsemanship" competition that it doesn't even make sense to talk about competition. Software engineers will appreciate that horse training doesn't scale. So Parelli decided to teach others to do what he does, and of course sell those folks a ton of high-end equipment and training products to help them do it. Nobody -- absolutely no other individual "horse whisperer" or company -- comes anywhere near Parelli in size and scope of the business - Parelli has two training centers, one in Colorado and one in Florida (combined over 700 acres for the facilities), and hundreds of thousands of participants in the home-study programs, clinics, and club membership. Their Parelli Horseman's University is one of the only state-accredited "natural horsemanship" programs in the US.

So that's the backstory. I have weeks' worth of posts to make on what I learned from Pat about the ways in which they've become such a passionate user success story, but today's post is about something I had completely wrong when I interviewed him:

Me: "So, you've recently made drastic changes to your program--a program that was already extremely successful. It's obvious that you've been really listening to your members and taking their feedback and using that to make these sweeping changes."

Pat: "No, listening to our members was maybe 20% of it, but the other 80% was something else."

And then he said it:

"We changed our entire program because WE knew we could do better. Because WE were still frustrated that people weren't learning quickly enough or progressing through the higher levels as well as we thought they could. People still weren't having the kind of relationship with their horse that we knew they could have, even though our students were delighted with the progress they were making. So we changed it all."

It turned out that most of the major changes they made to their program came not from user requests and suggestions, but from the Parelli team's own innovations. He went on to explain that their members/students/users had no idea what was needed to make better, faster, deeper breakthroughs. In fact, many of the changes went against what their user feedback seemed to suggest. In other words, in many ways the Parelli team deliberately did not listen to users.

They trusted themselves, and did what they believed was right for their users, even if it meant doing things that on the surface seemed even less user-friendly.

Most of us realize that focus groups are notoriously ineffective for many things, but we still assume that listening to real feedback from real users is the best way to drive new products and services, as well as improve on what we have. But there's a huge problem with that -- people don't necessarily know how to ask for something they've never conceived of! Most people make suggestions based entirely around incremental improvements, looking at what exists and thinking about how it could be better. But that's quite different from having a vision for something profoundly new.

True innovation will rarely come from what users say directly.

This doesn't mean that you don't listen to users--because the truth is embedded in what they say...but you have to look for the deeper meaning behind what they ask for, rather than always taking them at their word. If they ask for "D", as an improvement to "C", you might have to dig deeper to find out what it is about "D" that they want. And in that answer, you might find the nugget that leads you--and only you--to come up with "S" as a solution. And the "S" solution looks nothing at all like "D", but gets to the heart of what users really wanted and needed when they asked for "D".

Incremental1_1

In the end, you might have to trust yourself, even in the face of users who either want more than you know would be good or something less or different than you know you can offer if you keep innovating in revolutionary--not just incremental--ways. Our Head First books are among the top-selling computer books today, virtually all of them occupy the #1 slot for their topic category. But not only did nobody ask for such a bizarre format for a technical book, we were warned that it would never work. We were told that people would hate these books. That they were too different, too pictorial, too... tacky to be taken seriously. But we knew the brain science and learning theory behind the format, and trusted that the principles worked. That for most (not all) readers, this format really did lead to faster, deeper learning. We trusted that people would look beyond the surface aspects of the implementation, and that if they got real results from the book, they'd tell others.

Two other publishers turned us down for the series before O'Reilly took the chance. And I was nearly fired from Sun for trying to sneak 5% of what's in Head First into Sun courseware.

Are users/readers too clueless to know what to ask for? Of course not. But it's not a potential Java programmer's job to be a learning theory expert, anymore than I could have helped conceive of the iPod. I could make incremental suggestions about most of the tools I use, sure, but I don't have the background, skills, or vision to suggest the kind of revolutionary changes that create breakthrough products and services outside of my own very narrow domain.

What sparked this post was a somewhat contentious (and bold) 37Signals post, but I also remembered this post by Wiley editor Joe Wilcox.

This is tricky, of course, because it's not always obvious which user complaints/suggestions are based on real problems with your product, vs. naive feature requests that would do more harm than good. (Don't forget the Happy User Curve)

And this is NOT about giving them simply what we know is good for them but that they really don't want, because they probably won't stick around. This is about giving them what they really DO want... but simply don't realize it because they had no way to imagine it.

So maybe the key is to listen not only to what users say, but more importantly to what is motivating what they say. The rest is up to us. If we really care about our users, they'll just have to trust us... but more crucially--we have to trust ourselves.

Posted by Kathy on September 15, 2005 | Permalink

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d83451b44369e200d8345999ce69e2

Listed below are links to weblogs that reference Listening to users considered harmful?:

» People don't know how to ask for what they really want from Signal vs. Noise
Kathy Sierra has a wonderful story about horse-trainer Pat Parelli and his approach to workshops and training material. Commended on how well they were able to process user feedback, Pat responded that they really didn't do that too much. Their... [Read More]

Tracked on Sep 16, 2005 4:01:18 AM

» People don't know how to ask for what they really want from Signal vs. Noise
Kathy Sierra has a wonderful story about horse-trainer Pat Parelli and his approach to workshops and training material. Commended on how well they were able to process user feedback, Pat responded that they really didnt do that too much. Their... [Read More]

Tracked on Sep 16, 2005 4:03:29 AM

» Listening to users from Inline Comments
Other than the work we're doing right now, I believe the best major improvement we could make to Stellar is to restructure the materials, schedule and homework tools to be organized around class sessions. No one has asked for this change. [Read More]

Tracked on Sep 16, 2005 7:35:32 AM

» CPU Looks Into The Shade from shaded
CPU Looks Into The Shade [Read More]

Tracked on Sep 16, 2005 7:51:45 AM

» CPU Looks Into The Shade from shaded
CPU Looks Into The Shade [Read More]

Tracked on Sep 16, 2005 7:52:46 AM

» Listening To Customers Isn't Customer-Centric from The Other Bloke's Blog
Listening to Customers must be good. Clearly it's much better than Not Listening To Customers. However Kathy Sierra in her blog, Creating Passionate Users, has a most though-provoking item, Listening to users considered harmful? [Read More]

Tracked on Sep 16, 2005 7:55:08 AM

» Innovation and listening to customers from chiefskipper
Innovation and listening to customers [Read More]

Tracked on Sep 16, 2005 2:34:19 PM

» Listening to Users considered harmful? from tonygoodson
Kathy Sierra has (yet another!) great post, about (not) listening to users. It turned out that most of the major changes they made to their program came not from user requests and suggestions, but from the Parelli team's own innovations. [Read More]

Tracked on Sep 16, 2005 8:22:40 PM

» gebruikersgroep from elke's
[Read More]

Tracked on Sep 18, 2005 3:53:25 PM

» User triangulation: how to listen to customers from Hey Norton! - Ken Norton's blog
Quite by serendipity I've come across a few articles in the past week about listening to - and ignoring - users in the product development process. The first came from my friend Jonathan Alexander in a post to his blog:So [Read More]

Tracked on Sep 21, 2005 1:32:46 AM

» Listening to users considered harmful? from Geek Smithology
I read a post over on Kathy Sierras blog that really hit a high note with me. Go read the article right now and come back. Just in case you didnt read the article, heres a quote with the gist: we still assume that liste... [Read More]

Tracked on Sep 26, 2005 9:43:04 PM

» Opinionated Software from Andreas' Blog
Durch Rails wurde der Begriff opinionated Software bekannt. Was man darunter versteht macht ein Zitat des Railsentwicklers David deutlich:I consider stored procedures and constraints vile and reckless destroyers of coherence. No, Mr. Database, you can [Read More]

Tracked on Jan 15, 2006 9:02:52 AM

Comments

Great post, K! Though distilling customer requirements into real products has always been the role of marketing. Innovation often comes from outside.

What's a horse, by the way?

W.

Posted by: Wally | Sep 16, 2005 4:43:34 AM

This is exactly what the book "innovator's Dilemma" is about. Clayton Christensen did extensive research on this. Great book I highly recommend it.

Posted by: Lee White | Sep 16, 2005 5:35:49 AM

That's exactly what I mean about searching the noise. Not the user feedback noise, but any noise.

Passion starts before you design a product, before you show up for work in the morning. It even starts before you eat breakfast, get out of bed, or shake last nights dream out of your mind. Passion is your destiny.

You can buy the uniform, stitch on all the badges and recite the pledge, but unless you are already passionate about what you do without the recognition, your improvement is going to be incremental.

This movie Contact is all about this. Ellie (Jodie Foster) does not find traditional incremental improvements of astronomy acceptable, and she is rewarded Passionate User style buy an eccentric billionaire who understands passion when he sees it and by an alien intelligence.

The Wright brothers threw all the incremental research pertaining to flight out the window, and as uneducated high school drop outs exceeded the progress of the greatest minds in the world.

What they did not do, is read Kathy's blog, which would have taught them how to nurture the passion of the following they created, their users, and obtain even greater heights with their wind.

Posted by: Shaded | Sep 16, 2005 8:19:10 AM

I always use this example...

150 year ago, when asked what they needed, farmers might have asked for bigger, stronger horses or maybe more horses. None of them would have thought to ask for a tractor.

IDEO, one of the worlds best design organizations, relies more heavily on observation of users actually using products instead of asking them questions. Not only do people not know what they want (or what's possible) they tend to answer in ways that hide the fact that they had difficulty in a (subconscious) attempt to not look stupid. For example, people that struggle with a particular UI element for several minutes might report that they found it intuitive in order to not feel (appear) stupid.

User feedback is important but it doesn't always mean what it appears to mean.

At the risk of sounding repetitive and predictable, great post.

-Matt

Posted by: Matt Galloway | Sep 16, 2005 8:56:13 AM

This post is dead on, and I thought it may be useful to point out an example of an effective way to listen to customers.

Mark Hurst provides an example here: http://www.goodexperience.com/blog/archives/000324.php

"In our non-directed listening labs, we ask customers to use the Internet in the way they normally use it at home or work. While we do have a goal for the research, we try to let the customers lead us to the answer, rather than the other way around."

Posted by: Eric | Sep 16, 2005 11:04:24 AM

OK, I'll take that risk of sounding repetitive... great post!

This all reinforces the point of how authenticity and transparency are so frickin' important in the marketing AND innovation process. It's still really difficult for users to accept that they are not being bullshat upon. The passion of the innovators and the execution of that passion, however, can trump any marketing plan. What Kathy states so well is that there is process to the marketing side that old-school marketers rarely, if ever, take into consideration.

Posted by: Mike H. | Sep 16, 2005 11:09:13 AM

Every time I read articles saying that user research is a panacea, I remeber the funny art project "The Most Wanted Paintings", by Komar & Melamid. They answered the question: how art would be if guided by user surveys? They have a nice site displaying the terrible paintings based in these surveys.

Hollywood blockbusters suffer a lot from it. How many mainstream movies would be better without a happy ending, or explaing too much, killing all serendipity. I believe they do user testing in pre-screen sessions, and 15% think the movie is too dificult, or too sad, so they change it to please everybody.

Gilberto Gil, a great brazilian musician and the country current minister of culture, has a verse that says: "The people knows what they want, but the people also wants what they don't know".

Posted by: Paulo Eduardo Neves | Sep 16, 2005 12:34:49 PM

User studies aren't much good for innovating. They are good, however, for making sure that the your design actually works. The user isn't "writing" the design so much as proofreading what you've already created.

I've always had a sneaking suspicion that most user testing is done for one of two reasons:

1) To provide a plausible rationalization for doing what the designer already knows they need to do.

2) To make the design process seem less appealing for meddling managers and hangers-on who feel their business expertise translates directly into usability expertise. From their perspective designing things looks easy and fun. They want to stay and help. User testing looks boring and intimdating. They remember they have other things to do.

Posted by: Bryan c | Sep 16, 2005 1:45:07 PM

Another great post, Kathy. It goes against the grain of what you usually post and my brain was not expecting it. :)

Your mention of focus groups reminds me of "The Apprentice". Usually the team that wins is the one that uses focus groups. But I think this is because the teams are given tasks in industries that they've never worked in before. They have no clue what the users need because the team members have never gone through the frustrations that the users have experienced.

Great innovation blossoms from the belief that what we've got just isn't good enough.

I recently heard that Nintendo is completely redesigning their controllers. This is a perfect example of not listening to the users. Users demanded better graphics and the gaming industry acted accordingly. Now the gaming community complains that the games look great but aren't as fun as the games of long ago.

I think Nintendo discovered on their own that the better graphics demanded better interactivity. If a game can make me feel like I'm on a battlefield I don't want to shoot my gun by pressing 'A'.

On the original Super Mario Brothers we forced our own interactivity into the game by leaning when Mario ran or jumped (some people even bounced a little). We HAD to use our imaginations. The comparison of old games vs new [SMACKDOWN!] is much like that of books vs TV.

Our interactivity used to come from imagination but now that the experience has "improved" with visuals and sounds... well, we need something else to keep our brains stimulated. What we've got now just isn't good enough.

Posted by: Marc Peabody | Sep 17, 2005 7:13:15 AM

A great post.To innovate,we must go beyond what the users say.

Posted by: anand sharma | Sep 17, 2005 11:20:16 AM

Kathy - I'd like to take slight issue with your post. In your opening 'graph you ask, "Should you create or modify a product based solely on what you believe in... even if it doesn't match what users tell you?" I don't think that discounting user feedback is what Pat Parelli did.

I think the key distinction I want to make here is that users "tell" you what they want in any number of ways. This is a fine point, but I believe it is essential. Making feature requests, sending cards and emails, talking to developers... these are ways of communicating. But so is switching products, not upgrading your software, or even, in Parelli's view, not learning fast enough or having a deep enough relationship with your horse. Listening to your users means not just listening to what they say, but listening to what they do.

You report Pat Parelli as saying "WE were still frustrated that people weren't learning quickly enough or progressing through the higher levels as well as we thought they could." That's key: "as well as WE thought they could." To me, it seems that this has nothing to do with ignoring user requests; it has to do with paying attention to user behavior. Parelli's insight came from listening to his users on a deep level, listening beyond the words and accolades. Parelli kept his vision of user success firmly in mind, paid attention to the actual results his users were achieving, and changed his approach accordingly. THAT is what I call paying attention to user feedback.

I guess that's how I'd wish you would reframe your point: You've always got to listen to your users, but sometimes you've got to listen beyond the words.

Posted by: Stuart | Sep 17, 2005 3:37:15 PM

When I developing software for Blue Cross I wrote a project development methodology. It was not (and is not) focused on project management, but a pre-project innovation. The term I coined for what you are discussing is "The Myth Of Limitation".

Several comments above a reader speaks about farmers not knowing to ask for a tractor - this is a great analogy.

It is not that we should not listen to clients/users but we should not let their limited understanding of what is possible, limit where the solution/software/project can go. It is important to listen and draw additional information into the open. In this way, we can discover what is truly desired but never contemplated from the client's perspective.

The chapter on this methodology became the longest chapter of my book with Cisco Press, The IT Career Builder's Toolkit. It is a critical factor across many disciplines and specialties in Information Technology.

I'm glad I stumbled across this blog. I think I'll showcase you on my blog at IT Toolbox.

Thanks,
Matthew Moran
The IT Career Builder's Toolkit
http://www.cbtoolkit.com
http://blogs.ittoolbox.com/pm/career/

Posted by: Matthew Moran | Sep 17, 2005 6:06:29 PM

Or as Nathaniel Borenstein put it (in his great little book Programming As If People Mattered): "Listen to your users, but ignore what they say."

Posted by: Matthew Morgan | Sep 17, 2005 8:02:14 PM

Fabulous. Everyone on the web should read that post. And read it again. Thanks.

Posted by: anon | Sep 18, 2005 8:24:45 PM

Excuse me, I didn't notice that html tags were stripped. Here is the funny link to Komar and Melamid Homepage The Most Wanted Paintings on the Web: http://www.diacenter.org/km/

Posted by: Paulo Eduardo Neves | Sep 19, 2005 10:28:53 AM

The comments about listening to what the users are saying, what they're not saying, and how it's being said reminds me of the quote by Claude Debussy, "Music is the silence between the notes."

Posted by: Tim | Sep 21, 2005 2:07:25 PM

I think there's a difference between normative desires and technical desires. A company/organization can guide the vision of a product through normative changes and this makes sense, it's THEIR product and they have the most expertise on it. But there's a technical aspect to the product, the logistics, the execution of distribution, marketing, and selling is a different field. This is the consumer's specialty, they are the ones best aware of what good customer service looks like, buying and interacting with companies is the consumer's expertise.

So while I agree with the premise of this post, let's not paint too broad a brush here.

Posted by: G. M. | Dec 8, 2005 4:53:10 PM

Only the retarded offspring of a village idiot and a TV weather girl would assume that "listening to users" is the same as "bowing down and licking the soles of your user's feet and then obeying their every command".

Listening to users is essential.

Understanding their real needs and desires (which may not be stated explicitly in short bullet-point format) is both a skill to be mastered and a critical detail.

Smart people generally know how to do their job, what they want to do, and what would make them happy. What they *don't* know is how to achieve that - we need to listen to them, understand them, and then keep up our end of the bargain by giving them something great.

Listening to users doesn't mean abdicating responsibility for design and implementation to the users.

Posted by: Bob McBob | Sep 4, 2006 4:17:41 PM

Although unpopular like an excerpt out of Bertrand Russell's 'Unpopular Essays', what determines the first success of a product is merely its marketing. A product can be as bad as hell: if you obsessively bring it before the attention of the public, the public shall buy it. Simply.
They buy everything, as long as you can enter their attention scope. In a scale economy, it's not really a matter of quality: it's a matter of noise in order to pick from the big numbers.

No matter how disagreeable this fact may be, it's still a fact.

A product can be good or very good. If it has no funds for marketing, no one will ever consider it - they may even mock at it.

It is true that you can't sell a non working product for long, but you can definitely sell for how long as you want (lawsuits allowing) a barely working one provided you have more money to invest in its publicity than you had to invest in its making.

The outcome is that who has money, will have more money also churning out lower products, and who puts all her/his efforts in a decent product, will have less money than she/he had at the onset, with no return.

Posted by: Alberto | Jan 30, 2007 3:17:43 PM

The comments to this entry are closed.