Creating Passionate Users

Sensemaking 4: Summary of your comments

Wow!

That was my first thought about reading everyone’s comments on the sensemaking posts. So many thoughtful comments that it’ll be hard to address everyone. But the collection made for some very interesting reading. So first, a big thank you to everyone who commented. (And forgive me if I don’t mention you here. There really were more comments than I can include in a single post.  If you commented, trust me, I read your post at least twice.)

I read through all the commentary, then decided I couldn’t handle it all on my computer display… so I printed it out and went through with my markers, highlighting all the comments about personal process (in yellow), then highlighting all the comments about tools used (in pink), and then all the comments about metacognitive strategies (in blue). I then talked about it with some of my friends at Google to make sense of it all.

And here’s what struck me…

People resonated with the sensemaking model. That is, it seems to describe a lot of what people do when they’re faced with a sticky problem. That’s good news, as it could have been the other way. More than a few people rewrote the model in their own words, and that’s good (we know that an effective way to learn stuff is to reorganize it to fit your own world, and rewriting in your language is an effective way to do so).

More importantly, commenters point out the importance of continuing to ask questions during the process. This is a great metacognitive strategy in general (see below) and helps to drive sensemaking forward. It’s also important not to get lost in the question-asking weeds as you do this. One really can ask too many questions.

Collecting information is important. As you’d imagine, pulling together your personal cache of information is crucial. Many people pointed out that search engines are incredibly useful in finding data. Alan gave his perspective as a patent attorney who does a lot of collection when evaluating proposed patents. More than a few information-heavy jobs require great skills in information collection.

Yet surprisingly, I’m not sure how good we really are (as a culture). I teach lots of classes on how to be a more effective internet searcher, and I’m constantly surprised to discover how people think about internet search. (We’ll talk about this in a future CPU post.) But the bottom line here is that you can only make sense out of what you collect, as it is played against the backdrop of what you already know.  The collection process can easily skew your results. Be careful here!

Metacognitive strategies.  Aside from being a great word to toss into partytime conversation, “metacognitive” describes things people do to manage their own thinking. And you CPU commenters had lots of metacognitive things to say. LOTS of people commented on giving yourself time to “think about it” and “let the subconscious play” or allowing the ideas to “gel in time.” And that’s good advice—time can be your friend in discovering the intrinsic structure of your information.

Another metacognitive strategy is to consciously ask questions about what you’re doing. If you periodically step back and take the larger perspective, maybe even ask pointed questions “Why am I doing this?” or “What will I learn from doing this?” – then you’re being metacognitive. Congratulations. As is well known from many studies of learning behavior, metacognitive strategies often pay off really well. They help to guide your thinking, and in the case of sensemaking, guide your choices about how much time to spend on collecting, organizing and then working on the target task.

Intuition. Although intuition isn’t exactly a metacognitive strategy, it’s clear that people recognize patterns in the data in ways that they can’t talk about. That kind of inarticulate recognition (meaning that you can’t talk about it easily) is what we perceive as intuition. We’ve all got it, and good sensemakers have good intuitions about how things go together. As Malcolm Gladwell correctly points out in Blink (and we’ve discussed earlier in our blog), intuition isn’t just a mysterious upwelling of innate knowledge, but is based on lots of experience and practice in the field.

So be wary of your intuitions in fields that aren’t your own. I might think I have great intuitions about how steelworkers build skyscrapers, but I wouldn’t trust those intuitions farther than I can spit. I would trust my intuitions about cognitive science, however, since I’ve spent years marinating in that world.

Social aspects. Many of you suggested some wonderful social strategies… Ask a friend. Talk about it with someone who likes a good discussion. Talk about it with someone who disagrees with you. There are entire schools of sensemaking (such as Karl Weick) that see sensemaking as an inherently social process. To them, “sense” is what is made by the community out of many pieces of information—that is, a certain amount of sense is made by agreement. This is a finding with deep consequences. But for us, just working the idea over with friends and colleagues is an important way to structure the information you have at hand. One of my tests of whether or not I’ve made sense of something is whether or not I can tell a story about it. I might call someone (Kathy, let’s say) and try to tell her what it’s all about. If I can do that, then I’ve made at least some sense.

Another great social strategy several people mentioned was “looking for someone else who has already made sense” of the topic. In the internet age, this is an ever-increasingly good idea. (It’s always been a good idea—it’s just easier now, and with a whole lot more coverage than ever before.)

Iteration. Yes, plain old iteration is also key. The consensus of commenters was that nobody got things right the first time around, and that you need to repeatedly look FOR information and look AT the information you have. Just looking once, or just organizing once probably isn’t sufficient.

 

So… what surprised me?

Representations. The biggest surprise was the relative lack of discussion about the organizations and representations people build up of their sensemaking collections. Normally, I think about people collecting then organizing their information—but how does one organize a collection? In my earlier studies of sensemakers, the particular ways that information was grouped together, sorted out and interrelated was always important. Why didn’t this theme emerge here? Perhaps it’s because people have a set of organizations they always use. Or that it seems too obvious. Or maybe that it’s just too hard to talk about. But it surprised me!

Tools. Another surprise was the number of tools people mentioned. There were very few. Mindmaps, outlines, VooDooPad, Devonthink, search engines – and that was more-or-less it! Is it the case that we just don’t have that many tools to help people with their sensemaking tasks?

Personally, for my sensemaking tasks I use:

> lots of paper notes in various organizations (primarily piles, grouped by common topics), even going so far as to highlight items and use sticky notes as flags;

> many online files that have different roles in my sensemaking process (like “outline” “extras” and “things to do”;

> search engines for discovery and collecting (of course);

> personal desktop search (because I typically collect information into a central repository on my local computer) and I need to find it again;

> Powerpoint --not always, but often for outlining and sometimes for collecting stuff of various media types all together in one place

Sorry to be so low-tech... but it's what works for me!

All in all, I found our sensemaking conversation to be fascinating. It’s been wonderful to hear everyone’s voice in the comments and to get an idea of how varied and articulate the readers of CPU are. Thanks again!

Now we’ll go back to our regularly scheduled Creating Passionate User’s topics… I have a few other things to talk about as well. Custom search engines, color psychology, how people visually scan their world, what it takes to create a really useful user interface… More coming soon!

P.S.  A few people have noted that they're having trouble with the images in these posts.  Would you please comment if you're having trouble too?  I'd like to debug this soon! 

 

Sensemaking 3:The search for a representation

I promised a story about a sensemaking episode from my work, and here’s one. It’s a bit long, but I think you’ll find it amusing. It’s really a voyage of discovery as I try to figure out how to make sense of what people do… which is ultimately what my scientific life is all about:  What do people do?  How do they do it?  And why do they act that way? 

How do people manage interruptions?

A few years ago I became very interested in how people manage their interruptions. We’ve talked a lot about this in the CPU blog over the years.  See:   The Asymptotic Twitter Curve and Multitasking makes us stupid? and it’s still an area of active interest in the research community and here at Creating Passionate Users.   

But back in 1996 I was still at Apple and very much curious about how people did to actually manage their time and attention resources. So, being a good research guy, I went out and did a field study of some experts in attention management—our administrative assistants. After all, they somehow manage to simultaneously answer the phone, respond to email, process paperwork, deal with visitors and answer questions shouted out down the hall. If you watch a good assistant for any time, it’s clear that they’re masters at interrupt handling.  The ones that are good, are REALLY good.
 

So, how do they do it?

I videotaped several hours of assistants during the busy parts of their day and was pretty amazed. As I coded up each of the events they handled (phone-call, person-at-desk, email, etc.) I wrote down what the event was, when it happened, what resources the assistant needed to resolve the request and how long it took to handle it. Simple.

Then, from this data, I thought I’d build an interruption model that would allow me to predict how well a given person could handle various kinds, speeds and degrees of interruptions. What I was REALLY after was a kind of test bed where we could simulate different kinds of software tools and gadgets to help out with interruption handling. I was also curious about what the upper limit of interruptions is in the office environment. NASA has done some great work in modeling pilot workload factors, and I was wondering about the office worker—how would they respond under differing workloads?

So I collected my data and began to try and build a model of it.

Now there’s a representation of problem-solving behavior that’s been around since 1972 when Alan Newell and Herb Simon wrote about Human Problem Solving in their (now classic) tome. A problem-behavior graph (I’ll call them PBGs) is a boxes-and-arrows way to show what “knowledge states” people go through in the course of solving a problem.

 

It’s a way to show that people know a bunch of stuff about the problem (box 1), then think about it for a while and get themselves into a new knowledge state (box 2), think some more (and get to box 3). This goes on for a while, with the person trying out different ideas about how to solve the problem. Sometimes they’ll back up to something they’ve already thought about (for example, box 4 is really the same as box 3,  it just happens after box 3) and go forward with a new thought (box 5).  Etc.   

The diagram is what’s important here. Notice that it runs left-to-right, then top-down. That means that the problem-solving process has time going back and forth on the page. This is a great way to show how people learn stuff, change their minds, and generally go back and forth trying to solve the problem (and Newell & Simon typically used puzzles in their studies).

I’m a researcher who knows his literature, so I pulled this diagramming scheme out of their book and tried to use it for my real-life observations.

Now bear in mind that my goal (my “domain of interest” from yesterday’s posting) is to be able to understand what the admins are doing, how fast they’re doing it, and how well they can handle their interruptions.

But the PBG has a whacky way of showing time. The PBG is good at showing the space of alternatives considered by the person, but not so good at showing parallel activity or how events unfold in time.

So I came up with the parallel problem behavior graph – a nice variant way of diagramming what people were doing.

You can see that the parallel-PBG has time running left-to-right (the way you’d think), but also the ability to show events happening in parallel (such as when the admin answers the phone--event 2.0--while simultaneously writing an email to a different person--event 3.0). I even added a way to show that an event was an interruption—that’s what the dotted lines mean. Event 2.1 is an interruption that happens during event 2.0 (and that’s why you see 2.0 at the end—you have to go finish it up after handing the interruption event).

Great! So I coded up a bunch of events in this representation in a data file and used it to derive a model of how people work under various interruption loads.

Then I wrote a small process model program that I use to model how people worked. Basically, given the task, it would simulate how long the person would take to solve the task, then switch to the next task. Interruptions could happen while other tasks were taking place. Great!

Except it didn’t work.

Remember that I had lots of data from the field study. So I had broken the data up into two parts – the modeling part (which I used to base my model) and the testing part (which I used to test my model). But when I tried out my model against the testing data, it wouldn’t predict anything very well. My model said that the admin could handle 3.0 interruptions per minute, but my testing data showed they could actually handle much less. What was worse, as I tested the model under higher interruption rates, the model results got worse and worse.

What went wrong? I sat and looked at the model for a while, going over the algorithms and methods used. Nothing seemed apparently wrong.

So I went back and looked at the original data. What was I missing?

I remember the moment clearly: I was watching one of the tapes, double checking my coding to make sure I got all the event times right when I noticed something that didn’t quite fit the model. The assistant was answering email when the phone rang. As she picked up the phone, her boss asked a question from the next room.  Ah ha! Key insight: Interruptions can happen during the setup to handle an interruption!

That’s when I realized that my model, which was based on the Newell & Simon problem behavior graph, had a fundamental problem: the transitions between the tasks (the boxes in the diagrams) were assumed to be instant and atomic. That is, there was no way for the model to show that there was a setup time (that was always just included as part of the task). But now that I realized that setup could be interrupted as well, I could break the setup out as a separate time as well.

 

And that led me to this diagram, what I now call the “parallel task behavior graph.”

Now, the setup time for a task is explicitly represented, and it can be interrupted as well.

I went back to my model, made the changes to the simulator, and voila, the model now matched the testing data quite nicely.

From the sensemaking perspective, what happened to me followed the pattern quite nicely: I collected the data, then organized it by coding it and building a model. I then checked my work by validating the model against the test data. When the data didn’t fit into the model, I went back to the collected data, looking for another way of organizing it that would be better. That’s what led me to invent the “parallel task behavior graph” as a way of organizing this data. It’s not perfect, and it doesn’t explain everything about how people handle interruptions, but it does a decent job.

In particular, this model shows that as your interruption rate increases, eventually you start getting interrupts during your interrupts—and that’s when everything starts to fall apart. Your multitasking skills don’t matter at that point, and your ability to get anything done starts to fall apart.

But you get the idea… Sensemaking is in many ways a search for the right organization or the right way to represent what you know about a topic. It’s data collection, analysis, organization and performing the task.

And as all the comments are alluding, the more you know about the process of sensemaking, the better sensemaker you’ll be.

 

Tomorrow: Summary of the comments readers have sent in.

Sensemaking 2: What I do to make sense

I’m a professional research scientist. Research is what I do day in, day out. So what I do for sensemaking might not be what you do at all, or it might be really relevant. I don’t know.

But I do know that sensemaking is a big, big part of my job. And I know that there are sensemaking methods…and then there’s what actually happens. I’m going to ignore the sensemaking prescriptions for the moment and focus on what really goes on.

The common conception of research is that a scientist first thinks up a hypothesis, then collects data to test it, then writes up a neat analysis confirming or disconfirming the hypothesis. That’s beautiful, but it’s also almost completely wrong.

When I’m making sense of some complicated area, it’s more of a full-contact, sweaty, wrestling-around-with-data kind of thing. Trust me: it’s not nearly as antiseptic and passionless as the common conception would have it. This is red-blooded science as played on the field. It’s more of a rugby scrum than a still-life chess game. Here’s how it goes for me…

FIRST: Figure out what it is that you’re trying to understand or get done—let’s call this the domain. This is crucial because you can waste a lot of time doing various experiments or studies on the wrong thing. I find it valuable to write out a quick summary of what I’m trying to understand… often just as a note in my notebook, or maybe in an email to a friend. The point is that you need to frame things because the act of framing helps to focus on what to do next.

Of course, it’s not always so neat. Sometimes you only come to realize later what the topic really is. Sometimes you frame a domain one way and later discover that this isn’t a good framing at all. Then things shift and that’s where it gets interesting.  And then every so often you end up by accident in a domain that you hadn’t planned on studying. That’s serendipity, and it happens more than you’d think. Regardless, knowing your domain leads to the next step…

SECOND: Collect a lot of information about the domain. I like to read about how others think about the domain, what works and (just as importantly) what doesn’t work. Sometimes people will say how they think about the whole domain, and that’s often incredibly useful because they’ll have already organized things in a way that’s useful. I look for varying opinions and results. In the discrepancies between accounts there’s often something of interest. (Something like this: “Smith says people always do X, but Jones says they never do X… how can they both be right?”)

I should point out that often you can make sense of something just by doing this kind of research. It happens all the time—you’re working on a domain, and then someone writes an article that explains it all. A newspaper reporter would call that “being scooped.”

As you collect and organize, you'll find that you can target your collecting more finely.  That's when I'll start to do experiments (to collect exactly the data I need). 

THIRD: Organize the information.  Depending on what the domain is, you might look for an organization that helps you see the entire structure of what’s known, or you might end up building a very detailed model. In almost every case, I spend a lot of time figuring out how to organize the information I have into some kind of representation. This representation could be something as simple as a bunch of folders with topics, or it could be a much more complex spreadsheet model. But figuring out how to represent (and organize) what I know is key. Making this representation is usually when the “ah-ha” moments happen. It’s when pieces of the puzzle start falling into place and I realize the connections between everything.

FOURTH: Iterate. Realize that you almost never get it right the first time. Sometimes I’ll get the original domain wrong, and I’ll be studying the wrong thing. Sometimes I’ll collect a lot of junk information that I have to winnow out. And sometimes I’ll just create representations of that information that don’t work out.

In any case, keep iterating—on the domain, on the information set, on the representation—until you’re to a point where you can satisfy your need to make sense of the original domain.

FIFTH: Do. As in, do whatever it was you wanted when you figured out what the domain was. Research of the kind I work on is always trying to figure out something so you can do the next thing.

I know this is all a little abstract, but tomorrow I’ll post with a real example of a research sensemaking story that will map onto all of these steps. Then Friday I’ll summarize all of the amazing and remarkable comments folks have been making on the “Sensemaking 1” post. (Thanks for all your comments, btw, I’ve been enjoying reading them!)

Sensemaking 1

Here’s an important question for all of us: How do you make sense of something that’s big and complicated?  Say… something like why your users aren’t passionate about your product?

For the past several years I’ve been thinking a great deal about sensemaking—that is, the processes people go through when trying to “make sense” of a body of knowledge.

Think about it—sensemaking is what you do when you’re trying to organize your taxes, or when you want to understand what’s going on in the Middle East.  It’s what you do to figure out why your software just doesn’t seem to have the right zing for the customers. And it’s what you do when trying to wrap your mind around that great new idea for a startup. It’s figuring out how and why things make sense… or don’t.

What’s always struck me about sensemaking behavior is this: People just don’t seem to be all that good at it. They take notes on the topic, then never go over them, or lose them in the shuffle of life. People seem to rarely understand that sensemaking is a skill like language. You can be good or bad at it, and the level of skill makes a big difference.

So I’ve made this a central part of my research career: WHAT do people do when they’re trying to make sense of the world? And, just as importantly, WHY are people so bad at it? A great deal of my career (at PARC, Apple, IBM and now at Google) has been a long study of these sensemaking behaviors.

Let’s be more precise: If you’re trying to understand a fairly hefty topic… what is it that you do? How do you collect information, organize it and figure out what’s important (and what’s not)?

I know what I do (and I’ll tell you below)… but pause and think about this for a second. What do you do?

(I’ll wait.)

Okay. What’s the answer? When I ask people this question, I get two really interesting responses.

1. “I don’t know, I just sort of do it…” This is a fine answer. It tells me that sensemaking is a skill that you’ve practiced so much that it’s become automatic…. OR… it’s a skill you never practice. In either case, this is a better reply than someone who starts rambling long about what they do, but it’s clear that what they’re saying is just a confabulation (a nice word meaning “they’re making it up”).

2. I collect a bunch of information, then organize it, then I get the answer. This is also a fine answer. At least you’re aware of the “collection phase” and some basic collection organizational process. But that last step is the killer—how do you just “get the answer”?

Ahh.. there’s the magic! How DO you know what to do to get the answer?

In 1993 I wrote a paper with the somewhat forbidding title “The cost structure of sensemaking,” which basically points out that people take into account all kinds of factors when deciding what to do when making sense. They worry about how long it will take, how many errors will happen during the process and how much the whole process will cost. Interestingly, many of these “costs” are figured intuitively, and often incorrectly, leading people to do all kinds of strange things.

Which is why I think understanding how people “make sense” of their world is so fascinating. This is really why I went to Google—because there’s a ton of data there about what people do when trying to understand their world.

What do I do? Well, I’ll tell you.. I collect a ton of information, then organize it, then I map it to the task I’m trying to do. Then I repeat. The iteration is important because I almost never get the right information on the first pass, or I don’t know how to organize it, or I don’t know how to use it to get the task accomplished it once it’s all organized. (I’ll go into details in my next posting on Sensemaking in a day or two.)

I know CPU readers are a really interesting bunch.  So a question for everyone: What do you do when you do sensemaking? Can you illustrate with an example? I’ll summarize the most interesting responses in another post a week from today.

 

Rhythm method

Life runs on a pulse: without that basic rhythm, you’re dead. 

What’s more, that sense of a beat measuring out time is intrinsic to almost everything we do, be it conversations, living our lives or learning how to use software.

A story that illustrates the point…

A few years ago, while studying the way teachers teach Japanese in a classroom, I spent a lot of time watching videos of language classes. If you’ve ever spent time learning another language, you know what it’s like—lots of drilling and practice speaking. It’s an essential part of class; you need to get good at hearing and saying different word forms and that requires a great deal of back and forth between teacher and students.

But as I spent hours doing video analysis of the classes, I ended up doing a lot of rewinding and fast-forwarding. As you’d expect, I watched a lot of Japanese classes go by on the video monitor in 2X or 3X real time. And I noticed that when you watch these classes fly by on video, there’s an amazingly regular pulse to the class. The teacher prompts, the students reply… the teacher prompts, the students reply… on and on.

Curious about this unexpectedly regular pulse in the class, I actually went through one of the classes and noted each prompt and each response, wrote down the timecode, then plotted them out as a graph. To my amazement, the back-and-forth of the interactions was incredibly regular – each event showed up as a regularly spaced dot on my chart.

I’d found that there is a rhythm to a language class, a regular beat to the back-and-forth that makes the class work. I quickly noticed in the analysis that anytime the pulse was disrupted by more than a second or two, that was when something had broken down in the class—a student couldn’t answer a question, or when the teacher moved onto another segment of the class.

Unfortunately, looking for rhythmic structure in the classroom wasn’t the topic of that research study, so I filed this under “interesting stuff to think about in the future” and pressed on with my work.

But what I found completely fascinating was the incredible regularity and structure of the interactions. Was this true of other kinds of interaction settings as well?

Years later I found Edward T. Hall’s book, Dance of Life, that pretty much confirms that observation I’d made. Hall’s an anthropologist who’s made a career out of watching how people interaction in time and space. In this book he shows how people not only interlock the rhythms of conversation, but also how they move closer and farther apart, effectively dancing in their day-to-day interactions. 

More recently, I’ve been looking at the rhythms of people doing search queries. Lo and behold, a similar pattern stands out… If you watch someone’s timing of searches, you can see striking patterns of when they post a query, how long they go between queries and how each day is similar to other days.

Although it doesn’t always feel that way, people are amazingly rhythmic in their behaviors, up to and including their use of software. This is a point beautifully noted by Bo Begole, John Tang and Roscoe Hill in their paper Rhythm modeling, visualizations and applications.

What does this have to do with Creating Passionate Users? It prompts me to ask you, dear readers, a question. What rhythmic patterns do you see in the course of your work?  Do passionate users have a rhythm that works well, signaling happy use? I’m interested in hearing your notes on the time course of events. Do you find people using software in an interesting repeating pattern? Just using something everyday is dull, but perhaps you see other pulses, rhythms and beats that go beyond the ordinary. What about it?  Is rhythm a good thing, or the sign that things have become mechanical? Is rhythm a part of a flow experience, or its nemesis?