Creating Passionate Users

Sensemaking 3:The search for a representation

I promised a story about a sensemaking episode from my work, and here’s one. It’s a bit long, but I think you’ll find it amusing. It’s really a voyage of discovery as I try to figure out how to make sense of what people do… which is ultimately what my scientific life is all about:  What do people do?  How do they do it?  And why do they act that way? 

How do people manage interruptions?

A few years ago I became very interested in how people manage their interruptions. We’ve talked a lot about this in the CPU blog over the years.  See:   The Asymptotic Twitter Curve and Multitasking makes us stupid? and it’s still an area of active interest in the research community and here at Creating Passionate Users.   

But back in 1996 I was still at Apple and very much curious about how people did to actually manage their time and attention resources. So, being a good research guy, I went out and did a field study of some experts in attention management—our administrative assistants. After all, they somehow manage to simultaneously answer the phone, respond to email, process paperwork, deal with visitors and answer questions shouted out down the hall. If you watch a good assistant for any time, it’s clear that they’re masters at interrupt handling.  The ones that are good, are REALLY good.
 

So, how do they do it?

I videotaped several hours of assistants during the busy parts of their day and was pretty amazed. As I coded up each of the events they handled (phone-call, person-at-desk, email, etc.) I wrote down what the event was, when it happened, what resources the assistant needed to resolve the request and how long it took to handle it. Simple.

Then, from this data, I thought I’d build an interruption model that would allow me to predict how well a given person could handle various kinds, speeds and degrees of interruptions. What I was REALLY after was a kind of test bed where we could simulate different kinds of software tools and gadgets to help out with interruption handling. I was also curious about what the upper limit of interruptions is in the office environment. NASA has done some great work in modeling pilot workload factors, and I was wondering about the office worker—how would they respond under differing workloads?

So I collected my data and began to try and build a model of it.

Now there’s a representation of problem-solving behavior that’s been around since 1972 when Alan Newell and Herb Simon wrote about Human Problem Solving in their (now classic) tome. A problem-behavior graph (I’ll call them PBGs) is a boxes-and-arrows way to show what “knowledge states” people go through in the course of solving a problem.

 

It’s a way to show that people know a bunch of stuff about the problem (box 1), then think about it for a while and get themselves into a new knowledge state (box 2), think some more (and get to box 3). This goes on for a while, with the person trying out different ideas about how to solve the problem. Sometimes they’ll back up to something they’ve already thought about (for example, box 4 is really the same as box 3,  it just happens after box 3) and go forward with a new thought (box 5).  Etc.   

The diagram is what’s important here. Notice that it runs left-to-right, then top-down. That means that the problem-solving process has time going back and forth on the page. This is a great way to show how people learn stuff, change their minds, and generally go back and forth trying to solve the problem (and Newell & Simon typically used puzzles in their studies).

I’m a researcher who knows his literature, so I pulled this diagramming scheme out of their book and tried to use it for my real-life observations.

Now bear in mind that my goal (my “domain of interest” from yesterday’s posting) is to be able to understand what the admins are doing, how fast they’re doing it, and how well they can handle their interruptions.

But the PBG has a whacky way of showing time. The PBG is good at showing the space of alternatives considered by the person, but not so good at showing parallel activity or how events unfold in time.

So I came up with the parallel problem behavior graph – a nice variant way of diagramming what people were doing.

You can see that the parallel-PBG has time running left-to-right (the way you’d think), but also the ability to show events happening in parallel (such as when the admin answers the phone--event 2.0--while simultaneously writing an email to a different person--event 3.0). I even added a way to show that an event was an interruption—that’s what the dotted lines mean. Event 2.1 is an interruption that happens during event 2.0 (and that’s why you see 2.0 at the end—you have to go finish it up after handing the interruption event).

Great! So I coded up a bunch of events in this representation in a data file and used it to derive a model of how people work under various interruption loads.

Then I wrote a small process model program that I use to model how people worked. Basically, given the task, it would simulate how long the person would take to solve the task, then switch to the next task. Interruptions could happen while other tasks were taking place. Great!

Except it didn’t work.

Remember that I had lots of data from the field study. So I had broken the data up into two parts – the modeling part (which I used to base my model) and the testing part (which I used to test my model). But when I tried out my model against the testing data, it wouldn’t predict anything very well. My model said that the admin could handle 3.0 interruptions per minute, but my testing data showed they could actually handle much less. What was worse, as I tested the model under higher interruption rates, the model results got worse and worse.

What went wrong? I sat and looked at the model for a while, going over the algorithms and methods used. Nothing seemed apparently wrong.

So I went back and looked at the original data. What was I missing?

I remember the moment clearly: I was watching one of the tapes, double checking my coding to make sure I got all the event times right when I noticed something that didn’t quite fit the model. The assistant was answering email when the phone rang. As she picked up the phone, her boss asked a question from the next room.  Ah ha! Key insight: Interruptions can happen during the setup to handle an interruption!

That’s when I realized that my model, which was based on the Newell & Simon problem behavior graph, had a fundamental problem: the transitions between the tasks (the boxes in the diagrams) were assumed to be instant and atomic. That is, there was no way for the model to show that there was a setup time (that was always just included as part of the task). But now that I realized that setup could be interrupted as well, I could break the setup out as a separate time as well.

 

And that led me to this diagram, what I now call the “parallel task behavior graph.”

Now, the setup time for a task is explicitly represented, and it can be interrupted as well.

I went back to my model, made the changes to the simulator, and voila, the model now matched the testing data quite nicely.

From the sensemaking perspective, what happened to me followed the pattern quite nicely: I collected the data, then organized it by coding it and building a model. I then checked my work by validating the model against the test data. When the data didn’t fit into the model, I went back to the collected data, looking for another way of organizing it that would be better. That’s what led me to invent the “parallel task behavior graph” as a way of organizing this data. It’s not perfect, and it doesn’t explain everything about how people handle interruptions, but it does a decent job.

In particular, this model shows that as your interruption rate increases, eventually you start getting interrupts during your interrupts—and that’s when everything starts to fall apart. Your multitasking skills don’t matter at that point, and your ability to get anything done starts to fall apart.

But you get the idea… Sensemaking is in many ways a search for the right organization or the right way to represent what you know about a topic. It’s data collection, analysis, organization and performing the task.

And as all the comments are alluding, the more you know about the process of sensemaking, the better sensemaker you’ll be.

 

Tomorrow: Summary of the comments readers have sent in.

Sensemaking 2: What I do to make sense

I’m a professional research scientist. Research is what I do day in, day out. So what I do for sensemaking might not be what you do at all, or it might be really relevant. I don’t know.

But I do know that sensemaking is a big, big part of my job. And I know that there are sensemaking methods…and then there’s what actually happens. I’m going to ignore the sensemaking prescriptions for the moment and focus on what really goes on.

The common conception of research is that a scientist first thinks up a hypothesis, then collects data to test it, then writes up a neat analysis confirming or disconfirming the hypothesis. That’s beautiful, but it’s also almost completely wrong.

When I’m making sense of some complicated area, it’s more of a full-contact, sweaty, wrestling-around-with-data kind of thing. Trust me: it’s not nearly as antiseptic and passionless as the common conception would have it. This is red-blooded science as played on the field. It’s more of a rugby scrum than a still-life chess game. Here’s how it goes for me…

FIRST: Figure out what it is that you’re trying to understand or get done—let’s call this the domain. This is crucial because you can waste a lot of time doing various experiments or studies on the wrong thing. I find it valuable to write out a quick summary of what I’m trying to understand… often just as a note in my notebook, or maybe in an email to a friend. The point is that you need to frame things because the act of framing helps to focus on what to do next.

Of course, it’s not always so neat. Sometimes you only come to realize later what the topic really is. Sometimes you frame a domain one way and later discover that this isn’t a good framing at all. Then things shift and that’s where it gets interesting.  And then every so often you end up by accident in a domain that you hadn’t planned on studying. That’s serendipity, and it happens more than you’d think. Regardless, knowing your domain leads to the next step…

SECOND: Collect a lot of information about the domain. I like to read about how others think about the domain, what works and (just as importantly) what doesn’t work. Sometimes people will say how they think about the whole domain, and that’s often incredibly useful because they’ll have already organized things in a way that’s useful. I look for varying opinions and results. In the discrepancies between accounts there’s often something of interest. (Something like this: “Smith says people always do X, but Jones says they never do X… how can they both be right?”)

I should point out that often you can make sense of something just by doing this kind of research. It happens all the time—you’re working on a domain, and then someone writes an article that explains it all. A newspaper reporter would call that “being scooped.”

As you collect and organize, you'll find that you can target your collecting more finely.  That's when I'll start to do experiments (to collect exactly the data I need). 

THIRD: Organize the information.  Depending on what the domain is, you might look for an organization that helps you see the entire structure of what’s known, or you might end up building a very detailed model. In almost every case, I spend a lot of time figuring out how to organize the information I have into some kind of representation. This representation could be something as simple as a bunch of folders with topics, or it could be a much more complex spreadsheet model. But figuring out how to represent (and organize) what I know is key. Making this representation is usually when the “ah-ha” moments happen. It’s when pieces of the puzzle start falling into place and I realize the connections between everything.

FOURTH: Iterate. Realize that you almost never get it right the first time. Sometimes I’ll get the original domain wrong, and I’ll be studying the wrong thing. Sometimes I’ll collect a lot of junk information that I have to winnow out. And sometimes I’ll just create representations of that information that don’t work out.

In any case, keep iterating—on the domain, on the information set, on the representation—until you’re to a point where you can satisfy your need to make sense of the original domain.

FIFTH: Do. As in, do whatever it was you wanted when you figured out what the domain was. Research of the kind I work on is always trying to figure out something so you can do the next thing.

I know this is all a little abstract, but tomorrow I’ll post with a real example of a research sensemaking story that will map onto all of these steps. Then Friday I’ll summarize all of the amazing and remarkable comments folks have been making on the “Sensemaking 1” post. (Thanks for all your comments, btw, I’ve been enjoying reading them!)

Sensemaking 1

Here’s an important question for all of us: How do you make sense of something that’s big and complicated?  Say… something like why your users aren’t passionate about your product?

For the past several years I’ve been thinking a great deal about sensemaking—that is, the processes people go through when trying to “make sense” of a body of knowledge.

Think about it—sensemaking is what you do when you’re trying to organize your taxes, or when you want to understand what’s going on in the Middle East.  It’s what you do to figure out why your software just doesn’t seem to have the right zing for the customers. And it’s what you do when trying to wrap your mind around that great new idea for a startup. It’s figuring out how and why things make sense… or don’t.

What’s always struck me about sensemaking behavior is this: People just don’t seem to be all that good at it. They take notes on the topic, then never go over them, or lose them in the shuffle of life. People seem to rarely understand that sensemaking is a skill like language. You can be good or bad at it, and the level of skill makes a big difference.

So I’ve made this a central part of my research career: WHAT do people do when they’re trying to make sense of the world? And, just as importantly, WHY are people so bad at it? A great deal of my career (at PARC, Apple, IBM and now at Google) has been a long study of these sensemaking behaviors.

Let’s be more precise: If you’re trying to understand a fairly hefty topic… what is it that you do? How do you collect information, organize it and figure out what’s important (and what’s not)?

I know what I do (and I’ll tell you below)… but pause and think about this for a second. What do you do?

(I’ll wait.)

Okay. What’s the answer? When I ask people this question, I get two really interesting responses.

1. “I don’t know, I just sort of do it…” This is a fine answer. It tells me that sensemaking is a skill that you’ve practiced so much that it’s become automatic…. OR… it’s a skill you never practice. In either case, this is a better reply than someone who starts rambling long about what they do, but it’s clear that what they’re saying is just a confabulation (a nice word meaning “they’re making it up”).

2. I collect a bunch of information, then organize it, then I get the answer. This is also a fine answer. At least you’re aware of the “collection phase” and some basic collection organizational process. But that last step is the killer—how do you just “get the answer”?

Ahh.. there’s the magic! How DO you know what to do to get the answer?

In 1993 I wrote a paper with the somewhat forbidding title “The cost structure of sensemaking,” which basically points out that people take into account all kinds of factors when deciding what to do when making sense. They worry about how long it will take, how many errors will happen during the process and how much the whole process will cost. Interestingly, many of these “costs” are figured intuitively, and often incorrectly, leading people to do all kinds of strange things.

Which is why I think understanding how people “make sense” of their world is so fascinating. This is really why I went to Google—because there’s a ton of data there about what people do when trying to understand their world.

What do I do? Well, I’ll tell you.. I collect a ton of information, then organize it, then I map it to the task I’m trying to do. Then I repeat. The iteration is important because I almost never get the right information on the first pass, or I don’t know how to organize it, or I don’t know how to use it to get the task accomplished it once it’s all organized. (I’ll go into details in my next posting on Sensemaking in a day or two.)

I know CPU readers are a really interesting bunch.  So a question for everyone: What do you do when you do sensemaking? Can you illustrate with an example? I’ll summarize the most interesting responses in another post a week from today.