Sensemaking 3:The search for a representation

I promised a story about a sensemaking episode from my work, and here’s one. It’s a bit long, but I think you’ll find it amusing. It’s really a voyage of discovery as I try to figure out how to make sense of what people do… which is ultimately what my scientific life is all about:  What do people do?  How do they do it?  And why do they act that way? 

How do people manage interruptions?

A few years ago I became very interested in how people manage their interruptions. We’ve talked a lot about this in the CPU blog over the years.  See:   The Asymptotic Twitter Curve and Multitasking makes us stupid? and it’s still an area of active interest in the research community and here at Creating Passionate Users.   

But back in 1996 I was still at Apple and very much curious about how people did to actually manage their time and attention resources. So, being a good research guy, I went out and did a field study of some experts in attention management—our administrative assistants. After all, they somehow manage to simultaneously answer the phone, respond to email, process paperwork, deal with visitors and answer questions shouted out down the hall. If you watch a good assistant for any time, it’s clear that they’re masters at interrupt handling.  The ones that are good, are REALLY good.
 

So, how do they do it?

I videotaped several hours of assistants during the busy parts of their day and was pretty amazed. As I coded up each of the events they handled (phone-call, person-at-desk, email, etc.) I wrote down what the event was, when it happened, what resources the assistant needed to resolve the request and how long it took to handle it. Simple.

Then, from this data, I thought I’d build an interruption model that would allow me to predict how well a given person could handle various kinds, speeds and degrees of interruptions. What I was REALLY after was a kind of test bed where we could simulate different kinds of software tools and gadgets to help out with interruption handling. I was also curious about what the upper limit of interruptions is in the office environment. NASA has done some great work in modeling pilot workload factors, and I was wondering about the office worker—how would they respond under differing workloads?

So I collected my data and began to try and build a model of it.

Now there’s a representation of problem-solving behavior that’s been around since 1972 when Alan Newell and Herb Simon wrote about Human Problem Solving in their (now classic) tome. A problem-behavior graph (I’ll call them PBGs) is a boxes-and-arrows way to show what “knowledge states” people go through in the course of solving a problem.

 

It’s a way to show that people know a bunch of stuff about the problem (box 1), then think about it for a while and get themselves into a new knowledge state (box 2), think some more (and get to box 3). This goes on for a while, with the person trying out different ideas about how to solve the problem. Sometimes they’ll back up to something they’ve already thought about (for example, box 4 is really the same as box 3,  it just happens after box 3) and go forward with a new thought (box 5).  Etc.   

The diagram is what’s important here. Notice that it runs left-to-right, then top-down. That means that the problem-solving process has time going back and forth on the page. This is a great way to show how people learn stuff, change their minds, and generally go back and forth trying to solve the problem (and Newell & Simon typically used puzzles in their studies).

I’m a researcher who knows his literature, so I pulled this diagramming scheme out of their book and tried to use it for my real-life observations.

Now bear in mind that my goal (my “domain of interest” from yesterday’s posting) is to be able to understand what the admins are doing, how fast they’re doing it, and how well they can handle their interruptions.

But the PBG has a whacky way of showing time. The PBG is good at showing the space of alternatives considered by the person, but not so good at showing parallel activity or how events unfold in time.

So I came up with the parallel problem behavior graph – a nice variant way of diagramming what people were doing.

You can see that the parallel-PBG has time running left-to-right (the way you’d think), but also the ability to show events happening in parallel (such as when the admin answers the phone--event 2.0--while simultaneously writing an email to a different person--event 3.0). I even added a way to show that an event was an interruption—that’s what the dotted lines mean. Event 2.1 is an interruption that happens during event 2.0 (and that’s why you see 2.0 at the end—you have to go finish it up after handing the interruption event).

Great! So I coded up a bunch of events in this representation in a data file and used it to derive a model of how people work under various interruption loads.

Then I wrote a small process model program that I use to model how people worked. Basically, given the task, it would simulate how long the person would take to solve the task, then switch to the next task. Interruptions could happen while other tasks were taking place. Great!

Except it didn’t work.

Remember that I had lots of data from the field study. So I had broken the data up into two parts – the modeling part (which I used to base my model) and the testing part (which I used to test my model). But when I tried out my model against the testing data, it wouldn’t predict anything very well. My model said that the admin could handle 3.0 interruptions per minute, but my testing data showed they could actually handle much less. What was worse, as I tested the model under higher interruption rates, the model results got worse and worse.

What went wrong? I sat and looked at the model for a while, going over the algorithms and methods used. Nothing seemed apparently wrong.

So I went back and looked at the original data. What was I missing?

I remember the moment clearly: I was watching one of the tapes, double checking my coding to make sure I got all the event times right when I noticed something that didn’t quite fit the model. The assistant was answering email when the phone rang. As she picked up the phone, her boss asked a question from the next room.  Ah ha! Key insight: Interruptions can happen during the setup to handle an interruption!

That’s when I realized that my model, which was based on the Newell & Simon problem behavior graph, had a fundamental problem: the transitions between the tasks (the boxes in the diagrams) were assumed to be instant and atomic. That is, there was no way for the model to show that there was a setup time (that was always just included as part of the task). But now that I realized that setup could be interrupted as well, I could break the setup out as a separate time as well.

 

And that led me to this diagram, what I now call the “parallel task behavior graph.”

Now, the setup time for a task is explicitly represented, and it can be interrupted as well.

I went back to my model, made the changes to the simulator, and voila, the model now matched the testing data quite nicely.

From the sensemaking perspective, what happened to me followed the pattern quite nicely: I collected the data, then organized it by coding it and building a model. I then checked my work by validating the model against the test data. When the data didn’t fit into the model, I went back to the collected data, looking for another way of organizing it that would be better. That’s what led me to invent the “parallel task behavior graph” as a way of organizing this data. It’s not perfect, and it doesn’t explain everything about how people handle interruptions, but it does a decent job.

In particular, this model shows that as your interruption rate increases, eventually you start getting interrupts during your interrupts—and that’s when everything starts to fall apart. Your multitasking skills don’t matter at that point, and your ability to get anything done starts to fall apart.

But you get the idea… Sensemaking is in many ways a search for the right organization or the right way to represent what you know about a topic. It’s data collection, analysis, organization and performing the task.

And as all the comments are alluding, the more you know about the process of sensemaking, the better sensemaker you’ll be.

 

Tomorrow: Summary of the comments readers have sent in.