« Sensemaking 2: What I do to make sense | Main | Sensemaking 4: Summary of your comments »

Sensemaking 3:The search for a representation

I promised a story about a sensemaking episode from my work, and here’s one. It’s a bit long, but I think you’ll find it amusing. It’s really a voyage of discovery as I try to figure out how to make sense of what people do… which is ultimately what my scientific life is all about:  What do people do?  How do they do it?  And why do they act that way? 

How do people manage interruptions?

A few years ago I became very interested in how people manage their interruptions. We’ve talked a lot about this in the CPU blog over the years.  See:   The Asymptotic Twitter Curve and Multitasking makes us stupid? and it’s still an area of active interest in the research community and here at Creating Passionate Users.   

But back in 1996 I was still at Apple and very much curious about how people did to actually manage their time and attention resources. So, being a good research guy, I went out and did a field study of some experts in attention management—our administrative assistants. After all, they somehow manage to simultaneously answer the phone, respond to email, process paperwork, deal with visitors and answer questions shouted out down the hall. If you watch a good assistant for any time, it’s clear that they’re masters at interrupt handling.  The ones that are good, are REALLY good.
 

So, how do they do it?

I videotaped several hours of assistants during the busy parts of their day and was pretty amazed. As I coded up each of the events they handled (phone-call, person-at-desk, email, etc.) I wrote down what the event was, when it happened, what resources the assistant needed to resolve the request and how long it took to handle it. Simple.

Then, from this data, I thought I’d build an interruption model that would allow me to predict how well a given person could handle various kinds, speeds and degrees of interruptions. What I was REALLY after was a kind of test bed where we could simulate different kinds of software tools and gadgets to help out with interruption handling. I was also curious about what the upper limit of interruptions is in the office environment. NASA has done some great work in modeling pilot workload factors, and I was wondering about the office worker—how would they respond under differing workloads?

So I collected my data and began to try and build a model of it.

Now there’s a representation of problem-solving behavior that’s been around since 1972 when Alan Newell and Herb Simon wrote about Human Problem Solving in their (now classic) tome. A problem-behavior graph (I’ll call them PBGs) is a boxes-and-arrows way to show what “knowledge states” people go through in the course of solving a problem.

 

Pbg1_1

It’s a way to show that people know a bunch of stuff about the problem (box 1), then think about it for a while and get themselves into a new knowledge state (box 2), think some more (and get to box 3). This goes on for a while, with the person trying out different ideas about how to solve the problem. Sometimes they’ll back up to something they’ve already thought about (for example, box 4 is really the same as box 3,  it just happens after box 3) and go forward with a new thought (box 5).  Etc.   

The diagram is what’s important here. Notice that it runs left-to-right, then top-down. That means that the problem-solving process has time going back and forth on the page. This is a great way to show how people learn stuff, change their minds, and generally go back and forth trying to solve the problem (and Newell & Simon typically used puzzles in their studies).

I’m a researcher who knows his literature, so I pulled this diagramming scheme out of their book and tried to use it for my real-life observations.

Now bear in mind that my goal (my “domain of interest” from yesterday’s posting) is to be able to understand what the admins are doing, how fast they’re doing it, and how well they can handle their interruptions.

But the PBG has a whacky way of showing time. The PBG is good at showing the space of alternatives considered by the person, but not so good at showing parallel activity or how events unfold in time.

So I came up with the parallel problem behavior graph – a nice variant way of diagramming what people were doing.

Pbg2

You can see that the parallel-PBG has time running left-to-right (the way you’d think), but also the ability to show events happening in parallel (such as when the admin answers the phone--event 2.0--while simultaneously writing an email to a different person--event 3.0). I even added a way to show that an event was an interruption—that’s what the dotted lines mean. Event 2.1 is an interruption that happens during event 2.0 (and that’s why you see 2.0 at the end—you have to go finish it up after handing the interruption event).

Great! So I coded up a bunch of events in this representation in a data file and used it to derive a model of how people work under various interruption loads.

Then I wrote a small process model program that I use to model how people worked. Basically, given the task, it would simulate how long the person would take to solve the task, then switch to the next task. Interruptions could happen while other tasks were taking place. Great!

Except it didn’t work.

Remember that I had lots of data from the field study. So I had broken the data up into two parts – the modeling part (which I used to base my model) and the testing part (which I used to test my model). But when I tried out my model against the testing data, it wouldn’t predict anything very well. My model said that the admin could handle 3.0 interruptions per minute, but my testing data showed they could actually handle much less. What was worse, as I tested the model under higher interruption rates, the model results got worse and worse.

What went wrong? I sat and looked at the model for a while, going over the algorithms and methods used. Nothing seemed apparently wrong.

So I went back and looked at the original data. What was I missing?

I remember the moment clearly: I was watching one of the tapes, double checking my coding to make sure I got all the event times right when I noticed something that didn’t quite fit the model. The assistant was answering email when the phone rang. As she picked up the phone, her boss asked a question from the next room.  Ah ha! Key insight: Interruptions can happen during the setup to handle an interruption!

That’s when I realized that my model, which was based on the Newell & Simon problem behavior graph, had a fundamental problem: the transitions between the tasks (the boxes in the diagrams) were assumed to be instant and atomic. That is, there was no way for the model to show that there was a setup time (that was always just included as part of the task). But now that I realized that setup could be interrupted as well, I could break the setup out as a separate time as well.

 

And that led me to this diagram, what I now call the “parallel task behavior graph.”

Pbg4

Now, the setup time for a task is explicitly represented, and it can be interrupted as well.

I went back to my model, made the changes to the simulator, and voila, the model now matched the testing data quite nicely.

From the sensemaking perspective, what happened to me followed the pattern quite nicely: I collected the data, then organized it by coding it and building a model. I then checked my work by validating the model against the test data. When the data didn’t fit into the model, I went back to the collected data, looking for another way of organizing it that would be better. That’s what led me to invent the “parallel task behavior graph” as a way of organizing this data. It’s not perfect, and it doesn’t explain everything about how people handle interruptions, but it does a decent job.

In particular, this model shows that as your interruption rate increases, eventually you start getting interrupts during your interrupts—and that’s when everything starts to fall apart. Your multitasking skills don’t matter at that point, and your ability to get anything done starts to fall apart.

But you get the idea… Sensemaking is in many ways a search for the right organization or the right way to represent what you know about a topic. It’s data collection, analysis, organization and performing the task.

And as all the comments are alluding, the more you know about the process of sensemaking, the better sensemaker you’ll be.

 

Tomorrow: Summary of the comments readers have sent in.

Bydan_3

Posted by Dan Russell on January 25, 2007 | Permalink

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d83451b44369e200d83463995c69e2

Listed below are links to weblogs that reference Sensemaking 3:The search for a representation:

» Must Read List from Exodus Development
Here is my must-read list: [Read More]

Tracked on Feb 10, 2007 12:05:13 AM

Comments

Great piece. Might be interesting to hear what the "task set up" activity consisted of though. Perhaps it ties in with theories of "information chunking" whereby people process in chunks....

Posted by: PaulSweeney | Jan 25, 2007 11:42:19 AM

This post reminds me of Goldratt's book - "The Goal" - which is also focused on scheduling, albeit in the very different context of a manufacturing plant assembly line. When describing his Theory of Constraints (which focuses on optimizing bottlenecks for getting maximum throughput rather than maximum productivity), he goes through the same process of recognizing that "machine set up time" is an important factor affecting production capacity - scheduling programs of the past only looked at the capacity of an individual workstation without taking task-switching time/effort into account.

Sorry for the digression. This is a great series of posts! I'm looking forward to the next installment.

Posted by: NitinK | Jan 25, 2007 2:26:33 PM

http://hotelanyware.blogspot.com/

I remashed the 2000 Bloggers meme, It's muatted to my blog, and cut down to 150, a new z list.

Posted by: Ed Reif | Jan 25, 2007 2:58:19 PM

There is no Soviet Union anymore, but everybody remember those great victories and defeats. We trusted in idea and we made our history through great losses...
www.backinussr.com

Posted by: jimi | Jan 25, 2007 6:03:50 PM

This is a very "scary" (in my humble definition, which means "totally rocks") article...

I'm surprised that people actually do researches on this. :-)

Posted by: Hendy Irawan | Jan 25, 2007 9:07:50 PM

Ah, some of the graphics are not coming up, they are listed as on your local drive.

That is a very intriguing example of sensemaking. I have been in very interrupt rich environment and I am wondering if/how they handled being overloaded and later found something had "slipped through the cracks" and how that was handled?

Posted by: Stephan F | Jan 25, 2007 10:14:00 PM

I love getting your RSS feed but in past few days it's appearing in my reader (Thunderbird) as mangled HTML source code.
This passionate user is going blind trying to read your good stuff.

Posted by: Steve | Jan 26, 2007 2:24:52 AM

I suspect that one of the frustrations for people who deal directly with customers, whether internal or external, is the extent to which interruptions can dictate your day and totally hijack your time management. Customers' problems don't tend to happen when it's convenient for you to deal with them. You might have a to do list on which your customer's current problem is a low priority, but to the customer it's a very high priority and the customer is the customer, after all. The interruption must therefore be escalated to a higher priority than might otherwise have been the case, and thus frequently becomes the primary task.

This is how it goes every day for my husband (or so he tells me) - he calls it stomping out fires.

Posted by: Karyn Romeis | Jan 26, 2007 2:30:20 AM

Like the reference to the theory of constraints above. At one point in my life I did a lot of work with lean manufacturing principles, and the famous Toyota system. Within that system the set up time of each machine in a cell is a key constraint, and decreasing set up times the key contribution to increasing productivity. (apologies to those who have actually kept up with research in this area over the last ten years, I probably sound hopelessly out of date!)

Posted by: PaulSweeney | Jan 26, 2007 10:08:51 AM

One of the other things you can do is to also look at the opposite thing of what you are studying.
Here you are looking at interruption management, the opposite would be flow which need interruption management to figure out how to stay in flow longer. So books like Flow, and Getting Things Done and websites like 43folders are great sources since they of necessity talk about what you are looking at but from the perspective of reducing or eliminating it.
There is also the benefit of shaking up your thinking in looking at the same problem in a totally different way.

Posted by: Stephan F | Jan 27, 2007 10:58:55 AM

Heh, reminds me of the time I got lost outside of Nashville by accidentally turning onto an exit that turned into a highway, which in turn merged with another highway. I was screwed.

Kind of the same principle: compounded interruptions can get you very lost on our paths of productivity.

Posted by: Glen Stansberry | Jan 27, 2007 6:55:21 PM

Great story.
I find that this model fits my own learning model. For example, when I learn some new computer code or library, I build a representation in my head as I go. Many of the gaps are filled by intuition, because things that are designed for a purpose usually make sense. Pieces of the puzzle loosely come together. Then I would pick a specific area and testing my understanding, by verifying some detail in the documentation.

On the other hand, when learning in "directed" fashion, such as in a maths class, there was less room for jumping ahead. Instead the bricks were laid one by one for us by the instructor.
I still continue to learn on mathematics topics to this day, but found it much more challenging (possibly because I have a job ;-). Piecing the puzzle from various sources of information gets in the way of the learning itself. Even when information is already laid out in a linear format for you, in a tutorial for example, it often has some implicit expectation or dependency on prior knowledge from the reader.
The question is how could we structure our body of knowledge in a way that facilitates assimilation by the reader. The problem becomes harder as the sources of information multiply and incoherent (not aligned), as it is the case for the internet (in comparison with a class or a book).

Posted by: Julien Couvreur | Jan 27, 2007 11:16:37 PM

I love getting your RSS feed but in past few days it's appearing in my reader (Thunderbird) as mangled HTML source code.
This passionate user is going blind trying to read your good stuff.

Posted by: Bank zdjec | Aug 18, 2007 3:52:56 AM

The comments to this entry are closed.