Editor's note: Enrico Coiera, MB, BS, PhD, is a professor and director of the Centre for Health Informatics (Australian Institute of Health Innovation) at the University of New South Wales. Dr. Coiera has researched and written about clinical communication processes and information systems. We spoke with him about how interruptions and distractions in the clinical environment influence patient safety.

Dr. Robert Wachter, Editor, AHRQ WebM&M: How big a problem are interruptions and distractions in the world of patient safety?

Enrico Coiera: Interruptions happen every day to every clinician, nurse, and doctor. It's become clear over the last decade that in some clinical settings, not only are interruptions frequent but they're also a patient safety risk. We're not saying every interruption is a bad interruption, but we do know that for certain places and times, they can lead to significant patient risk.

RW: Which areas have the highest risk?

EC: The challenge is in understanding why interruptions can cause harm. They essentially disrupt working memory with the consequence that you can forget to do what you're about to do or, very oddly, repeat the task you've already done. For example, you might have administered a medication to a patient and then been interrupted, but because you do the task so often any individual act is not particularly memorable, and you give the same dose again.

An interruption can also result in the interrupted task being incorrectly executed. My classic example is driving to the shops on a Saturday morning, when the cellphone goes off. By the time you've finished your call, you don't find yourself at the shops but instead in the car park at work. The call has occupied your attention and because you are distracted, you follow an initially similar, but ultimately wrong plan, which dominated because it is well rehearsed and easily enacted.

Psychology tells us certain variables predict higher risk of memory disruption. Probably the most important is working memory load, which is governed by many things you have to remember at any one time, and how complicated each is. If a task involves mental calculation or many steps, then your working memory load will be high and thus put you at more risk of disruption by interruption. Another issue is if the interrupting task is very similar to what you're currently doing, this also increases the risk that it will lead to memory disruption.

Another variable is the interruption modality—is the interrupting task visual or auditory for example? Those modalities are processed differently in the brain. It's quite possible to be talking about one thing and looking at a something different, to multitask. If the interruption uses the same modality however, like two visual tasks, then you have more chance of one disrupting the other. The final variable is how good you are at the task you're doing. If you have a lot of practice and experience, your opportunity to do well with interruption increases.

So, the kinds of clinical tasks that we worry about include administering medications, preparing injectable chemotherapy and IVs, induction of anesthesia, or putting a central line in. They're all tasks for which cognitive and task variables suggest that they are at risk of disruption by interruption.

RW: In terms of the tasks being interrupted, is it worse if the task is a mindless behavior? Something rote and fairly automatic? Or is it worse if it's something that you have to think very hard about, with a high cognitive load associated with it?

EC: They both have their special problems. If you're performing a high cognitive load task, then you are at very much risk of disruption. If it's rote, for example if there's a sequence of tasks, then a lot depends on where in the task sequence you get interrupted. If you get disrupted towards the end of a task sequence, you may experience what we call a post-completion error, which means you forget to do the last element in the sequence—the cleanup bit. An easy example is using a cash machine: you put your card in the machine and then pull the cash out. You have completed the primary task, which is to get the cash, and you walk away leaving the card in the machine—omitting the post-completion or cleanup step. They've now designed cash machines so that you have to pull your card out before getting any cash, so you cannot make a post-completion error. Similarly, you might complete a clinical task but then forget to write the notes or otherwise communicate what you have done because of disruption.

One thing that mitigates risk is how much of the primary task context is available when you resume it after interruption. We did some experiments on the impact of interruption on online order entry for medications and showed that interruption had limited to no impact. We think this is because after an interruption, when you return to the computer screen, it typically shows you where you left off. You can get back into the task. You're reminded by the cues on the screen where you were in task sequence, so you can recover very easily. It's very different if you're walking down a corridor and don't have a memory aid. We know that your control over the short time before you get interrupted is very important in such circumstances. If you see somebody about to talk to you and can say to yourself, "I was about to do x. I'm going to just rehearse that in my mind now," then attend to the interruption, you have a good chance of returning to x. But if you're interrupted with no control, you don't have that short moment to remind yourself what you're up to, and you thus have more chance of forgetting.

RW: I was interested in what you just said about the individual's capacity to get out of an interruption safely. In that example of getting pulled aside in the hallway, the individual is doing the equivalent of putting a rubber band around a finger as a reminder that he or she was interrupted and to remember to do the next thing. I imagine people probably feel that they have some capacity to protect themselves against this. The question is: do they or are we kidding ourselves?

EC: We're often kidding ourselves. The mental model to have is that we can carry about five primary tasks in working memory. (We used to think it was seven, but it appears to be only five.) That's your mental to-do list, and when you get interrupted you can lose any item from it. But you don't know which item you have just lost. However, mentally rehearsing an item just before an interruption increases its chance of survival, as does creating an external cue or reminder. In aviation for example, when they're going down a checklist, they keep their finger on the clipboard next to the item they are up to, so that whatever happens to them, they know where their finger was in the task list. To cope with interruptions, we thus need a set of strategies that create cues to where we are in the world and where we are up to—even a scrap of paper will do. If you're in the middle of a complex task, the obvious one is a mental calculation, it's very good to have written something down as you go. So, without some form of obvious cueing or without a chance to actually actively rehearse, "I was about to go and see Mrs. Jones in bed three," then your chances of being disrupted are quite high if you're busy.

RW: Let's say after listening to you someone says, "I not only want to build better systems, but I want to do whatever I can personally about this." Would you recommend that when they get interrupted in the middle of a task, before they go in and see a patient, they automatically say to themselves, "I was just interrupted," and they pull an index card out of their pocket and write down where they were, or is that just not going to work?

EC: That's a good idea, but it's too late if you begin to actively engage with the interruption task. You need to do it ahead of the interruption. We can be taught a number of different interruption-handling strategies. The first is to say "no," signaling unavailability. You can just put your hand up or shake your head, "I'm on the phone," that sort of thing. That's perfectly legitimate if you know what you're doing is primary and important. The second thing you can do is to quickly note down the state of the world you're in, which might be where you were in a task, where your finger is on the clipboard, or just rehearse in your mind where you're up to. If you do any of those things just ahead of the interruption task coming in, then you're fine. It's harder to recall afterwards, especially if you're very busy.

RW: We've already given examples from aviation in the cockpit and from the bank machines. Any other examples you've seen from industries outside of health care? How can interruptions be mitigated?

EC: Very interesting work has come out of the design of computer systems and human–computer interaction. This is all around being able to record the state of the world at the time you're interrupted, being able to design systems that are interruption-proof in some way. They teach us a lot about the sorts of systems we can eventually put in place to minimize distraction. For example, computer designers know that it is a really bad thing to have an alert window pop up and interrupt you on the computer, sometimes even demanding that you click on it before you can return to what you were doing. Luckily it doesn't have to be that way, and there are ways of designing computer interaction that are not interruptive or distracting and allow you to switch tasks. Nice computer environments put in action what psychology tells us.

RW: How does teaching from that world inform the issue of alerts and notices; for example, that we're about to give a patient a medicine to which they're allergic? It sounds like thinking in the field has changed about how to organize that kind of alerting.

EC: I don't believe that the design of modern health IT systems actually reflects this literature and understanding very well at all. Most modern e-health systems presume that the user is not doing anything else and that the screen has their full attention. I think you have to design clinical systems so that they can be used in messy clinical environments, with the understanding that the person using your system might be distracted, turn away to do something else, and then come back to the screen. There needs to be the equivalent of a digital finger pointing to where you were in your onscreen task sequence. Design the screens so it is very easy to identify cues to what you were doing and to quickly resume it post-distraction. The design research is clear, and many industries are ahead of us. I'm not that excited by what I see in terms of modern day clinical systems.

RW: When you say that our working memory has gone down from seven to five items, is this like global warming—our memories are going bad because we've become dependent on the numbers in our cellphones and on Google—or is it just better research tells us that the true number was five all along?

EC: It was five all along. However, when psychologists say there are five items in working memory, that doesn't mean five facts—they mean five pointers to different "chunks" of stored memory. If you're a very experienced clinician, each chunk could be a very complex process, meaning that you are functionally able to store an awful lot in five working memory items. A more junior clinician will not have that degree of compiled expertise and carry much smaller identifiable chunks. This means their effective recall using working memory is smaller. That is one reason experience and practice makes you much more capable of dealing with interruptions.

RW: Let's talk about risk mitigation strategies that have been tried in health care. I've heard about nurses wearing signs that basically say, "Leave me alone. Don't talk me to me, I'm mixing meds." I've heard about people putting tape on the ground to create a "zone of quietness" around them. What have you seen and what works?

EC: There are three separate classes of intervention that we can try.

The first is to simply to reduce interruption rates in a unit. It's like putting fluoride into the water. Let's only interrupt when it's appropriate. To do this we first have to ask: Why do we interrupt in the first place? Often it's to seek information to execute a task, but the task may not be urgent, and so can be delayed until the individual we want to talk to is not busy. The first and probably most powerful intervention to reduce interruption rates is to simply educate staff that they should be considerate about where, when, and why they interrupt each other. Another intervention makes interruption unnecessary by providing alternate information sources. If for example certain requests for information are frequent in a unit, then one can create an alternate information source to avoid having to ask the question. Often people who study organizations talk about tacit knowledge, which is the knowledge on how to get things done. When I was a young doctor, when you newly joined a medical team, you would often be given a little worn notebook that included all the special rules for dealing with the senior doctors you're working with and all the phone numbers to carry out the special tasks associated with that team. When I first looked at interruptions about 15 years ago, I studied a hospital in the UK, and the most common source of interruptions were "how to" questions, such as: "Who do I call to organize a venogram? How do I do x? Who's on service today?" Those bits of information are very easy to collate, so it removes the primary generator of interruption. All this information can now be captured in information systems. Being able to provide the information that people interrupt each other for in another place, perhaps using a team or unit wiki, can be very powerful.

Another interruption reduction strategy that works really well in other industries is to move from education to enforcing formal rules for both preventing interruption and enabling interruption. In airline team training, they try to deal with status asymmetry, so that a flight attendant is empowered to interrupt the pilot when he or she believes there's risk to the plane. We also see such status asymmetry playing out in health care where junior staff don't feel empowered to raise concerns despite the risk of patient harm. We should have rules for when not to interrupt, but also to empower our people to interrupt because the last thing anybody wants to do is to prevent crucial communication from occurring.

The second class of interventions signal, "I'm about to perform a high-risk task, and you shouldn't interrupt me right now because it might cause harm." This is the notion of the sterile cockpit, the "no interruption zone." There are lots of ways of doing it. As you said, when administering medications, one can wear a bright color vest that says, "Medicine Administration Occurring," or put up a sign or tape around you. All those things can work. The closest setting to the cockpit in health care is when we execute procedures like surgery or insert lines. It should be pretty easy to train people that the doctor is putting the line in, so wait until that is done.

The third set of interventions is to design work and workflow to be interruption-tolerant. We work in an environment where there are interruptions; we know we cannot avoid them. So let's put in handling strategies and processes that minimize their impact. Let's educate staff in basic interruption-handling strategies, empowering them to say no, or to rehearse, or to write things down. Record cues in the environment to allow people to return to the interrupted task knowing where they were. Whether it's in the digital space, having a computer system that shows you clearly what you were doing, whether you just scribbled two words down on a whiteboard or we're carrying handheld devices that can catch a written or a voice note. There are lots of opportunities for technology to do that. I often say that there is no universal answer because the risks for interruption vary with task, so each clinical setting has its own risk profile.

RW: I wonder whether there are individual characteristics. Some people love to have background music while they're doing complex tasks like surgery and others want complete quiet. Do you have to sort out individual preferences and the way individual brains work?

EC: I guess so. What we're talking about are very hardwired basic attributes of the human brain. So a lot of people say "I'm a good multitasker," presuming that they're somehow better than others, but the evidence is usually against that. Our perceptions of what we can and cannot do and what we actually can and cannot do are very different. Just because somebody likes music when they're operating doesn't mean that it is a safe thing to do. So if you're doing surgery, which is visual and tactile, you have a free auditory channel. This notion of modality allows you to listen and not be too disrupted. But if someone were to interrupt you with an auditory interruption then you'd be in trouble. Maybe in 10 years time listening to Bruce Springsteen while doing surgery will be like smoking.

RW: We ran a case on AHRQ WebM&M about this, in which someone was about to do something on rounds and a text message came in saying there's a big party tonight. Of course the person answered the text message and went back and forgot to order the medicine and it turned out to be a crucial medication. So it brings up this issue of smartphones and the ubiquity of technology. You've mentioned that you could pull it out of your pocket and make a note to yourself. But with the number of inputs, the blurring of the line between professional and personal communications has become tremendous. How do you think this has all changed in the world of smartphones?

EC: This is a very important issue. These devices have great potential to be interruptive. The example you gave is important and it could have been a message about a party or it could have been a message about a patient in another ward, it really doesn't matter.

But there's nothing inherent in smartphones that means they must interrupt you. We haven't adjusted the settings appropriately for the clinical environment. It's up to us to configure them in the right way. I don't understand why for example, when e-mail comes in, which is an asynchronous, non-interruptive form of communication, people tolerate an interruptive alert that makes the machine go "bing." I'm particularly disturbed by the number of smartphone applications, which now as default trigger alerts for the most trivial of events.

It's very important to recognize that these devices are built for the consumer world; while they're very powerful, when we bring them into a work setting, we need to configure them so that they don't interfere with work. Turn off alerts and reminders. If you're in a clinical setting where things are particularly at risk, then there should be strict rules about when you can and cannot use certain devices or functions.

RW: The question is this balancing act of what if it was about a patient in another ward who was sick? We have this culture such that people are hypersensitive about interrupting someone, but somebody needs to be interrupted. You've mentioned rules for interruptions, but how hard is getting this balance right?

EC: That's a very important cultural issue. For the last decade and a half whenever I talk about interruption, the first question I get from clinicians is, "Are you saying that I cannot communicate on behalf of my patient because interruptions are a crucial way for me to get the message across. The patient might be unwell." And the answer is that you must communicate when you have concerns. As we know one of the most important rules for activating medical emergency teams is concern by clinicians. So if you're worried, communicate. But most of today's interruptive communication is not in that class.

RW: What got you interested in this in the first place?

EC: In the early 1990s when there was an enormous push toward electronic records, I went to observe clinical work. What became clear was that, despite certainly very important reasons to support record taking with information systems, most communication happens face-to-face and verbally. That was not really understood in the literature. When you look at what happens face-to-face, you quickly notice the extent and toll of interruptions, multitasking, and all manner of things that had never been explored before in the informatics literature. It's very clear now that if you talk about supporting information needs in health care and you're only thinking about the electronic record, then you're missing probably 80% of the information space. The biggest information repository in health care sits in the heads of all the clinicians that you work with, and the biggest network is not the computer network but the network of conversations. What happens in health care is essentially communication-based and person-to-person. Computational tools should support that; they shouldn't dominate.

RW: What does this look like 10 or 15 years from now? Do IT tools get sufficiently mature and nuanced that in fact it's no longer 80% verbal but it's 50/50? Or do you think it will always be that this exchange of verbal data is too hard to encapsulate in an electronic system?

EC: I think what will happen is that we'll increasingly be able to use computer systems for information-seeking conversations: How do I do x? Who does y? But most conversations are sense-making conversations; they're discussions, they're sharing information, and decisions about treatment. We will always want to have discussions about those things. They're not going to go away.

RW: I wrote on my blog about my sense with our implementation of Epic—which was that it needed a field where the user was instructed to say what the hell is going on. That in some ways Epic was forcing a very granular, problem-oriented, if x then y, record of what was happening with the patient, and we were beginning to lose the big picture. How do you make sense of this entire enterprise?

EC: If you ask me what I think about most of the e-health systems that we have today, I would say that they are 1980s constructs. They are very focused around the idea that the clinical record is primary and therefore all work needs to conform to the needs of the record. But that's a broken idea—technology demanding that work fits around its limitations. In my view, good information tools are communication-based and focused on clinical tasks not clinical records. If you ask me what the future looks like, I would say look to devices like Google Glass. Look to augmented reality where we have a very different way of engaging with information. The idea that I can put on a set of spectacles that display task-specific information as I walk through a ward is quite powerful I think. Imagine if before I touch a patient, "Please wash your hands," flashes up on the screen of the glasses because they know I have not been to a washbasin—that changes my interaction with the world. A lot of such intelligence can happen in the background once technology focuses on the physical context, the people around you, and the task at hand. The way we engage with each other at work doesn't have to be through forms and records. It can be entirely different. In my fantasyland, the EMR disappears into the distance and work-supporting technology becomes primary. I think the record should be an artifact that arises incidentally out of what you do. It shouldn't dictate what you do. We're seeing work at the moment dictated by technology, and it doesn't need to be that way.

Editor's note: Dr. Kronick has served as director of the Agency for Healthcare Research and Quality since August 2013 and will be stepping down from the role this month. He has conducted and coordinated research on policies relating to public health, health care delivery, health insurance, and health care financing programs. We spoke with him about AHRQ's efforts to develop measurements and implement improvements in patient safety.

Dr. Robert M. Wachter: Tell us what you see the largest issues in safety are in which AHRQ can make a difference.

Dr. Richard Kronick: AHRQ has been a leader in the patient safety movement since its inception. It's been nearly 15 years since the IOM's initial report on patient safety galvanized the movement. We have made substantial progress, as evidenced by the remarkable improvement in hospital safety over the past few years. As shown in a recent AHRQ report, in 2010 there were 145 adverse events for every 1000 hospitalizations; in 2014 that rate decreased to 121/1000. As a result of this 17% improvement in safety, there were 2.1 million fewer adverse events over the 2010 to 2014 period than would have occurred if the 2010 rate had remained unchanged, and, more importantly, 87,000 fewer deaths and close to $20 billion in cost savings. However, health care is not nearly as safe as it should be or can be. The rate of 121 adverse events per 1000 hospitalizations still represents substantial harm to patients caused by the delivery of care, and we are in the early stages of figuring out how to improve safety in nonhospital settings—particularly ambulatory and long-term care. Key challenges include developing evidence about how health care can be made safer, working with our partners inside and outside of government to make sure that providers and patients really understand that evidence, and working to create an environment in which there are incentives to overcoming barriers to implementing the evidence.

Much of our work has focused on improving safety in the hospital, but outside the hospital there are many challenges as well. So we're working on accelerating patient safety improvement in nursing homes and in ambulatory care—particularly focusing on advancing our understanding of diagnostic safety. We have made extensive progress in changing the culture in hospitals and other health care institutions and in reducing adverse events. There has been a lot of work and some really significant signs of progress, and there is a tremendous need for more work and many challenges to overcome in the future.

RW: What research has AHRQ done in measuring quality and disparities? For example, part of the challenge is how do you measure diagnostic errors? The measurement challenge becomes an obstacle as you try to figure out how to understand the scope of the problem and how to improve it.

RK: AHRQ publishes an annual report that describes many measures of quality, safety, and disparities. It is very valuable at the national level, and many of these measures drill down to the state or local level so that it becomes more actionable. The 2014 report showed progress in a variety of areas—most notably access to care. Recent estimates are around 20 million fewer people uninsured than just a few years ago. It also showed improvements in quality on a number of measures publicly reported by CMS. At the same time, the report shows that, across a broad array of measures, only about 70% of recommended care is delivered, and on some measures even less: for example, barely more than 50% of people with high blood pressure have it under control. And the report shows that there has been very little progress in reducing disparities, although the disparities data in the report did not include the effects of the 2014 coverage expansion. There's clearly still a lot of work to do.

As noted in the recent IOM report, we don't yet have good measures of diagnostic accuracy. In 2015, we released a Funding Opportunity Announcement describing our interest in funding grants that will make progress in reducing diagnostic errors, and part of our interest is in funding work that will develop better measures of diagnostic accuracy. We look forward to funding work that will make progress in meeting this challenge.

RW: In terms of individual measures, everybody's a little whipsawed these days. You hear people say, "We need more and better measures," and you also hear about measurement fatigue—"There are too many measures." How do you navigate that?

RK: CMS and private payers have recently made substantial progress by reaching agreement on a core set of measures to use when assessing physician performance. Full implementation of this agreement, which will take a few years, will make progress on the "too many measures" problem. However, CMS and private payers were necessarily limited to choosing core measures from among the measures that have been developed and validated, and it is clear that we do not have valid measures for many of the most important features of high quality medical care. For example, we don't yet have good measures of diagnostic acumen, or of the quality of a physician's advice about the benefits and harms of treatment options, or about the extent to which patients and families are meaningfully engaged in the decision-making process. In some of these areas, developing useful measures may be possible. We need further work to develop better measures, as well as to better understand inherent limits in our ability to measure much of what matters in medical care.

RW: Patient and family engagement has been a priority area for AHRQ. What are you doing in that area, and what resources have you developed?

RK: Patient and family engagement is crucial to improving quality and safety. AHRQ research, as well as what's been sponsored elsewhere, has demonstrated that quite strongly. To promote greater engagement, AHRQ has developed a guide to help patients, families, and health professionals work together as partners. The Guide to Patient and Family Engagement in Hospital Quality and Safety focuses on four primary strategies for promoting patient and family engagement in hospital safety. First, encouraging patients and family members to participate as advisors. Second, promoting better communication among patients, family members, and health care professionals starting from the point of hospital admission and continuing throughout the stay. Third, enhancing continuity of care by keeping the patient and family informed through nurse bedside change-of-shift reports, assuring that during shift changes it is not simply one nurse talking with another, but also involving the patient and family. Fourth, engaging patients and families in discharge planning throughout the hospital stay. This guide is a very good example of how our programs identify opportunities to help others understand scientific evidence and how to use it. This particular resource is focused on inpatient hospital care. We are working on developing similar resources for ambulatory care.

RW: Talk about some of the resources AHRQ has developed in patient safety.

RK: One of AHRQ's priorities is patient safety, and we are a leader in developing tools and resources to improve health care quality and safety. Let me name a few of them. The Comprehensive Unit-based Safety Program, or CUSP, toolkit includes training tools to make care safer by improving the foundation of how physicians, nurses, and other clinical team members work together. It builds the capacity to address safety issues by combining clinical best practices with the science of safety to ensure that evidence-based care is delivered. This tool has been used widely throughout the nation. A second tool, the Team Strategies and Tools to Enhance Performance in Patient Safety, or TeamSTEPPs, is an evidence-based teamwork system aimed at optimizing patient outcomes by improving communication and teamwork skills among health care professionals. It includes a comprehensive set of ready-to-use materials and a customizable training curriculum to successfully integrate teamwork principles into health care systems. We've trained many thousands of master trainers, who in turn train hundreds of thousands of health care workers in these communication tools and TeamSTEPPs. A third tool is the Re-Engineered Discharge toolkit, or RED, which was designed to help hospitals reengineer and improve their discharge process. The study modules and supporting materials will help hospitals become familiar with RED's process and components, determine metrics for evaluating impact, and learn how to implement RED.

We have been working on developing a medical liability improvement toolkit titled, Communication and Optimal Resolution, or CANDOR. The objective is to create a toolkit to assist hospitals in implementing a communication resolution program. Key elements of these programs include full disclosure of an adverse event, apology, and then fair and rapid compensation. These efforts are intended to reduce the risk of harm, improve patient?provider communication, and reduce the likelihood of litigation. We've seen remarkable success with these kinds of early disclosure communication and resolution programs. It has been challenging to take the success in institutions where there is a charismatic leader and real strong supporter for these programs and implement them more widely. And we're working on developing a toolkit to help implement them more widely.

RW: As you reviewed those tools it struck me there was a theme in many of them—that they had initially been developed, often with AHRQ's support, at a single institution and with a visible leader. Peter Pronovost comes to mind for CUSP, and Rick Boothman at Michigan for the liability one. How do you decide that a certain tool is ready for scaling? Maybe it worked so well in a given environment because of the single champion who was charismatic and sold it, but now you're trying to scale it across 50 states.

RK: It's a really good question. AHRQ is an evidence-producing agency; it's fundamental to what we do. We then try to make sure that the evidence is understood and used. We try to make evidence-based decisions on whether and when tools are ready for scaling. We do research to figure out what happens if we take this tool and try to implement it at many more institutions. Was it implemented successfully? Did we see the outcomes that we expected to see? And if not, then we make changes to the tool if that is what's needed to try achieve the desired outcomes. When we have evidence that suggests something works, then we work at broader scale implementation.

In the case of CUSP for central line infections, we had substantial resources to actually do quite a bit of that broader implementation ourselves, but in many instances, we are working within HHS, particularly with CMS, on broader scale implementation. So in the case of the patient and family engagement toolkit—where we again had some evidence that it actually was useful and worked—it was the Hospital Engagement Networks, which CMS funds as part of Partnership for Patients, that used that toolkit much more broadly. We have seen substantial progress in patient and family engagement in hospitals across the country.

As you note, one key problem in safety and quality is always figuring out whether an intervention that works in a couple of places can really be brought to scale and how to do that successfully.

RW: AHRQ has gotten into health IT in a major way over the last 5 or 6 years. What do you think you've accomplished? What are some of the unique challenges that you've confronted?

RK: Our goal is to develop and disseminate evidence to inform policy and practice related to how health IT can improve the quality of health care. We work on trying to figure out how health IT can be used to improve quality and how we can make sure that health IT is used safely. Our research is focusing on how health IT can be designed and used to improve care, particularly looking at workflow—how clinical decision support systems can be best used to inform decisions at the point of care. We have done seminal work on barcoded medication administration and computerized provider order entry. The work we did there established the body of evidence showing that these technologies can improve patient safety. We put the resources Congress gives us—approximately $22 million this fiscal year—into research and the development of evidence. But this research is clearly a small part of an enormous industry, with annual revenues in the neighborhood of $40 billion. It is challenging to develop evidence that will be useful in actually leading to improvements in how health IT is used. I think we have been successful in that challenge, but it is an ongoing and constant part of the environment.

RW: Talk about the political environment that you find yourself in. You're a political scientist by career. You're at a moderate-sized agency with considerable impact, but a lot of the work you do has to be mediated through or certainly influenced by other federal organizations and then foundations, delivery systems, IT vendors, and others. How do you navigate that?

RK: Carefully. Our mission statement is to produce evidence to make health care safer, higher quality, more accessible, equitable, and affordable, and to work within HHS and with other partners to make sure that the evidence is understood and used. That second phrase is a really important part of the mission statement. It's to remind everybody who works here that the production of evidence in and of itself doesn't do anything. We have to be working with partners to make sure that that evidence is understood and used. As you said in the question we're a moderate-sized agency, about 300 people, a $430 million budget for FY 2016—not as much as we would want, although not an inconsequential amount either—but a teeny part of a $3 trillion health care system with 800,000 physicians and 5,000 plus hospitals. We must constantly pay attention to partnerships and try to ensure that the evidence is understood and used. We have a tremendous leg up in that in that we are part of a department that's purchasing more than a trillion dollars' worth of health care. We have colleagues who look to us for the production of evidence. For example, efforts such as the Partnership for Patients Program are dependent on the evidence that we produce. We are constantly focusing on how to make sure that what we do changes policy and practice. To do that, we must work with our partners both inside and outside of government at every turn.

RW: Given that second part of your charge, the one that involves trying to identify what works and then be sure it's implemented, you must hit political walls. Some interest groups have vested interests in doing more of x or less of y. Sometimes the evidence supports that, and sometimes it doesn't. How do you navigate those waters?

RK: We need to be cognizant of barriers to implementation and to adoption of evidence, and we need to try to figure out how to overcome those barriers. In any environment (whether political or corporate) that has its own politics, you always need to consider the obstacles to accomplishing the goals of the organization and how to overcome those obstacles.

RW: Some people will say there's been no progress in safety and still quote a New England Journal of Medicine paper from 2010 showing no improvements in North Carolina hospitals. Is that just a compelling and easy to understand story? What's the obstacle to getting the good news out?

RK: There's a lot of information in the environment. The report we released in 2015 showing that hospital care has become much safer and that 87,000 fewer people died as a result, did receive very good coverage in all the major media, as did a similar report that we issued in 2014. President Obama cited the 2014 report in the rollout of the Learning in Action Network that was part of the Administration's delivery system reform effort. After he cited it, the fact checker from the Washington Post did quite a lot of digging to try to see where this number came from and whether it was right number. After quite a bit of digging, Glenn Kessler—who looks at what politicians say and rates them typically as "3 Pinocchios," "2 Pinocchios," or "1 Pinocchio"—gave this claim what he called the "elusive Gepetto checkmark." However, it is still the case that few people are aware of this remarkable result—I often ask audiences at talks I give whether they know that hospital care was much safer in 2014 than in 2010 and that 87,000 fewer people died as a result of the improvement in safety. Typically only a smattering of audience members will raise their hands. We need to work harder and smarter in communicating this information.

RW: As you prepare to leave AHRQ, what is your vision for the future of health services research?

RK: We will continue to support research that identifies the most effective ways to improve safety with a much more rapid cycle going from ideas to evidence to implementation. We've been making great progress. There is always frustration at the length of time, and we'll be working on trying to reduce that cycle time. Beyond safety, we'll be doing work on trying to support the efforts to figure out how to purchase health care more effectively. The mantra is that we want to pay for value, and we're making progress in trying to figure out how to do that. But there are also tremendous challenges in figuring out how to do that, and particularly figuring out how to construct incentive systems that catalyze providers' intrinsic motivations for improvement as opposed to becoming check-box exercises that have the danger of crowding out the intrinsic motivations for improvement.

RW: Do you see the EHR as a fundamental enabler of improving cycle time? We always hoped that once we had computers and could embed practices in them that it wouldn't be 7 years between an article and a change in practice, but that you could make that happen more quickly. I think most people are a little disappointed so far. Do you think that will eventually work out?

RK: I think it will. It will take concerted effort from the research side, the certification side, and in the development and vendor community to make that happen. There is a sort of natural impatience. I'm reminded of the introduction of the sewing machine: I'm told that when sewing machines were introduced, both workers and owners thought it didn't fit into their workflow, and it reduced productivity for a while. It does make sense that change would take some time. There is potential for improving the information available to clinicians and to patients and for having clinical decision support tools to deal with some of the tremendous uncertainty that providers and patients constantly face. The potential is clearly there, and I'm sure we'll be working on getting from the potential to actuality for many years.