Case Objectives

• List the goals of having order sets in the electronic health record.
• Describe the evidence supporting the use of order sets in electronic health records.
• Understand how modern digital technology may encourage a superficial analysis of information.
• Appreciate that order sets may encourage a mechanistic focus that may prevent a deeper consideration of clinical issues.
• List key principles in the optimal design of order sets.

The Case

A 55-year-old woman with a history of metastatic cancer of unknown origin was sent to the emergency department (ED) after a magnetic resonance imaging (MRI) scan of her brain (done for cancer staging) showed a right subdural hematoma with a very small (5 mm) midline shift. The patient was alert and oriented when she arrived in the ED, but she did report falling and hitting her head a few weeks before. Her vital signs were normal and she had a normal neurologic examination. A noncontrast head computed tomography (CT) scan showed the subdural hematoma was unchanged when compared to the MRI. She was examined by a neurosurgeon in the ED who recommended no acute intervention and repeat imaging in one week. She was admitted to a hospitalist service for observation.

She did well with no new complaints or complications and was discharged from the hospital the following afternoon (about 36 hours after admission). She left the hospital and went directly to a previously scheduled positron emission tomography CT scan, also being done for her cancer workup. On that study, the radiologist noted that the subdural hematoma had enlarged and the midline shift had increased to 11 mm. The patient was readmitted to the same hospital medicine team that had cared for her before. She was stable on admission with no neurologic complaints and a normal neurologic exam.

Unfortunately, the next day, her mental status deteriorated and a repeat CT scan showed an enlarging subdural hematoma. She was taken to the operating room for evacuation of the blood. The surgery was uncomplicated, but postoperatively she developed sepsis secondary to a hospital-acquired pneumonia. Despite maximal efforts, the sepsis progressed to multi-organ system failure. Care was ultimately withdrawn and she died peacefully 10 days after admission.

The hospital medicine service routinely reviewed all deaths on their service. In reviewing the death, the case review committee discovered the patient had been given low-molecular-weight heparin (LMWH) for venous thromboembolism (VTE) prophylaxis during the first admission. They spoke with the admitting provider for that admission. She did not realize she had prescribed the LMWH and stated she certainly didn't intend to prescribe it in light of the subdural hematoma. She stated that she was just "clicking boxes" on the admission order set in the electronic health record (EHR), and that she was used to ordering it as nearly all patients she admitted met criteria for VTE prophylaxis. She did recall being distracted by another complex patient at the time of entering the admission orders.

The case review determined that ordering the LMWH was a medical error that may have contributed to the patient's death. They realized there clearly were many benefits to using order sets in the EHR, but wondered about the risks associated with order sets and how best to balance the risks and benefits.

The Commentary

by Anthony C. Easty, PhD

In many industries where professionals provide important monitoring and intervention activities, the use of checklists has become routine. A commonly used example is in aviation, where completion of a preflight checklist containing a list of tasks to be performed is mandatory. The purpose of the checklist is to improve flight safety by ensuring that no important tasks are forgotten. Failure to correctly conduct a preflight check using a checklist is often a major contributing factor to aircraft accidents.(1) Aviation checklists are designed to meet the specifications of different types of aircraft and the environments in which they are operated. The expectation is that each and every item will be checked and complied with to ensure that the flight proceeds as safely as possible.

Similar thinking has led to the development of checklists in medicine. One type of checklist used in health care is order sets, which aggregate orders or steps for a given condition (often in the electronic health record [EHR]). The intent is to ensure that all appropriate tests are ordered, and the order set often extends to collecting vital signs and ordering medications as well. Order sets may also guide providers to best practices. When the majority of interventions on a list are always appropriate, there is a tendency on the part of the user to assume that every listed intervention is desirable due to the psychological phenomenon known as confirmation bias.(2) Just as in aviation, the aim is to ensure that every patient receives optimal care, and that all relevant tests and interventions are ordered.

Order sets are now common in health care institutions in many parts of the world, often implemented electronically as part of the larger EHR system. Order sets have found widespread acceptance since they prompt us to ensure that every step in a diagnostic or treatment pathway is followed, and they are now applied in many areas of modern health care. For example, the American Academy of Family Physicians has published 30 standardized hospital admission orders, covering a very wide range of conditions.(3) Order sets have resulted in improved outcomes such as reduced mortality, readmissions, and length of stay, and this, coupled with the recognition that it is very difficult for any care provider to remember every desired step in a course of diagnosis or treatment, has led to their widespread adoption.(4)

Although similarities exist between checklists used in aviation and order sets used in health care, this case highlights one of the key differences. Aviation preflight checklists contain a series of checks, all of which must be accurately followed prior to takeoff. Order sets, on the other hand, contain a series of boxes that might be selected by the care provider rather than a complete list that must be followed without exception. This key difference is clearly necessary as each patient is unique and can present to health care professionals with a wide range of symptoms. The preflight checklist model, with its rigid, unvarying list of tasks to be completed is unlikely to be workable or appropriate for every patient.

In principle, all health care providers understand this, and, in a perfect world, users presented with a range of checklist options would always choose only those that are appropriate for each patient. For a patient presenting in the emergency department with a subdural hematoma, prescribing the anticoagulant heparin is strongly contraindicated. Indeed, it can be inferred from this case description that the care provider herself realized the inappropriateness of this order at the time of the review. How then did it happen, and what are the implications for the use of checklists and order sets in medicine?

In a study on chemotherapy order errors using computerized physician order entry, the rate of problematic orders (those requiring significant clarifications) fell from 30.6% with handwritten paper orders to 12.6% with preprinted paper orders, and fell even more (to 2.2%) with computerized provider order entry (CPOE) order sets. Further, the incidence of errors capable of causing harm dropped from 4.2% with handwritten paper orders to 1.5% with preprinted orders to 0.1% with CPOE order sets.(5) This generally encouraging result was qualified by the authors as follows: "An important finding is that problems and errors are still possible even with CPOE, sometimes occurring in unique ways.?In at least a few cases, the CPOE system lent authority to errors that otherwise might have been questioned, a phenomenon that has been previously described and labeled 'cybernetic mysticism,' defined as 'the belief that computers are almost magical devices.'" In summary, though a net benefit was realized from CPOE use, we see that users occasionally will cede judgment to the computerized order since it has the appearance of infallibility and correctness.

Some have argued that the easy access of information on the Internet, and the ability to move from topic to topic with just a few keystrokes, leads us to take in chunks of information in a swift stream and often at a superficial level.(6) We may gradually be losing the ability to read and think in depth about an issue, causing us to jump to rapid conclusions based on a quick perusal of digitally sourced information. We tend to skim digitally sourced material rather than reading it in depth and pick out those points that seem salient to us, often seeking those that confirm our opinions and biases. As most modern EHRs present clinicians with voluminous information and allow them to quickly move from one task to the next, it is easy to superficially consider some complex issues and check boxes without deep thought. In this case, the admitting provider stated that nearly all patients she admitted met the criteria for VTE, and she recalled being distracted by another complex patient at the time of entering the admission orders. The combination of large volumes of information, inappropriate application of her usual practice (giving VTE prophylaxis), and distractions set her up for a medical error.

It is tempting to step back at this point and say that any competent care provider should have realized that prescribing low-molecular-weight heparin for a patient with a subdural hematoma is contraindicated. At face value, this statement is true enough, but it assumes a calm and ordered environment, and a deliberate and careful assessment of the patient's needs. In sum, this statement fails to consider the environment in which tasks are being performed and the influence of factors such as competing demands for attention and the use of tools such as order sets. Sadly, such instances of misprescribing that have led to dire outcomes are not uncommon.(7,8) The desire to punish the involved providers is an understandable human reaction, but doing so does nothing to address the root causes of these incidents and so the underlying factors remain in place, ready to reemerge in the form of another incident in due course.

When my colleagues and I conducted a simulation study on the double-checking of list-based medication orders, we wanted to determine whether a caution at the end of the order stating, "Stop! Knowing all that you know, does this order make sense to you?" would prompt participants to step back from the mechanistic task of checking an order and detect an error. It failed to do so—this general reminder did not increase rates of error detection. We concluded that it was unrealistic to expect busy clinicians to mechanistically review all their medication orders, and then to change cognitive modes to consider the appropriateness of their order—notwithstanding a prompt built into the EHR. In those situations in which such a big picture view of the case is needed, we came to believe that it may be necessary to separate this process from the more mechanistic task of completing the checklist.(9)

What then can we conclude from this? Order sets and other types of checklists have important roles to play in modern health care. They help to keep us on track when we are busy and dealing with multiple issues at once. At the same time, they place us into a mode where we are unlikely to question the appropriateness of the suggested items in the orders, even if we have to select them one by one. If the usual practice for most patients is to "check all the boxes," then it is unlikely that we will deviate from it even when inappropriate, as we saw in this case. This places a great responsibility on the designers of tools such as order sets. When designing a tool that is going to be followed in a mechanistic fashion, we need to make sure that if all steps on a list are followed, there is minimal possibility for an adverse outcome with any patient we treat. Even if the order set in this case contained a warning beside the order for low-molecular-weight heparin stating, "Stop! Does the patient have active bleeding?," the study above (9) suggests that it is unlikely that the user would notice and would likely prescribe the medication regardless. We thus need to design systems of orders and checklists that reduce the risk of an adverse outcome, just as the aviation preflight checklists contain a list of items to be followed in all circumstances without leading to adverse incidents.

In this case, a "slip" on the part of a provider using a standard order set led to a serious adverse event. The case highlights the hazards of the EHR and how modern engagement with digital technology may drive all providers to only superficially engage in the content. Application of optimal design principles can potentially mitigate the risk of error.

Recognizing that order sets offer many benefits to outcomes in health care, there are steps that can be taken to optimize their design. A description of the steps that should be taken to develop, design, implement, approve and maintain order sets is available.(10) In this case, a sensible redesign might be to separate the current lengthy order set such that essential items are prominent and optional (i.e., potentially contraindicated) items are clearly, prompting users to consider whether these optional items are necessary on a case-by-case basis. Such redesign should be simulation tested to ensure that it will be used safely and effectively.

Take-Home Points

• Checklists and order sets have become commonplace in electronic health record systems, covering a range of diagnostic tests and therapeutic interventions.
• Users tend to follow all elements of an order set or checklist due to an intrinsic bias to accept the options that the system is offering them, particularly when they are busy and the default setting of checking all the boxes is usually correct.
• System designers and implementers need to be aware of the propensity to "tick all the boxes," and should design groupings of lists that are intrinsically safe for all situations, recognizing that users tend not to apply critical thinking when presented with a series of boxes to tick.
• Published guidelines on the development of standard order sets should be followed.

Anthony C. Easty, PhD Adjunct Professor, Institute of Biomaterials & Biomedical Engineering Senior Fellow, Massey College University of Toronto

Faculty Disclosure: Dr. Easty has declared that neither he, nor any immediate member of his family, has a financial arrangement or other relationship with the manufacturers of any commercial products discussed in this continuing medical education activity. In addition, the commentary does not include information regarding investigational or off-label use of pharmaceutical products or medical devices.

References

1. Dagani F, Weiner EL. Cockpit checklists: concepts, design, and use. Hum Factors. 1993;35:345-359. [Available at]

2. Nickerson RS. Confirmation bias: a ubiquitous phenomenon in many guises. Rev Gen Psychol. 1998;2:175-220. [Available at]

3. Wiprud R. 30 standardized hospital admission orders. Fam Pract Manag. 2001;8:49-51. [go to PubMed]

4. Krive J, Shoolin JS, Zink SD. Effectiveness of evidence-based pneumonia CPOE order sets measured by health outcomes. Online J Public Health Inform. 2015;7:e211. [go to PubMed]

5. Meisenberg BR, Wright RR, Brady-Copertino CJ. Reduction in chemotherapy order errors with computerized physician order entry. J Oncol Pract. 2014;10:e5-e9. [go to PubMed]

6. Carr N. Is Google Making Us Stupid? The Atlantic. July/August 2008. [Available at]

7. Drug overdose "was manslaughter." BBC News. September 25, 2009. [Available at]

8. Fatal chemo overdose prompts Alberta reforms. CBC News. May 8, 2007. [Available at]

9. White RE, Trbovich PL, Easty AC, Savage P, Trip K, Hyland S. Checking it twice: an evaluation of checklists for detecting medication errors at the bedside using a chemotherapy model. Qual Saf Health Care. 2010;19:562-567 [go to PubMed]

10. Grissinger M. Guidelines for standard order sets. P T. 2014;39:10-11,50. [go to PubMed]