The Case

An 87-year-old woman with hypercholesterolemia, osteoporosis, and mild dementia presented to the emergency department after a mechanical fall and was found to have a hip fracture. The patient had well-treated hypertension, good exercise tolerance, and no known heart disease. On physical examination, her vital signs were stable and she exhibited no evidence of delirium. She had normal renal function, a mild anemia, and an electrocardiogram (EKG) with evidence of old Q waves but no acute or dynamic changes. The patient was admitted for orthopedic surgery, and a formal preoperative assessment placed her at "low risk" (based on the Revised Cardiac Risk Index scale). The patient remained on metoprolol and simvastatin in addition to enoxaparin for deep venous thrombosis (DVT) prophylaxis until her scheduled operation.

The patient had an uneventful preoperative period but suffered a pulseless electrical activity arrest in the operating room at the time of wound closure. Though she responded to resuscitative measures and remained in the intensive care unit the following day, she developed acute renal failure and shock liver. Based on previously expressed wishes, the family requested that the patient be made comfortable with no further interventions. At the family's request, no autopsy was performed; however, providers felt that the patient suffered a massive pulmonary embolus based on an intraoperative echocardiogram that suggested right ventricular strain. The death was unexpected, particularly given the patient's low preoperative risk, and the family and providers struggled to explain the outcome.

The Commentary

This patient's death certainly warrants the label "unexpected." In a prospective regional study of 580 patients, no patient who received venous thromboembolism prophylaxis at the time of hip fracture surgery (as this patient received) developed a fatal pulmonary embolism.(1) Even if the patient in this case did not die of pulmonary embolism, she would be considered a "low-risk" surgical candidate based on the Revised Cardiac Risk Index instrument, implying a risk for all major complications (not just death) of less than 0.5%.(2) What then should we infer from her unfortunate and unexpected death: chance misfortune or a problem with her care?

The human mind generally favors causal explanations over chance, which would lead many to infer a problem in her care. Many hospitals traditionally review all maternal deaths precisely for this reason—with a maternal mortality rate in the United States of approximately 1 in 10,000, any death is unexpected.(3) Though less emotionally charged, many other Diagnostic Related Groups (DRGs [a method developed for Medicare as part of the prospective payment system to classify hospitalized patients into one of many "groups" that are expected to have similar hospital resource use because the patients within each group have the same condition or underwent the same procedure]) are also associated with low expected risks of death. In this context, "death in low-risk DRGs" has been developed as a potential indicator of patient safety problems. In fact, it is one of 20 "high-level hospital indicators" on the Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators (Table 1).(4)

One of the benefits of using death in low-risk DRGs as a patient safety indicator is that it can be measured using administrative data routinely collected for billing purposes. This is a far more efficient method of identifying potential safety problems than most other methods, such as reviewing charts. In the major chart review–based patient safety studies (5,6), nurses screened charts for "triggers" that prompted a more detailed review by physicians. These triggers included death, unplanned admission to intensive care units, and some 15 other potential indicators of quality problems. Among hospitalized patients whose records contained one of these triggers, approximately 20% experienced an adverse event.(7) Outside of research settings, even this event rate of 20% is probably too low: most hospitals cannot afford to have physicians review hundreds of charts just to have a 1 in 5 chance of finding potential safety problems. Hospitals need safety indicators that identify cases that will prove on investigation to involve true safety problems more often than not; they do not have the resources to investigate high numbers of false-positive triggers.

The efficiency of the ascertainment method and the rate of false positives represent just two characteristics to consider in choosing patient safety indicators, whether it is for local or national consumption. My colleague Dr. Alan Forster and I have considered the features listed in Table 2 as a framework for evaluating the advantages and disadvantages of specific strategies for detecting patient safety problems. Using this framework, "death in low-mortality DRG" would score highly on clinical importance and (in part for that reason) might also elicit reasonable buy-in from clinicians. But it would score low on sensitivity, as many important safety problems would go undetected, as they often do not result in death in any DRG, let alone low-risk ones. On the other hand, one would expect this indicator to do well in terms of having a low false-positive rate—it seems reasonable to posit that a substantial portion of unexpected deaths will involve deficiencies in care.

So, the key question becomes the extent to which deaths in low-mortality DRGs truly reflect safety problems. The literature review for the Patient Safety Indicators (4) described only one study (8) that quantified the prevalence of problems with care among patients who died in low-mortality DRGs. Among 8000 randomly selected deaths from New York hospitals, patients who died in low-mortality DRGs (defined as a risk of death less than 0.5%) were 5.2 times more likely than all other patients who died to have received "care that departed from professionally recognized standards."(8) In absolute terms, however, the rate of quality problems was only 9.8% among patients who died in low-mortality DRGs. Thus, for every 10 deaths in low-mortality DRGs, careful review may reveal no apparent problems in care in nine of them. In my judgment, based on the information presented, I would characterize the case presented here as falling in the latter category: the patient was appropriately risk stratified prior to surgery, she received prophylaxis for venous thromboembolism, and she underwent the planned procedure without any technical complications.

While the researcher (and purist) in me wants to emphasize this high false-positive rate associated with death in low-mortality DRGs—it seems unfair to identify 10 cases as being potentially due to substandard care when the evidence suggests that only one of them is—the clinician in me acknowledges that we often assiduously pursue diagnoses that have less than a 10% chance of being present in a given patient. Reviewing deaths in low-mortality DRGs falls within the norm of clinical practice insofar as it is a moderately expensive diagnostic process that targets a relatively unlikely condition. One can justify using an indicator like death in a low-mortality DRG as a screening test, that, when positive, leads an institution to follow up using a more accurate test, namely a thorough investigation of the case that occurs very soon after the death. However, I do not think one can justify reporting deaths in low-mortality DRGs as prima facie evidence of quality problems, as currently happens with the Patient Safety Indicators.

Ultimately, as a matter of local quality improvement and assessment, I come down on the side of the clinical perspective. Just as I routinely rule out conditions in my patients that are relatively unlikely (but important to find), I regard it as part of my professional responsibility to investigate potential problems with my patients' care, at both the individual provider and system level, even if the probability is relatively low. Using this logic, hospitals may successfully convince clinicians to participate in internal peer review processes driven by the detection of unexpected deaths, in the spirit of professionalism.

Hospital administrators must make it clear that indicators such as deaths in low-mortality DRGs are screening tests to prompt internal investigations for potential safety problems. The message should not in any way be conveyed that these events represent prima facie evidence of deficient care. While I think this message can be conveyed successfully within hospitals, the chance of success with the public and the media is low. In my view, death in low-mortality DRGs carries too high a potential for misunderstanding and stigma in the eyes of the public as evidence of egregious care to justify its use as a publicly reported indicator, especially given its relatively low yield.

Admittedly, and notwithstanding its low sensitivity and high false-positive rate, it is still difficult to shake the intuition that unexpected deaths such as the one that occurred in the present case represent a "canary in the coal mine." Given sufficient time, the hospital with multiple safety problems will likely have more deaths in low-mortality DRGs than the safer hospital, just as the casino with its slight statistical advantage will always eventually beat the average bettor. In any given year, however, a perfectly safe hospital may have some deaths in low-mortality DRGs and a terrible hospital may have none. Capturing even fairly substantial differences in safety between hospitals would require many years of mortality data.

Another concern is that culprit quality problems in cases such as the present one may be undetectable through chart review. For instance, a technical problem with surgery or failure to administer thromboembolism prophylaxis might be undetected by chart review and might even require interviews with the personnel involved. Since the time required to obtain data for patient safety indicators is often many months, such detailed investigations would be severely compromised.

The situation here is analogous to the perspective advocated by the modern patient safety movement: incompetent individuals may cause many adverse events, but most adverse events involve competent providers (working within poorly designed systems). If we design incident reporting and other patient safety systems to catch the small percentage of poor performing providers, we will demoralize the majority of competent providers who are trying to deliver good care. Similarly, if we design (and then publicly report) patient safety indicators to catch outlier hospitals with gross safety problems, we will demoralize personnel at the vast majority of hospitals whose average rates of deaths in low-mortality DRGs do not actually indicate significant problems with care. Forcing hospitals to investigate publicly reported patient safety indicators with high false-positives rates, rather than using such indicators on an internal basis only, will alienate hospital personnel and hinder patient safety efforts, much like the traditional "blame and shame" attitude within hospitals has traditionally hindered progress in patient safety.

Take-Home Points

• Patient safety indicators drawn from administrative datasets present the possibility of efficiently identifying cases with a high potential for involving problems with care.
• Although death in low-mortality DRGs has high intuitive appeal as an indicator of safety problems, investigation will reveal no deficiencies in care in the majority of such deaths.
• This high rate of false positives makes deaths in low-mortality DRGs problematic as a publicly reported patient safety indicator.
• Nevertheless, the yield of cases with safety problems is probably high enough to warrant thorough internal hospital investigation of all such deaths.

Kaveh G. Shojania, MD Canada Research Chair in Patient Safety and Quality Improvement Associate Professor of Medicine University of Ottawa

References

1. Todd CJ, Freeman CJ, Camilleri-Ferrante C, et al. Differences in mortality after fracture of hip: the east Anglian audit. BMJ. 1995;310:904-908. [go to PubMed]

2. Lee TH, Marcantonio ER, Mangione CM, et al. Derivation and prospective validation of a simple index for prediction of cardiac risk of major noncardiac surgery. Circulation 1999;100:1043-1049. [go to PubMed]

3. Berg CJ, Chang J, Callaghan WM, Whitehead SJ. Pregnancy-related mortality in the United States, 1991-1997. Obstet Gynecol. 2003;101:289-296. [go to PubMed]

4. McDonald KM, Romano PS, Geppert J, et al. Measures of Patient Safety Based on Hospital Administrative Data—The Patient Safety Indicators. Technical Review 5. AHRQ Publication No. 02-0038; August 2002. Available online at: http://www.ahrq.gov/downloads/pub/evidence/pdf/psi/psi.pdf. Accessed November 27, 2007.

5. Brennan TA, Leape LL, Laird NM, et al. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. N Engl J Med. 1991;324:370-376. [go to PubMed]

6. Thomas EJ, Studdert DM, Burstin HR, et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care. 2000;38:261-271. [go to PubMed]

7. Forster AJ, O'Rourke K, Shojania KG, van Walraven C. Combining ratings from multiple physician reviewers helped to overcome the uncertainty associated with adverse event classification. J Clin Epidemiol. 2007;60:892-901. [go to PubMed]

8. Hannan EL, Bernard HR, O'Donnell JF, Kilburn H, Jr. A methodology for targeting hospital cases for quality of care record reviews. Am J Public Health. 1989;79:430-436. [go to PubMed]

Tables

*A complete list of the AHRQ patient safety indicators, their definitions, and technical specifications can be found in Appendix E of reference 4.