Case Objectives

• Describe current issues with training clinicians to perform procedures.
• Understand how simulation can be used to ensure trainees are competent in procedures.
• Describe mastery learning and how it is relevant to procedural training.
• Appreciate that simulation-based mastery learning improves patient outcomes.

Case & Commentary—Part 1:

A 67-year-old woman with a history of cirrhosis who was status post cholecystectomy was admitted at midnight with worsening ascites and abdominal pain. As part of the evaluation of the abdominal pain (to rule out spontaneous bacterial peritonitis) and to relieve the symptoms of abdominal distention, the admitting team (comprised of a second-year resident and an intern) decided to perform a diagnostic and therapeutic paracentesis (removal of fluid from the abdomen).

The second-year resident had performed 6 paracenteses as an intern so felt reasonably comfortable with the procedure. While she had not taken a formal ultrasound training course, she had been informally taught how to use the ultrasound to identify the fluid to perform the paracentesis safely. The resident felt comfortable proceeding with the procedure with the intern, who had never done one before, so she did not call the night hospitalist, who was available to supervise procedures overnight.

Medical educators champion evidence-based medicine for everything except the methods used to educate. By neglecting advances in our understanding of how medical trainees learn, educators miss a major opportunity to improve graduates' skills and patient safety. Serious or fatal iatrogenic complications occur in 2.9% to 3.7% of hospitalized patients (1), and medical procedures are the second most common cause of these complications.(2,3) Many procedures in teaching hospitals are performed by unsupervised trainees.(4,5) For generations, a pardigm known as "see one, do one, teach one" was used to ensure unsupervised trainees were capable of performing procedures. This form of vicarious learning allowed trainees to perform procedures after watching one, and allowed teaching the procedures to other trainees after performing only one. The traditional "see one, do one, teach one" approach to teaching procedural skills exposes patients to substantial risk of harm from trainees who are not yet fully competent.

Safely performing and teaching procedures in academic hospitals can be challenging. While guaranteed faculty supervision might seem like an answer, it is unclear whether supervision of trainees prevents complications because not all internists and hospitalists who supervise trainees routinely perform procedures themselves.(6,7) In fact, trainees sometimes have more knowledge about the procedure than the supervisor. In the above case, we are not told whether the night hospitalist was adequately trained to safely perform a paracentesis. Therefore, we cannot be confident that requiring the presence of a supervisor prevents complications.

On the other hand, if the trainee was reliably competent, there is no reason to require supervision (aside, perhaps, for billing purposes). One could imagine that competence to perform a procedure safely and effectively is determined by the experience of the clinician (i.e., number of previous procedures performed). However, experience alone does not guarantee competence.(8) Studies show that the number of procedures performed does not correlate with competence for central venous catheter insertion (9-11), thoracentesis (12), lumbar puncture (13), or paracentesis.(14) If the 6 paracenteses previously performed by the resident in this case had been performed incorrectly, the procedure might continue to be performed incorrectly for the remainder of this physician's career. The resident could then propagate this incorrect performance through teaching interns and medical students.

In contrast, highly trained and skilled personnel who perform many procedures achieve low complication rates.(15,16) This highlights a central point: practitioners and educators must distinguish the major difference between experience and expertise. Experts reach the highest level of achievement for a given task and maintain that level of proficiency through deliberate practice over time.(17) Expertise, instead of experience, is a more useful predictor of high reliability and safe systems.(18) The number of procedures required to attain expertise varies widely and unpredictably. In the past, the American Board of Internal Medicine (ABIM) required residents to perform and document 5 procedures to be considered competent.(19) However, after data emerged showing internists in practice were not performing many procedures (6,7) and after realizing that 5 procedures were not enough to ensure competency, the ABIM dropped their competency requirement.(19) ABIM no longer requires competency for many bedside procedures.(19) Despite the lack of evidence of a threshold volume–competency linkage, many hospitals still use 5 procedures as the cutoff for clinicians to obtain privileges to perform internal medicine procedures. Self-assessment of skill or confidence does not correlate with the ability to perform a competent procedure (10,12-14,20) and should never be used to determine a clinician's capabililty.

Multiple studies show that traditional medical education methods do not yield competent trainees. A study of 40 graduating internal medicine residents showed that not one could adequately perform a simulated thoracentesis procedure (removal of fluid from the pleural space).(12) The residents' average test score on a skills checklist was 52% items correct.(12) This study was performed at the time when ABIM still expected graduating internal medicine residents to be competent in thoracentesis (requirements were subsequently changed). Failure of the "see one, do one, teach one" approach to procedure training was also confirmed in evaluations of graduating nephrology fellows' ability to insert a temporary dialysis catheter (11) and neurology residents' ability to perform a lumbar puncture (13)—both procedures are required for certification at graduation in respective training programs. It is clear that the old model of procedure training has been ineffective, places patients at risk for injury, and is inadequately preparing trainees for future practice and teaching.

Case & Commentary—Part 2:

With the ultrasound, the resident and intern found what appeared to be a safe place to obtain the ascites fluid, 8 cm below her cholecystectomy scar, and proceeded with the tap. After collecting about 700 cc of clear yellow fluid, the ascitic fluid became blood-tinged, and the catheter was removed. At the time, the patient did not complain of abdominal pain—she actually felt improved—and her vital signs were stable.

About 20 minutes later, the intern was called due to the patient's complaints of lightheadedness with a blood pressure of 70/40 mm Hg (down from 110/65 mm Hg). The patient appeared pale and was complaining of increased abdominal pain. Her abdomen was tense and tender. With concerns for acute bleeding into the peritoneum, the resident and intern began resuscitation and contacted the night hospitalist. Interventional radiology and general surgery were quickly consulted.

A stat angiogram revealed a bleeding omental vessel that was pressed against the abdominal wall because of an adhesion, likely secondary to the patient's previous cholecystectomy. The interventional radiologist was unable to embolize the vessel, so the patient underwent emergent laparotomy with removal of liters of clotted blood and ligation of the affected vessel. She had a prolonged course in the intensive care unit (ICU) but ultimately survived and was discharged to a rehabilitation facility 3 weeks later.

It is unclear whether further training or supervision could have prevented the complication in this case. Even highly competent operators occasionally injure patients. However, use of rigorous evidence-based methods for procedure training, such as simulation-based deliberate practice (17) and mastery learning (21) decrease the overall rate of procedure complications.(9,22) As an educational framework, mastery learning allows medical educators to ensure competency before trainees are allowed to work with actual patients. Mastery learning requires that all learners meet or exceed a minimum passing score on a skills test before the completion of training. This means that training time varies while education outcomes are constant.(21) In other words, a struggling trainee may require 10 hours to reach the same level of competency that another trainee achieves in 4 hours.

One recognized way to achieve mastery is through the use of simulation.(23) Extensive research documents that simulation-based procedure training increases the knowledge and skills of trainees while assuring competence (9-14,24) and improving health care quality and patient safety.(9,10,22,25-28) Evidence shows that using simulation to teach procedures, such as colonoscopy (25), laparoscopic cholecystectomy (26), advanced cardiac life support (27), and bronchoscopy (28), can improve patient outcomes.

Most published studies in simulation feature standardized training time and content without requiring a minimum standard of proficiency (mastery learning). Our research group documented how medical simulation training using the mastery learning model improves clinical skills for bedside procedures such as paracentesis (14), lumbar puncture (13), thoracentesis (12), and central venous catheter insertion.(9) Additionally, the risk of procedure-associated injury is significantly reduced using this technique for central venous catheter insertion.(9,22) Simulation-based mastery learning (SBML) for central venous catheter insertion improves skills (9-11), retention of skills (29), patient care (9-10), and outcomes (22), while reducing hospital costs.(30) In a recent systematic review, mastery learning was featured in a subset analysis as showing a potential benefit in learning outcomes.(31)

Seldom used in medical education in the United States, SBML's innovative approach may transform how educators train physicians. Traditional models of medical education use bell-shaped curves and 2 standard deviation criteria for passing. This results in many trainees passing examinations despite unacceptably low performance. For medical procedures and other clinical competencies, this is not acceptable because of the high stakes of patient complications and medical errors. SBML requires trainees to reach a high level of achievement on a simulator before being allowed to work directly with patients. Thus, learners truly become experts. The SBML minimum passing score is set at a high level by experts in the field that understand the implications of errors on patients. Moreover, the use of SBML reduces skill variation among trainees so poor performers are not applying their inadequate clinical skills to patients—the major problem with traditional training methods. Studies show small standard deviations (variation) in the posttest skill performance of trainees who have undergone SBML for lumbar puncture (13), paracentesis (14), central venous catheter insertion (9), and advanced cardiac life support.(24)

Some argue that medical resources should not be used to train internal medicine residents on bedside procedures because internists and hospitalists are no longer performing procedures. With internists' and hospitalists' busy schedules and the availability of Interventional Radiology (IR) physicians who perform a large volume of procedures (2,6,7), many clinicians feel that it is best for procedures to be performed in IR. Despite this reasoning, trainees are still performing the majority of procedures in academic hospitals. Although one may argue that patient care is better when experts perform procedures, there is no evidence that care is better when procedures are performed in IR instead of at the bedside by trained internists. In fact, patients are highly satisfied with bedside procedures performed by hospitalists and trainees (32), these procedures can be done safely (with proper training), and delays in scheduling an IR procedure can potentially increase hospital days and overall patient costs (unpublished data; submitted). In some instances it may be dangerous for a sick patient to leave the bedside to go to IR for a procedure.

In conclusion, medical educators need to ensure that future clinicians are properly trained and evaluated so that they can perform bedside procedures such as paracentesis. Direct supervision, the number of procedures performed, and self-assessment of confidence or competency do not predict the safety of bedside procedures. Therefore, SBML should be used to achieve this aim because it improves procedures skills, ensures trainee competency, reduces skill variation among trainees, and improves clinical outcomes with lower costs.

Take-Home Points

• Traditional training models for procedures are ineffective and do not produce competent clinicians.
• Simulation-based training improves the skills of medical trainees.
• Simulation-based mastery learning is superior to simulation alone because it ensures all trainees demonstrate competency on a simulator before working with actual patients.
• Studies of simulation-based mastery learning show how it improves patient outcomes.

Jeffrey H. Barsuk, MD, MS

Associate Professor of Medicine

Division of Hospital Medicine

Northwestern University Feinberg School of Medicine

Faculty Disclosure: Dr. Barsuk has declared that neither he, nor any immediate member of his family, has a financial arrangement nor other relationship with the manufacturers of any commercial products discussed in this continuing medical education activity. The commentary does not include information regarding investigational or off-label use of pharmaceutical products or medical devices.

References

1. Kohn LT, Corrigan JM, Donaldson MS, eds. To Err Is Human: Building a Safer Health System. Washington, DC: Committee on Quality of Health Care in America, Institute of Medicine, National Academies Press; 2000. ISBN: 9780309068376.

2. Duffy FD, Holmboe ES. What procedures should internists do? Ann Intern Med. 2007;146:392-393. [go to PubMed]

3. Brennan TA, Leape LL, Laird NM, et al. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. N Engl J Med. 1991;324:370-376. [go to PubMed]

4. Lucas BP, Asbury JK, Wang Y, et al. Impact of a bedside procedure service on general medicine inpatients: a firm-based trial. J Hosp Med. 2007;2:143-149. [go to PubMed]

5. Berns JS, O'Neill WC. Performance of procedures by nephrologists and nephrology fellows at U.S. nephrology training programs. Clin J Am Soc Nephrol. 2008;3:941-947. [go to PubMed]

6. Thakkar R, Wright SM, Alguire P, Wigton RS, Boonyasai RT. Procedures performed by hospitalist and non-hospitalist general internists. J Gen Intern Med. 2010;25:448-452. [go to PubMed]

7. Wigton RS, Alguire P; American College of Physicians. The declining number and variety of procedures done by general internists: a resurvey of members of the American College of Physicians. Ann Intern Med. 2007;146:355-360. [go to PubMed]

8. Choudhry NK, Fletcher RH, Soumerai SB. Systematic review: the relationship between clinical experience and quality of health care. Ann Intern Med. 2005;142:260-273. [go to PubMed]

9. Barsuk JH, McGaghie WC, Cohen ER, O'Leary KJ, Wayne DB. Simulation-based mastery learning reduces complications during central venous catheter insertion in a medical intensive care unit. Crit Care Med. 2009;37:2697-2701. [go to PubMed]

10. Barsuk JH, McGaghie WC, Cohen ER, Balachandran JS, Wayne DB. Use of simulation-based mastery learning to improve the quality of central venous catheter placement in a medical intensive care unit. J Hosp Med. 2009;4:397-403. [go to PubMed]

11. Barsuk JH, Ahya SN, Cohen ER, McGaghie WC, Wayne DB. Mastery learning of temporary hemodialysis catheter insertion by nephrology fellows using simulation technology and deliberate practice. Am J Kidney Dis. 2009;54:70-76. [go to PubMed]

12. Wayne DB, Barsuk JH, O'Leary KJ, Fudala MJ, McGaghie WC. Mastery learning of thoracentesis skills by internal medicine residents using simulation technology and deliberate practice. J Hosp Med. 2008;3:48-54. [go to PubMed]

13. Barsuk JH, Cohen ER, Caprio T, McGaghie WC, Simuni T, Wayne DB. Simulation-based education with mastery learning improves residents' lumbar puncture skills. Neurology. 2012;79:132-137. [go to PubMed]

14. Barsuk JH, Cohen ER, Vozenilek JA, O'Connor LM, McGaghie WC, Wayne DB. Simulation-based education with mastery learning improves paracentesis skills. J Grad Med Educ. 2012;4:23-27. [Available at]

15. McGee DC, Gould MK. Preventing complications of central venous catheterization. N Engl J Med. 2003;348:1123-1133. [go to PubMed]

16. Gordon CE, Feller-Kopman D, Balk EM, Smetana GW. Pneumothorax following thoracentesis: a systematic review and meta-analysis. Arch Intern Med. 2010;170:332-339. [go to PubMed]

17. Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2004;79(suppl 10):S70-S81. [go to PubMed]

18. Weick KE, Sutcliffe KM. Managing the Unexpected: Resilient Performance in an Age of Uncertainty. 2nd ed. San Francisco, CA: Jossey-Bass; 2007.

19. Internal Medicine Policies. American Board of Internal Medicine. [Available at]

20. Wayne DB, Butter J, Siddall VJ, et al. Graduating internal medicine residents' self-assessment and performance of advanced cardiac life support skills. Med Teach. 2006;28:365-369. [go to PubMed]

21. Block JH, ed. Mastery Learning: Theory and Practice. New York, NY: Holt, Rinehart, and Winston; 1971. ISBN: 9780030860737

22. Barsuk JH, Cohen ER, Feinglass J, McGaghie WC, Wayne DB. Use of simulation-based education to reduce catheter-related bloodstream infections. Arch Intern Med. 2009;169:1420-1423. [go to PubMed]

23. Issenberg SB, McGaghie WC, Hart IR, et al. Simulation technology for health care professional skills training and assessment. JAMA. 1999;282:861-866. [go to PubMed]

24. Wayne DB, Butter J, Siddall VJ, et al. Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practice. J Gen Intern Med. 2006;21:251-256. [go to PubMed]

25. Cohen J, Cohen SA, Vora KC, et al. Multicenter, randomized, controlled trial of virtual-reality simulator training in acquisition of competency in colonoscopy. Gastrointest Endosc. 2006;64:361-368. [go to PubMed]

26. Seymour NE, Gallagher AG, Roman SA, et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg. 2002;236:458-464. [go to PubMed]

27. Wayne DB, Didwania A, Feinglass J, Fudala MJ, Barsuk JH, McGaghie WC. Simulation-based education improves quality of care during cardiac arrest team responses at an academic teaching hospital: a case-control study. Chest. 2008;133:56-61. [go to PubMed]

28. Blum MG, Powers TW, Sundaresan S. Bronchoscopy simulator effectively prepares junior residents to competently perform basic clinical bronchoscopy. Ann Thorac Surg. 2004;78:287-291. [go to PubMed]

29. Barsuk JH, Cohen ER, McGaghie WC, Wayne DB. Long-term retention of central venous catheter insertion skills after simulation-based mastery learning. Acad Med. 2010;85(suppl 10):S9-S12. [go to PubMed]

30. Cohen ER, Feinglass J, Barsuk JH, et al. Cost savings from reduced catheter-related bloodstream infection after simulation-based education for residents in a medical intensive care unit. Simul Healthc. 2010;5:98-102. [go to PubMed]

31. Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA. 2011;306:978-988. [go to PubMed]

32. Mourad M, Auerbach AD, Maselli J, Sliwka D. Patient satisfaction with a hospitalist procedure service: is bedside procedure teaching reassuring to patients? J Hosp Med. 2011;6:219-224. [go to PubMed]