Educational agenda for diagnostic error reduction
=================================================

* Robert L Trowbridge
* Gurpreet Dhaliwal
* Karen S Cosby

## Abstract

Diagnostic errors are a major patient safety concern. Although the majority of diagnostic errors are partially attributable to cognitive mistakes, the most effective means of improving clinician cognition in order to achieve gains in diagnostic reliability are unclear. We propose a tripartite educational agenda for improving diagnostic performance among students, residents and practising physicians. This agenda includes strengthening the metacognitive abilities of clinicians, fostering intuitive reasoning and increasing awareness of the role of systems in the diagnostic process. The evidence supporting initiatives in each of these realms is reviewed and a course of future implementation and study is proposed. The barriers to designing and implementing this agenda are substantial and include limited evidence supporting these initiatives and the challenges of changing the practice patterns of practising physicians. Implementation will need to be accompanied by rigorous evaluation.

*   Diagnostic errors
*   Medical education
*   Cognitive biases

## Introduction

Diagnostic error is well established as a major cause of patient harm. With the rate of missed, incorrect or delayed diagnoses estimated to be as high as 10% to 15%,1 ,2 diagnostic error may result in over 40 000 deaths annually in the USA alone.3 In addition, because diagnostic errors are not easily identifiable as 'events', they are difficult to track and are likely under-reported. Although relatively neglected within the field of patient safety, there are increasing calls for improvements in diagnostic reliability.3–5 Suggested interventions for improving diagnostic safety have primarily centred on the development of systems-based approaches such as clinical decision support systems and electronic medical records.6–8 Although systems-based interventions may remedy or mitigate some cognitive errors,3 these efforts will need to be coupled with improvements in the cognitive performance of clinicians to achieve significant improvements in diagnostic reliability.

Over the past decade, there has been increased interest in teaching clinical reasoning during medical training in order to equip novice clinicians with the basic skills necessary to become competent diagnosticians.5 ,9–11 The best approach for teaching clinical reasoning and the extent to which biomedical instruction should be supplanted or supplemented by clinical reasoning instruction (acknowledging the fundamental role content knowledge plays in reasoning) are unknown. In this article, we propose an educational agenda for increasing diagnostic reliability among trainees and practising physicians. This agenda is delineated into the three general themes: improving metacognitive abilities, developing non-analytical (intuitive) capabilities and increasing systems awareness. We aim to describe the rationale for the development of educational programmes in each of these realms, review the currently available evidence and suggest potential areas of innovation and study.

### Metacognition

The dual process model of clinical reasoning emphasises the interplay between intuitive reasoning (a rapid and unconscious application of previous knowledge and experiences, triggered by environmental cues, into decisions and actions) and analytical reasoning (a conscious approach to solving a problem with extended or deliberate analysis) in making diagnoses and solving problems in clinical settings.12 Metacognition is the analytic practice of explicitly examining one's thought processes and affective state and considering their effects on diagnostic reasoning.13–15 Teaching diagnosticians 'how doctors think' and enhancing their understanding of cognitive pitfalls and biases may allow clinicians to devise cognitive forcing strategies (structured methods of recognising and counteracting heuristics and biases) to limit the influence of these biases on the diagnostic process.14 ,16 ,17

To date, there is little evidence that explicitly teaching the mechanics of the clinical reasoning process can improve diagnostic performance.18 ,19 Sherbino *et al*,20 for example, examined the utility of cognitive forcing strategies as a means of overcoming search satisficing and availability bias. No long-term effect was seen, although the intervention was limited in scope. Mamede showed that the tendency towards the availability bias could be partially attenuated in residents through the use of conscious reflection structured as a reanalysis of the case findings.21 Finally, Ogdie showed that residents who are taught about cognitive biases can successfully reflect on the role of bias in their own experiences, but the actual effect on diagnostic performance was not assessed.22

An inherent weakness of these debiasing approaches is that they target the same human tendencies to think and problem solve in a certain way (also termed *cognitive predispositions to respond*) that often results in the *correct* diagnosis.19 Furthermore, novices may benefit from different cognitive developmental processes than experts.23 Given the paucity of evidence but underlying logic of this approach, we propose that clinicians should be instructed in the basic process of clinical reasoning (such as the interplay between analytical reasoning and intuitive reasoning and the strengths and weaknesses of heuristics) and that further interventions that promote metacognitive techniques such as cognitive forcing strategies be designed and implemented and their impact carefully studied. The net effect of these forms of interventions will likely be dependent on a number of variables specific to both learner and content. It is probable that the best approach to designing interventions is the one that reflects the underlying complexity of the clinical reasoning process and acknowledges there is no one ‘best practice’.18

### Fostering intuition

Although the medical decision making, evidence-based medicine and clinical reasoning literatures generally favour analytical reasoning over intuitive reasoning, modern neuroscience recognises the dynamic interplay and cross-checking between the intuitive (system 1) and analytical (system 2) cognitive systems in the dual process model and, in some cases, the superior performance of intuition.24 An emerging body of medical education studies in varied domains (eg, electrocardiograms and mental health disorders)25–27 encourages medical educators to promote the development of intuition among trainees.

Specific methods to develop and promote intuitive reasoning can be derived from a synthesis of the learning, judgement/decision-making and expertise literatures. These approaches are similar to the mental training on the job and outside of work that underpins naturalistic decision making.28 As detailed below, several approaches can be used to strengthen learners' intuitive skills on their own, with the help of a dedicated coach, or through a curriculum or organisational culture. Teachers must also help trainees develop their metacognitive skills to recognise clinical situations when intuition can be relied upon and when it must be abandoned for or buttressed by analytical reasoning.

#### Progressive problem solving

Progressive problem solving describes the mindset of professionals who avoid the routinisation of work (eg, ‘yet another case of pneumonia’) by creating challenges that increase the cognitive burden of the encounter *even when the circumstances do not require it*. Top performing professionals pursue this approach (eg, precontemplating an algorithm if the pneumonia fails to respond to doxycycline) in an effort to train their minds (ie, sharpen intuition) for future encounters.29 Sargeant found that highly rated (by 360° evaluation) physicians maintained this dual purpose mindset of taking care of patients, focusing on patient outcomes while habitually generating learning.30 Mylopoulos similarly confirmed that high-performing diagnosticians build their knowledge through purposeful, continuous engagement and learning in clinical practice.31

#### Feedback

In the current training environment, only a minority of decisions are coupled with patient-outcome feedback (eg, “did the patient I assessed as low risk for coronary artery disease ultimately have coronary artery disease?”). This amounts to the forfeiture of multiple opportunities for learning and judgement optimisation. Feedback is a cornerstone of the deliberate practice model of skill development, where practice is focused upon a high-value task and regular feedback is used to close the gap between current and goal performance.32 Although the documentation of feedback leading to improved diagnostic performance is limited to a small number of studies,33 ,34 clinicians should consider how their learning could mirror that of other high accuracy professions which are characterised by copious direct feedback.35

#### Simulation

The intuitive system is developed and refined through experience, but there is a limit to the number of patients and hours that a clinician can be exposed to in the workplace setting during training or over a career. Simulation exercises, however, can broaden a learner's experience base and serve as a reference in future patient encounters. Free cognitive simulators abound in the form of on-line and journal-based case reports. Virtual patient experiences have been shown to enhance reasoning skills,36 although there is no data to demonstrate a reduction in diagnostic errors through this approach.37

#### Summary

In dynamic fields analogous to medicine where complex knowledge and judgement are required (eg, firefighting and military), experts are superior problem solvers who possess a large and rich repertoire of mental models (ie, illness scripts in medicine) of the problems they face at work.38 Trainees and their teachers, supported by curricula and institutional culture, can strengthen these illness scripts by increasing the quantity *and quality* of clinical experience through the methods described above, which amount to mental training derived from current encounters in preparation for future problem solving. These approaches to improve intuitive reasoning have great potential in medical education, but require careful development and research to determine if they can fulfil their promise.

### Systems-level awareness

A systems view of diagnosis regards clinical reasoning as a cognitive process that is dependent upon a number of external factors that impact the ability to gather information, acquire tests and imaging, interpret results and communicate with others. From an educational perspective, a systems approach to teaching diagnostic skills must equip learners with the knowledge, skills and attitudes to interact with people and processes to make a reliable diagnosis. It should also include instructions on how to recognise and compensate for imperfect systems that hinder optimal decision making.

#### Knowledge of systems

The essential message of teaching about systems is simple, yet rarely acknowledged in medical education: the ability to make a diagnosis relies on a host of factors largely outside one's immediate control. The ability to recognise system flaws, compensate for them and work to continually improve the process is vital. A review of Reason's Swiss cheese model provides basic introductory content.39 There is additionally a growing literature base of potential system interventions to support the cognitive work of diagnosis that will likely offer new ideas and strategies to teach for future improvement.7

#### Systems-based skills

Trainees and clinicians can learn about diagnostic failure by investigating and analysing real-world cases. A variety of different frameworks and tools are available to facilitate analysis, some of which are specific to settings or specialties.40–45 The ability to critique cases, identify contributing factors, map the diagnostic process and recognise problems and potential sources of error can provide opportunity for learning and strategies for improvement. Case conferences and bedside teaching can be modified to include discussion about systems factors that impair the flow of accurate information and predispose patients to misdiagnosis-related harm. Morbidity and mortality (M&M) conferences, in particular, may be an ideal format for discussing diagnosis failure 46–54 and the addition of a structured, systems-oriented tool can facilitate recognition of system flaws.53 At one institution, the addition of a structured ‘system audit’ to M&M investigations improved awareness of system issues and made residents more aware of opportunities for system improvement.53 Conferences which address systems errors, however, should be balanced by discussions of cognitive error as an exclusive focus on systems may decrease the perceived educational value of the conference and has the unintended consequence of diminishing any one person's sense of ownership or accountability.47 ,55

#### Systems-based attitudes

The most significant part of educating about safety in systems is fostering an open attitude to discussing and addressing diagnostic errors, and promoting an environment that encourages engagement in system improvement.56 ,57 Attitudes can be modelled by mentors, fostered by institutional culture and enhanced by opportunities for improvement projects. Success in changing a system is a powerful incentive, particularly for young trainees. Education should help the trainee recognise the need not only to achieve personal excellence, but also to think more broadly about extrinsic factors that limit his/her success. A systems perspective comes full circle: first viewing ‘the system’ as an inanimate network of imperfect processes, and then recognising that the system is a complex network of talented resourceful individuals who need frequent readjustment and well-designed processes to effectively optimise care.

## Implementation

Diagnostic errors are committed by medical students, residents and practising clinicians. Therefore, this educational agenda is proposed for the selective application to all levels of medical professionals. Medical schools could provide basic instruction in the mechanics of clinical reasoning, and this training could be augmented and reinforced by initiatives designed for both postgraduate trainees and practising physicians. Graduate medical education, for example, could routinise discussion of common cognitive errors and the role of systems, while practising physicians could be incentivised or required by certification organisations to demonstrate basic understanding of reasoning principles, diagnostic errors and continuous quality improvement of systems. A notable barrier to the implementation of these programmes is the lack of evidence that the proposed techniques and initiatives effect a clear improvement in diagnostic performance.37 This, however, is more reflective of a lack of investigation in this field than evidence suggesting these techniques are detrimental or ineffective. Rigorous and well-designed investigations, including randomised controlled studies, are required to determine the effect of individual educational techniques on diagnostic performance. The effect of feedback on the performance of emergency department physicians in the evaluation of patients with chest pain, for instance, could be assessed by randomising one cohort to feedback on patient outcomes. Similarly, the effect of instructing paediatricians in meta-analytical reasoning surrounding the diagnosis of acute otitis media could be assessed by tracking prescribing activity and patient outcomes. Finally, specific cognitive forcing strategies such as metacognitive checklists could be deployed in the primary care setting with careful measurement of the effect on diagnostic outcomes for common conditions like low back pain and respiratory infections.

Competing curricular demands are another major impediment. Medical school and residency curricula are already overloaded and struggling with the inclusion of new initiatives in palliative care, team-based care and quality improvement. The insertion of another domain may be met with substantial resistance. Similarly, many practising physicians are challenged by significant productivity pressures and state-mandated continuing education programmes, limiting the time that may be devoted to these methods. Finally, teaching many of these approaches requires faculty with a basic understanding of clinical reasoning, cognitive psychology and informatics. Faculty development is necessary to develop a cadre of educators with the necessary skill and experience to do so on a widespread basis.

## Conclusions

Diagnostic unreliability remains a significant cause of error-related patient harm. Although initiatives designed to improve systems will be essential in elevating diagnostic performance, enhancing the cognitive functioning of clinicians remains an equally important and perhaps more elusive goal. The current educational model in which clinical reasoning expertise is developed passively has produced disappointing results. We propose that explicit instruction in the clinical reasoning process should begin at the earliest stages of medical school as a foundational ‘basic science’ and be strongly emphasised throughout the undergraduate curriculum. Building upon this basic foundation, this tripartite agenda for improving diagnostic performance can be adopted by clinicians at all levels of experience.

This agenda is rooted in the dual process model of clinical reasoning and serves to improve the functioning of both intuitive and analytical processes. In developing such complementary skills, clinicians will be far better equipped to deal with the complexity and difficulties of the diagnostic process. The agenda also emphasises the importance of the clinician's ability to interact with and influence healthcare systems in improving diagnostic performance. Knowledge of such systems and the ability to harness their potential in improving diagnostic performance should be core skills for all clinicians.

## Footnotes

*   Contributors All of the listed authors were continuously involved in writing the paper, including: conception and design, acquisition of data or analysis and interpretation of data; drafting the article and revising it critically for important intellectual content; and final approval of the version published.

*   Competing interests None.

*   Patient consent Obtained.

*   Provenance and peer review Commissioned; externally peer reviewed.

this is an open access article distributed in accordance with the creative commons attribution non commercial (cc by-nc 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. see: [http://creativecommons.org/licenses/by-nc/3.0/](http://creativecommons.org/licenses/by-nc/3.0/)

## References

1.  Berner ES, Graber ML. Overconfidence as a cause of diagnostic error in medicine. Am J Med 2008;121(5 Suppl Diagnostic Error in Medicine):S2–23.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/j.amjmed.2008.01.001&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=18440350&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 

2.  1.  Higgs J
    
    Elstein A. Clinical reasoning in medicine. In: Higgs J, ed. Clinical reasoning in the health professions. Oxford: Butterworth-Heineman, 1995:49–59.
    
    

3.  Newman-Toker DE, Pronovost PJ. Diagnostic errors—the next frontier for patient safety. JAMA 2009;301:1060–2.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.2009.249&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=19278949&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000264015800031&link_type=ISI) 

4.  Wachter RM. Why diagnostic errors don't get any respect—and what can be done about them. Health Aff (Millwood) 2010;29:1605–10.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToiaGVhbHRoYWZmIjtzOjU6InJlc2lkIjtzOjk6IjI5LzkvMTYwNSI7czo0OiJhdG9tIjtzOjI1OiIvcWhjLzIyL1N1cHBsXzIvaWkyOC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

5.  Graber ML, Wachter RM, Cassel CK. Bringing diagnosis into the quality and safety equations. JAMA 2012;308:1211–12.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/2012.jama.11913&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=23011708&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000309103600020&link_type=ISI) 

6.  Singh H, Graber M. Reducing diagnostic error through medical home-based primary care reform. JAMA 2010;304:463–4.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.2010.1035&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=20664048&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 

7.  Singh H, Graber ML, Kissam SM, et al. System-related interventions to reduce diagnostic errors: a narrative review. BMJ Qual Saf 2012;21:160–70.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjIxLzIvMTYwIjtzOjQ6ImF0b20iO3M6MjU6Ii9xaGMvMjIvU3VwcGxfMi9paTI4LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

8.  Wachter RM, Holmboe ES. Diagnostic errors and patient safety. JAMA 2009;302:258; author reply 9–60.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.2009.1006&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=19602685&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 

9.  Bowen JL. Educational strategies to promote clinical diagnostic reasoning. N Engl J Med 2006;355:2217–25.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1056/NEJMra054782&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=17124019&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 

10. Eva KW. What every teacher needs to know about clinical reasoning. Med Educ 2005;39:98–106.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1111/j.1365-2929.2004.01972.x&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=15612906&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000226255200016&link_type=ISI) 

11. Rencic J. Twelve tips for teaching expertise in clinical reasoning. Med Teach 2011;33:887–92.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.3109/0142159X.2011.558142&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=21711217&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 

12. Croskerry P. A universal model of diagnostic reasoning. Acad Med 2009;84:1022–8.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/ACM.0b013e3181ace703&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=19638766&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000270711400012&link_type=ISI) 

13. Croskerry P. The cognitive imperative: thinking about how we think. Acad Emerg Med 2000;7:1223–31.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1111/j.1553-2712.2000.tb00467.x&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=11073470&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000167382500011&link_type=ISI) 

14. Croskerry P. The importance of cognitive errors in diagnosis and strategies to minimize them. Acad Med 2003;78:775–80.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/00001888-200308000-00003&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12915363&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000220070500003&link_type=ISI) 

15. Graber M. Metacognitive training to reduce diagnostic errors: ready for prime time? Acad Med 2003;78:781.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/00001888-200308000-00004&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12915364&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000220070500004&link_type=ISI) 

16. Croskerry P. Cognitive forcing strategies in clinical decisionmaking. Ann Emerg Med2003;41:110–20.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1067/mem.2003.22&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12514691&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000180369200016&link_type=ISI) 

17. Teal CR, Gill AC, Green AR, et al. Helping medical learners recognise and manage unconscious bias toward certain patient groups. Med Educ 2012;46:80–8.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1111/j.1365-2923.2011.04101.x&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=22150199&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 

18. Norman GR, Eva KW. Doggie diagnosis, diagnostic success and diagnostic reasoning strategies: an alternative view. Med Educ 2003;37:676–7.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1046/j.1365-2923.2003.01528.x&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12895245&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 

19. Norman GR, Eva KW. Diagnostic error and clinical reasoning. Med Educ 2010;44:94–100.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1111/j.1365-2923.2009.03507.x&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=20078760&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000272866400019&link_type=ISI) 

20. Sherbino J, Yip S, Dore KL, et al. The effectiveness of cognitive forcing strategies to decrease diagnostic error: an exploratory study. Teach Learn Med 2011;23:78–84.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1080/10401334.2011.536897&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=21240788&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 

21. Mamede S, van Gog T, van den Berge K, et al. Effect of availability bias and reflective reasoning on diagnostic accuracy among internal medicine residents. JAMA 2010;304:1198–203.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.2010.1276&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=20841533&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 

22. Ogdie AR, Reilly JB, Pang WG, et al. Seen Through Their Eyes: Residents’ Reflections on the Cognitive and Contextual Components of Diagnostic Errors in Medicine. Acad Med 2012;87:1361–7.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/ACM.0b013e31826742c9&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=22914511&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000309544000018&link_type=ISI) 

23. Mamede S, Schmidt HG, Rikers RM, et al. Conscious thought beats deliberation without attention in diagnostic decision-making: at least when you are an expert. Psychol Res 2010;74:586–92.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1007/s00426-010-0281-8&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=20354726&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 

24. Dhaliwal G. Going with your gut. J Gen Intern Med 2011;26:107–9.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1007/s11606-010-1578-4&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=21104451&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 

25. Ark TK, Brooks LR, Eva KW. Giving learners the best of both worlds: do clinical teachers need to guard against teaching pattern recognition to novices? Acad Med 2006;81: 405–9.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/00001888-200604000-00017&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=16565197&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000236618200015&link_type=ISI) 

26. de Vries M, Witteman CL, Holland RW, et al. The unconscious thought effect in clinical decision making: an example in diagnosis. Med Decis Making 2010;30:578–81.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToic3BtZG0iO3M6NToicmVzaWQiO3M6ODoiMzAvNS81NzgiO3M6NDoiYXRvbSI7czoyNToiL3FoYy8yMi9TdXBwbF8yL2lpMjguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

27. Regehr G, Cline J, Norman GR, et al. Effect of processing strategy on diagnostic skill in dermatology. Acad Med 1994;69(10 Suppl Diagnostic Error in Medicine):S34–6.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/00001888-199401000-00005&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=7916820&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=A1994PM98500013&link_type=ISI) 

28. Klein G. Sources of power: how people make decisions. Cambridge, MA: MIT Press, 1998.
    
    

29. Bereiter C, Scardamalia M. Surpassing ourselves. Peru, IL: Open Court, 1993.
    
    

30. Sargeant J, Mann K, Sinclair D, et al. Learning in practice: experiences and perceptions of high-scoring physicians. Acad Med 2006;81:655–60.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/01.ACM.0000232422.81299.b7&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=16799293&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000238899100011&link_type=ISI) 

31. Mylopoulos M, Lohfeld L, Norman GR, et al. Renowned Physicians’ Perceptions of Expert Diagnostic Practice. Acad Med 2012;87:1413–17.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/ACM.0b013e31826735fc&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=22914510&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000309544000027&link_type=ISI) 

32. Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med 2004;79(10 Suppl Diagnostic Error in Medicine):S70–81.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/00001888-200407001-00018&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=15240220&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000224173900023&link_type=ISI) 

33. Tudor GR, Finlay DB. Error review: can this improve reporting performance? Clin Radiol 2001;56:751–4.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1053/crad.2001.0760&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=11585397&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 

34. Wood D, Tracey T. A brief feedback intervention for diagnostic overshadowing. Train Educ Prof Psychol 2009;3:218–325.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1037/a0016577&link_type=DOI) 

35. Shanteau J. Competence in experts: The role of task characteristics. Organ Behav Hum Decis Process 1992;53:252–62.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/0749-5978(92)90064-E&link_type=DOI) 

36. Cook DA, Triola MM. Virtual patients: a critical literature review and proposed next steps. Med Educ 2009;43:303–11.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1111/j.1365-2923.2008.03286.x&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=19335571&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 

37. Graber ML, Kissam S, Payne VL, et al. Cognitive interventions to reduce diagnostic error: a narrative review. BMJ Qual Saf 2012;21:535–57.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjIxLzcvNTM1IjtzOjQ6ImF0b20iO3M6MjU6Ii9xaGMvMjIvU3VwcGxfMi9paTI4LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

38. Klein G. The power of intuition. New York, NY: Doubleday, 2003.
    
    

39. Reason J. Human error: models and management. BMJ 2000;320:768–70.
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjAvNzIzNy83NjgiO3M6NDoiYXRvbSI7czoyNToiL3FoYy8yMi9TdXBwbF8yL2lpMjguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

40. Cosby KS. A framework for classifying factors that contribute to error in the emergency department. Ann Emerg Med 2003;42:815–23.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/S0196-0644(03)00635-8&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=14634609&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 

41. 1.  Marx J, 
    2.  Hockberg P, 
    3.  Walls R
    
    Croskerry P, Chisholm C. Process improvement and patient safety in the emergency department. In: Marx J, Hockberg P, Walls R, eds. Rosens emergency medicine; concepts in clinical practice. 6th edn. St. Louis: Mosby, 2006:3119–27.
    
    

42. 1.  Croskerry P, 
    2.  Cosby K, 
    3.  Wears R
    
    Schenkel S. Morbidity and mortality conference and patient safety in emergency medicine. In: Croskerry P, Cosby K, Wears R, eds. Patient safety in emergency medicine. Lippincott, Williams and Wilkins, 2009;295–301.
    
    

43. Schiff G, Kim S, Abrams R, et al. Diagnosis diagnostic errors: lessons from a multi-institutional collaborative project. New advances in patient safety: from research to implementation. AHRQ, 2005:255–78.
    
    

44. Vincent C, Taylor-Adams S, Chapman EJ, et al. How to investigate and analyse clinical incidents: clinical risk unit and association of litigation and risk management protocol. BMJ 2000;320:777–81.
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjAvNzIzNy83NzciO3M6NDoiYXRvbSI7czoyNToiL3FoYy8yMi9TdXBwbF8yL2lpMjguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

45. Zhang J, Patel VL, Johnson TR, et al. A cognitive taxonomy of medical errors. J Biomed Inform 2004;37:193–204.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/j.jbi.2004.04.004&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=15196483&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000222373900005&link_type=ISI) 

46. Aboumatar HJ, Blackledge CG Jr., Dickson C, et al. A descriptive study of morbidity and mortality conferences and their conformity to medical incident analysis models: results of the morbidity and mortality conference improvement study, phase 1. Am J Med Qual 2007;22:232–8.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToic3Bham0iO3M6NToicmVzaWQiO3M6ODoiMjIvNC8yMzIiO3M6NDoiYXRvbSI7czoyNToiL3FoYy8yMi9TdXBwbF8yL2lpMjguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

47. Bechtold ML, Scott S, Nelson K, et al. Educational quality improvement report: outcomes from a revised morbidity and mortality format that emphasised patient safety. Qual Saf Health Care 2007;16:422–7.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjE2LzYvNDIyIjtzOjQ6ImF0b20iO3M6MjU6Ii9xaGMvMjIvU3VwcGxfMi9paTI4LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

48. Berenholtz SM, Hartsell TL, Pronovost PJ. Learning from defects to enhance morbidity and mortality conferences. Am J Med Qual 2009;24:192–5.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToic3Bham0iO3M6NToicmVzaWQiO3M6ODoiMjQvMy8xOTIiO3M6NDoiYXRvbSI7czoyNToiL3FoYy8yMi9TdXBwbF8yL2lpMjguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

49. Higginson J, Walters R, Fulop N. Mortality and morbidity meetings: an untapped resource for improving the governance of patient safety? BMJ Qual Saf 2012;21:576–85.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjIxLzcvNTc2IjtzOjQ6ImF0b20iO3M6MjU6Ii9xaGMvMjIvU3VwcGxfMi9paTI4LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

50. Ksouri H, Balanant PY, Tadie JM, et al. Impact of morbidity and mortality conferences on analysis of mortality and critical events in intensive care practice. Am J Crit Care 2010;19:135–45; quiz 46.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpjYyI7czo1OiJyZXNpZCI7czo4OiIxOS8yLzEzNSI7czo0OiJhdG9tIjtzOjI1OiIvcWhjLzIyL1N1cHBsXzIvaWkyOC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

51. Liu V. Error in Medicine: the role of morbidity and mortality conference. Virtual Mentor 2005; 7. http://virtualmentor.ama-assn.org/2005/04/msoc1-0504.html (accessed 30 May 2013).
    
    

52. Orlander JD, Barber TW, Fincke BG. The morbidity and mortality conference: the delicate nature of learning from error. Acad Med 2002;77:1001–6.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/00001888-200210000-00011&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12377674&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000179365700010&link_type=ISI) 

53. Schwarz D, Schwarz R, Gauchan B, et al. Implementing a systems-oriented morbidity and mortality conference in remote rural Nepal for quality improvement. BMJ Qual Saf 2011;20:1082–8.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjEwOiIyMC8xMi8xMDgyIjtzOjQ6ImF0b20iO3M6MjU6Ii9xaGMvMjIvU3VwcGxfMi9paTI4LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

54. Szostek JH, Wieland ML, Loertscher LL, et al. A systems approach to morbidity and mortality conference. Am J Med 2010;123:663–8.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/j.amjmed.2010.03.010&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=20609691&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 

55. Casarett D, Helms C. Systems errors versus physicians’ errors: finding the balance in medical education. Acad Med 1999;74:19–22.
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=9934289&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000078070700011&link_type=ISI) 

56. Blumenthal D. Making medical errors into ‘medical treasures’. JAMA 1994;272:1867–8.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.1994.03520230077043&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=7990223&link_type=MED&atom=%2Fqhc%2F22%2FSuppl_2%2Fii28.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=A1994PW98000038&link_type=ISI) 

57. McIntyre N, Popper K. The critical attitude in medicine: the need for a new ethics. Br Med J (Clin Res Ed) 1983;287:1919–23.
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6MzoiUERGIjtzOjExOiJqb3VybmFsQ29kZSI7czozOiJibWoiO3M6NToicmVzaWQiO3M6MTM6IjI4Ny82NDA5LzE5MTkiO3M6NDoiYXRvbSI7czoyNToiL3FoYy8yMi9TdXBwbF8yL2lpMjguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9)