5th August 2016, Volume 129 Number 1439

Jennifer Weller, David Cumin, Ian Civil, Jane Torrie, Alexander Garden, Andrew MacCormick, Nishanthi Gurusinghe, Matthew Boyd, Christopher Frampton, Ludvig Selander, Martina Cokorilo, Magnus Tranvik, Lisa Carlsson, Tracey Lee, Wai Leap Ng, Michael Crossan, Alan Merry

Treatment injury is a frequent cause of patient morbidity and mortality. A recent study estimated the number of adverse events in healthcare at approximately 10% of hospitalisations globally, resulting in 23 million years lost through disability or death every year.1 Failures in teamwork and communication contribute to many of these adverse events,2 notably in the operating room (OR).3,4,5

The OR is a high acuity, complex environment, and therefore prone to errors, with particular need for good communication and teamwork. However, staffing patterns in the OR may render it particularly prone to errors in communication. In a large hospital, the composition of the OR team varies from day to day, or even over the course of the day, with limited time for staff to gain an understanding of each other’s capabilities and establish the sense of mutual respect and trust required for open communication and effective teamwork. The OR team is typically comprised of three disciplinary groups (surgical, anaesthesia, nursing) with different backgrounds and training. Established hierarchies and professional boundaries may inhibit speaking up and the sharing of information.6

There is some evidence that training OR teams can improve teamwork,7 and convincing evidence that using the Surgical Safety Checklist (the Checklist), a tool designed to improve information sharing among OR team members, can reduce the morbidity and mortality associated with surgery.8,9 However, the way the Checklist is used varies, and in consequence, so does its effect on patient outcomes.10 We suggest that a receptive and supportive culture is required to fully realise the potential benefits of communication tools such as the Checklist and that an understanding of the benefits to patient outcomes of effective teamwork and communication would help to promote such a culture.

To this end we devised the Multidisciplinary Operating Room Simulation (MORSim) intervention, comprising a day of simulated, challenging surgical cases, debriefing and discussion for OR teams to increase their understanding of the importance of communication and teamwork. We based MORSim on a theoretical framework of teamwork proposed by Salas.11 This framework incorporates five key dimensions of effective teams and three underpinning mechanisms. The key dimensions are: leadership, team orientation, mutual performance monitoring, back up behaviour and adaptability. The underpinning mechanisms are: shared mental models, mutual trust and closed loop communication.

This study is part of a programme of research with the overall aim of implementing simulation-based training of OR personnel in teamwork and communication in all hospitals in New Zealand. Our end-of-course evaluation of MORSim described positive participant reactions to the course, self-reported evidence of learning and improved scores for teamwork and communication.12 In this study we looked for transfer to clinical practice, measured through change in observable teamwork and communication behaviours across all general surgical ORs in the two participating hospitals.

We used the Behavioural Marker Risk Index (BMRI)13 to measure the impact of our intervention, with an additional question specific to MORSim. The BMRI measures six domains of behaviour: briefing, information sharing, inquiry, contingency management, assertion and vigilance. These are measured at three phases of surgery, defined by the original authors as the induction phase (from when the patient enters the OR until the incision), intraoperative phase (from incision until wound closure) and the handoff phase (from wound closure until transition to the next level of care is complete). The rationale for using this particular measure was the previously documented link between BMRI scores and patient outcomes. In Mazzocco’s original BMRI study,13 poor BMRI scores for teamwork were significantly associated with patient death or complications after surgery. In this study, our focus was the improvement of inter-disciplinary information sharing, so we added a seventh domain to assess information sharing between the three teams (surgeons, anaesthetists and anaesthetic technicians, and nursing staff: Table 1).

Table 1: Domains used in scoring the modified BMRI.13




Situation/relevant background is shared; patient, procedure, site/side are identified; plans are stated; questions are asked; ongoing monitoring and communication is encouraged.

Information sharing

Information is shared; intentions are stated; mutual respect is evident; social conversations are appropriate.


Input and other relevant information is asked for.



Relevant risks are identified; backup plans are made and executed.


The members of the team speak up with their observations and recommendations during critical times.


Tasks are prioritized; attention is focused; patient/equipment is checked, monitoring is maintained; tunnel vision is avoided; red flags are identified.

Inter-disciplinary information sharing

Information is shared between the surgical, anaesthesia and nursing teams.

We aimed to test the following hypothesis; “That, using the extended BMRI measurement tool, overall scores for teamwork and communication across all the general surgical operating room teams in the two study hospitals would improve from the period before MORSim to the period following MORSim.”


Ethics approval was obtained from Auckland Regional Ethics Committee (NTX/12/EXP/067) and the ethics committees of the two hospitals involved in the study. (Australia and New Zealand Clinical Trials Registry ID 12612001088831.)

The MORSim Intervention

The Intervention was a full-day multidisciplinary OR simulation course (MORSim) based at the Simulation Centre for Patient Safety (SCPS), University of Auckland. It consisted of three simulations with debriefs and presentations on communication strategies. The three simulations were each of 40 minutes duration and required OR teams to manage acute surgical cases. We created a realistic OR environment similar to that in our participants’ hospitals. We used real drug ampoules, fluids, sterile syringes, needles and fluid giving sets as found in the clinical environment, artificial blood presented in packaging and identifiers as provided by the blood bank, equipment such as rapid infusion devices, fluid warmers, anaesthetic machine, suction, diathermy and sterile surgical instruments and drapes. Patient clinical notes and investigations were available electronically. We designed the simulations so that the participants worked together on the case without prompts or input from faculty, as they would do in their normal working environment. A faculty nurse in the simulation room assisted only when requested by the participants eg by helping them to locate equipment, take blood, confirming (or not) the presence of a rash. We used Laerdal 3G SimMan (Stavanger, Norway) and METI® HPS™ (Sarasota, FL, USA) manikins. We commissioned a special effects company (Main Reactor, Auckland New Zealand) to manufacture life-like surgical models that integrated with both the manikins to allow surgeons to operate on the models using surgical instruments and, when appropriate, control blood loss.

Two scenarios depicted patients with acute abdominal pathology: appendicitis complicated by sepsis and subsequent allergic reaction, and a stab wound with a lacerated inferior vena cava (IVC) complicated by cardiovascular collapse. The third scenario involved a trauma patient with leg amputation following an explosion, complicated by lung barotrauma.

Simulations were preceded by familiarisation with the simulation environment. Participants were then given standardised written clinical briefings. These briefings were substantially identical, but each team member received a unique, additional, clinically relevant and important item of information about the patient. We chose items that could plausibly be known by the particular team member, but not necessarily by the others. Examples included: the patient was carrying an asthma inhaler; the patient was recently on long haul flight and had calf pain 24 hours ago; and metronidazole had been charted in the Emergency Department but not yet administered.

All scenarios were followed by a 40-minute debrief using a structured framework to guide discussion about teamwork, information sharing and communication strategies, with particular reference to whether the unique items of information given to each team member had or had not been shared, and the reasons for this.

We ran 20 study days with 120 participants, who were drawn equally from the two study hospitals. On any day, the six invited participants were from the same hospital (and thus likely to work together clinically) and comprised a consultant surgeon, a surgical resident, a consultant anaesthetist or anaesthetic fellow, an anaesthetic technician and two OR nurses.

Data collection

We used the extended BMRI to score teamwork in the ORs of the two study hospitals before and after the intervention. We coded within-discipline information sharing to the BMRI original domain “information sharing”, and between-disciplinary information sharing to a new domain “Inter-disciplinary information sharing”. We followed the scoring methodology used by Mazzocco et al.13 Each of the seven domains was scored during Mazzocco et al.’s three phases of surgery described previously: induction, intraoperative, and handoff. Each domain was scored on a scale from 0–4 according to how frequently relevant behaviours were observed in each of the three phases. To enable comparison with Mazzocco et al.’s results, scores for each domain were then converted to binary form (3 or 4=1; 1 or 2=0) where 0 indicates that all behaviours were observed frequently and 1 indicates that the behaviours were never or only infrequently observed. An average BMRI score from 0 to 1 was then calculated for the three phases, as well as an overall BMRI for the case. To assess any potential confounding influences we also recorded case duration, the patient American Society of Anesthesiologists (ASA) score,14 the duration and type of operation and the number of OR staff present.

Sample size for OR observations

We aimed to collect observations on 200 cases before and 200 cases after the intervention guided by the Mazzocco study,13 where a relationship was shown in BMRI scores and patient outcomes comparing two groups each containing 150 cases.

Observers and training

The four observers involved in the study were medical students in their third year of study. Two observers (LC, MT) carried out all the observations before the MORSim course and two different observers (LS, MC) carried out all observations after the course. Two weeks of training were conducted with experienced researchers (DC, MB) for each pair of observers. The training included an introduction to teamwork behaviour studies and the extended BMRI tool as well as orientation to the OR environment, as recommended by Carthey et al.15 To ensure standardisation between the four observers, they participated in a series of training exercises before undertaking the clinical observations. In these, videos of eight surgical cases were rated by the observers using the extended BMRI rating form. The training cases had previously been rated by the two trainers using the same instrument to establish the standard. These cases included videos of simulated surgical cases and cases from the OR. Any discrepancies between raters and between the instructor score were discussed until consensus was reached. This process was repeated and inter-rater agreement was calculated at each step until acceptable agreement (RWG >0.8) was reached. (RWG=within-group inter-rater agreement.)

To ensure inter-rater agreement was maintained during the observation periods between raters, calibration sessions were held after each rater had completed 5–10 observations during the initial rating period, and then after every 50 observations, in accordance with recommended good research practice for observational work.16,17 These calibration sessions followed the same protocol as the training sessions except that if acceptable agreement (RWG>0.8) was not reached, the previously observed cases, back to the most recent acceptable calibration, were discarded from analysis.

Selection of procedures

Data were collected before the first MORSim course, between August and December 2012, and after the last MORSim course between September and November 2013. Cases to be scored were selected at the start of each workday during the collection period. In the pre-MORSim observations, if more than one general surgical case was scheduled at the same time, we selected cases of shorter duration to expedite data collection. In the post-MORSim observations, if more than one general surgical case was scheduled at the same time, we selected cases where at least some of the OR staff had attended MORSim. Therefore, staff observed included a mix of those who had and had not attended MORSim.


Extended BMRI scores pre- and post-MORSim were compared using ANOVA. Covariates included in the preliminary analysis were the operation time of day, the duration of the case, patient ASA score and total number of people in the OR. Independent variables with significant univariate effects were included in the final model. To test for the effect of the course on any specific item(s), a logistic regression model was used with the binary score of the domain as the dependent variable and the same independent variables as above. For statistical tests, significance was set at 0.05 and analysis was performed using R v3.0.1 (http://cran.r-project.org). We used a Bonferroni correction in a secondary analysis of pre-post effect on individual domains.

Table 2: The number (and percent) of observed cases with unadjusted extended BMRI scores of 3 or 4 (frequently or always) in each BMRI domain across the three phases of the case pre MORSim (pre) and post MORSim (post).


N (%) scored 3-4

BMRI Phase




BMRI Domain








132 (58.9)

175 (82.2)

6 (2.7)

60 (28.2)

135 (60.3)

164 (77.0)

Information sharing

212 (94.6)

179 (84.0)

153 (68.3)

169 (79.3)

178 (79.5)

114 (53.5)


144 (64.3)

123 (57.7)

64 (28.6)

86 (40.4)

74 (33.0)

63 (29.6)


210 (93.8)

191 (89.7)

191 (85.3)

188 (88.3)

169 (75.4)

179 (84.0)

Contingency management

6 (2.7)

15 (7.0)

1 (0.5)

2 (0.9)

1 (0.4)

7 (3.3)


2 (0.1)

2 (0.1)

4 (1.8)

5 (2.3)

0 (0)

1 (0.5)

information sharing

136 (60.7)

180 (84.5)

45 (20.1)

119 (55.9)

138 (61.6)

144 (67.6)


A total of 453 cases in the OR were observed. At an early rater calibration session, agreement of one rater was below the threshold (RWG >0.8) and 14 observations were discarded. Data from two observations were incomplete. This left 437 total observations for analysis (224 pre-MORSim, 213 post-MORSim), distributed evenly between the two hospitals.

Of the 213 post-MORSim cases observed, 145 involved at least one MORSim participant and 67 involved no MORSim participants. Of these 145 cases, 86 had at least one surgeon, 20 had at least one anaesthetist, 89 had at least one nurse and 28 cases had at least one anaesthetic technician who had attended MORSim.

The domains ‘contingency management’ and ‘assertion’ were observed on only 56 occasions in the 1,311 observation periods (437 cases, three phases) and were therefore excluded from further analysis as was the case in the Mazzocco et al. study.13

In respect of potential cofounders, BMRI was significantly related to the time of day the case started (p<0.001), the duration of the case (p<0.001), the number of staff in the OR (p<0.001) and patient ASA score (p<0.001). These factors were reasonably evenly distributed between groups (Table 3) and were included in the final model.

Table 3: Details of the cases observed for modified BMRI ratings before (Pre-MORSim) and after the MORSim intervention (Post-MORSim). Values are mean and standard deviation (Mean (SD)) for numbers of staff present and case duration, and as number and percentage of cases in each category (number (%)) for start time, and ASA status.




Number of staff present

Mean (SD)

Mean (SD)

Total staff in OR

8.3 (1.6)

10.0 (2.2)


1.7 (0.7)

2.0 (0.7)


1.3 (0.5)

1.7 (0.6)


3.2 (0.6)

3.5 (0.9)


1.1 (0.3)

1.3 (0.5)


0.9 (1.0)

1.4 (1.1)

Case duration

Mean (SD)

Mean (SD)


95.6 (61.9)

110.7 (67.6)

Start time for case

Number (%)

Number (%)


101 (45.1%)

99 (46.5%)


75 (33.5%)

79 (37.1%)


34 (15.2%)

33 (15.5%)


14 (6.3%)

2 (0.9%)

Patient ASA status

Number (%)

Number (%)


61 (27.2%)

57 (26.9%)


111 (49.6%)

80 (37.7%)


46 (20.5%)

70 (33.0%)


6 (2.7%)

5 (2.3%)

 *ASA = American Society of Anesthesiologists 

In the final model, overall the extended BMRI decreased (improved) pre- to post-MORSim by more than 20%, (0.41 v 0.32, p<0.001). There was statistically significant improvement in the extended BMRI for the induction and intraoperative phase pre- and post-MORSim in a repeated measure ANOVA (pre-post BMRI scores for induction, 0.255 v 0.005 p=0.005; intraoperative, 0.590 v 0.413 p<0.001 and handoff 0.380 v 0.346 p=0.22).

Individual domains in each of the three operative phases where BMRI scores of 3–4 were more frequently observed post-MORSim were: induction—‘briefing’, ‘interdisciplinary information sharing’, ‘information sharing’; intraoperative—‘briefing’, ‘interdisciplinary information sharing’; handoff—‘information sharing’, ‘vigilance’. However, we found that for ‘information sharing’ at the induction and handoff phases BMRI scores of 3–4 were less frequently observed post-MORSim with an odds ratio of less than 1 (Table 4).

Table 4: Odds ratios (95% confidence levels) for pre-post effect on individual domains rating highly (3 or 4) in the extended BMRI tool, after controlling for confounders. Significant changes are denoted with *at 0.05, **at 0.01, ***at 0.001 level after a Bonferroni correction.






4.0 (2.4–6.9)***

12.0 (5.2–32.9)***

2.1 (1.3–3.4)*

Info sharing

0.3 (0.1–0.6)**

1.5 (0.9–2.6)

0.3 (0.2–0.5)***


0.7 (0.4–1.0)

1.5 (1.1–2.9)

0.8 (0.5–1.2)


0.4 (0.2–1.0)

1.3 (0.7–2.5)

2.9 (1.6–5.7)**


5.1 (2.9–9.4)***

7.3 (4.4–12.4)***

1.2 (0.8–2.0) 


Following a multidisciplinary simulation-based team training intervention, extended BMRI scores for teamwork and communication in the clinical environment improved by more than 20% (0.41 v 0.32, p<0.001). Extrapolating from the work of Mazzocco et al.13 (as an indication of the potential order of magnitude of benefit) suggests that this could translate into a relative reduction of 14% in 30-day rates of complications and mortality in surgical patients.

We have previously reported improved BMRI scores in simulated cases over the course of the MORSim training day, positive participant evaluations and examples of learning and change in attitude from analysis of post-simulation debriefs.12 This current report extends our prior work by indicating that these changes appear to be maintained over time and are associated with changes in behaviour in actual clinical practice.

Our study thus adds to evidence supporting the relationship between simulation-based team training interventions for OR staff and improved clinical practice which is likely to manifest in improved patient outcomes. In a recent systematic review of what works in OR teamwork training,7 we found only one OR simulation-based intervention that provided evidence of change in clinical practice. This was in the form of participant self-report of changes in the OR.18 A subsequent report on an insurer-funded multidisciplinary simulation-based OR team training intervention also reported that participants intended to make changes in their clinical practice after the course.19 Our findings go further by demonstrating improved scores for the extended BMRI measure of teamwork and communication in the clinical setting following our intervention, which is important because of the previous link between BMRI scores and improved patient outcomes.

We also noted a number of previously unreported factors influencing the BMRI scores which are of relevance to other researchers using the BMRI. Scores deteriorated for cases scheduled later in the day. Perhaps this was due to less perceived need for communication as the team became better acquainted with each other, or perhaps they became more fatigued. The effect of case duration on the BMRI scores may be attributable to the greater opportunity provided by longer cases for raters to observe the relevant behaviours. Scores were also better when there were more staff and more complicated cases.


Ideally the raters would have been blinded to the intervention, however this was not possible in our naturalistic, observational pre-post study design. To limit bias, the raters were not involved in any other aspect of the MORSim intervention. Using different pairs of raters, pre and post intervention may have introduced error into the scores but using the same raters pre and post may have added other bias as the raters would have been looking for change. We attempted to mitigate rater error by regular calibration against set standards. These calibration sessions used the same video recordings, and ratings were consistent over time suggesting this did not have an important effect on the scores.

The observations for BMRI scoring of surgical teams occurred in actual ORs, which introduced elements beyond our control. We acknowledge there may have been other educational, quality improvement or organisational factors over the period of the study that contributed to our findings.20 Ideally a control group would be used, but this would require a much larger, multicentre study, and will be an area for future research. As discussed above, the fact that the study took place in an actual clinical environment could also be considered one of its strengths.

As this study involved observations of actual clinical practice, we could not observe the same teams or individuals before and after MORSim, or only MORSim participants. Thus, we were measuring the effect of the intervention on the general surgical theatres as a whole, rather than on individuals or specific teams. Even so, we were able to demonstrate an effect, and could postulate an even greater effect would be observed if all OR staff were able to participate in training. Participation in MORSim was voluntary, and so our preference for observations of OR cases that included at least some MORSim participants could have introduced an element of bias. While we selected mainly senior clinical staff, increasing team familiarity over the period of the study may have influenced BMRI scores.

Our estimation of the potential order of magnitude of benefit depends on extrapolation from the previous research by a different team in a different clinical context and should be taken as indicative rather than precise. It is possible that the additional question we added to the BMRI may have somehow influenced the previously established association between the BMRI and patient outcomes, but we would expect this to be a stronger effect if anything.

Future research in this area could usefully explore what percentage of staff should be exposed to the intervention to have an effect, the influence of participant discipline on subsequent impact on BMRI scores, and additional evidence of an association between BMRI scores and patient outcomes.


We have demonstrated improved scores for teamwork and communication in the general surgical ORs in two major hospitals following a multidisciplinary simulation-based intervention. Based on previous studies these improved scores could translate to a clinically important reduction in morbidity and mortality of surgical patients.

These results, along with previous research, provide support for incorporation of simulation-based team training into quality and safety initiatives for OR staff. Improving teamwork and communication in the OR could have a major impact on outcomes for surgical patients.


Failures in teamwork and communication are a contributing factor in the majority of unintended events causing patient harm. To address this, we provided team training for 20 complete general surgical teams from two large metropolitan hospitals in the Multidisciplinary Operating Room Simulation (MORSim) course. To evaluate if this team training was effective, we observed operating room teams in over 200 cases before and after the training, and scored teamwork performance. We found a significant improvement in teamwork scores following MORSim. Previous research suggests this would translate to a clinically important reduction in patient harm for surgical patients. On the basis of these findings, ACC has provided funding for the national roll out of MORSim in every public hospital in New Zealand.



We ran a Multidisciplinary Operating Room Simulation (MORSim) course for 20 complete general surgical teams from two large metropolitan hospitals. Our goal was to improve teamwork and communication in the operating room (OR). We hypothesised that scores for teamwork and communication in the OR would improve back in the workplace following MORSim. We used an extended Behavioural Marker Risk Index (BMRI) to measure teamwork and communication, because a relationship has previously been documented between BMRI scores and surgical patient outcomes.


Trained observers scored general surgical teams in the OR at the two study hospitals before and after MORSim, using the BMRI.


Analysis of BMRI scores for the 224 general surgical cases before and 213 cases after MORSim showed BMRI scores improved by more than 20% (0.41 v 0.32, p<0.001). Previous research suggests that this improved teamwork score would translate into a clinically important reduction in complications and mortality in surgical patients.


We demonstrated an improvement in scores for teamwork and communication in general surgical ORs following our intervention. These results support the use of simulation-based multidisciplinary team training for OR staff to promote better teamwork and communication, and potentially improve outcomes for general surgical patients.

Author Information

Jennifer M Weller, Associate Professor and Head of Centre for Medical and Health Sciences Education, University of Auckland and Anesthesiologist, Auckland City Hospital, New Zealand; David Cumin, Lecturer, Department of Anesthesiology, University of Auckland, Auckland; Ian D Civil, Professor of Surgery, University of Auckland, and Trauma and Vascular Surgeon Auckland City Hospital, New Zealand; Jane Torrie, Senior Lecturer, Department of Anesthesiology, University of Auckland and Anesthesiologist, Auckland City Hospital; Alexander Garden, Adjunct Professor, Victoria University of Wellington, Anesthesiologist, Capital and Coast Health, Wellington, New Zealand; Andrew D MacCormick, Department of Surgery, University of Auckland, General Surgeon, Counties Manukau Health, Auckland, New Zealand; Nishanthi Gurusinghe, Department of Surgery, Launceston Hospital, Tasmania, Australia; Matthew J Boyd, National Health Committee, New Zealand Ministry of Health, and Adapt Research Ltd, Wellington; Christopher Frampton, Professor, Psychological Medicine, University of Otago, Christchurch; Ludvig Selander, Medical Student, Linköping University, Linköping, Sweden; Martina Cokorilo, Medical Student, Linköping University, Linköping, Sweden; Magnus Tranvik, Medical Student, Linköping University, Linköping, Sweden; Lisa Carlsson, Medical Student, Linköping University, Linköping, Sweden; Tracey Lee, Auckland City Hospital, Auckland; Wai Leap Ng, Anesthesiologist, Counties Manukau Health, Auckland, New Zealand; Michael Crossan, Teaching Fellow, School of Nursing, University of Auckland, Auckland; Alan F Merry, Professor of Anesthesiology, University of Auckland and Anesthesiologist, Auckland City Hospital, Auckland.


We wish to acknowledge Wendy Guthrie, Sharee Johnson and Brenda Knowles for their contribution to recruiting and running MORSim. 


Jennifer Weller, Centre for Medical and Health Sciences Education, School of Medicine, University of Auckland. Private Bag 92019, Auckland, New Zealand 1142.

Correspondence Email


Competing Interests



  1. Jha AK, Larizgoitia I, Audera-Lopez C, et al. The global burden of unsafe medical care: analytic modelling of observational studies. BMJ Qual Saf. 2013;22:809-15.
  2. Sutcliffe KM, Lewton E, Rosenthal MM. Communication failures: an insidious contributor to medical mishaps. Acad Med. 2004;79:186-94.
  3. Lingard L, Espin S, Whyte S, et al. Communication failures in the operating room: an observational classification of recurrent types and effects. Qual Saf Health Care. 2004;13:330-4.
  4. Undre S, Sevdalis N, Healey A, et al. Teamwork in the operating theatre: cohesion or confusion? J Eval Clin Pract. 2006;12:182-9.
  5. Sevdalis N, Wong H, Arora S, et al. Quantitative analysis of intraoperative communication in open and laparoscopic surgery. Surg Endosc. 2012;26:2931-8.
  6. Weller J, Shedding new light on tribalism in health care. Med Educ. 2012;46:134-6.
  7. Weller J, Boyd M. Making a Difference Through Improving Teamwork in the Operating Room: A Systematic Review of the Evidence on What Works. Curr Anesthesiol Rep. 2014;4:77–83.
  8. Haynes AB, Weiser TG, Berry WR, et al. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med. 2009;360:491-9.
  9. Haugen AS, Søfteland E, Almeland SK, et al. Effect of the World Health Organization Checklist on Patient Outcomes: A Stepped Wedge Cluster Randomized Controlled Trial. Ann Surg. 2015;261:821-8.
  10. Mayer EK, Sevdalis N, Rout S, et al. Surgical Checklist Implementation Project: The Impact of Variable WHO Checklist Compliance on Risk-adjusted Clinical Outcomes After National Implementation: A Longitudinal Study. Ann Surg. 2016;263:58-63.
  11. Salas E, Sims DE, Burke CS. Is there a “Big Five” in teamwork? Small GR Res. 2005;36:555-99.
  12. Weller J, Cumin D, Torrie J, et al. Multidisciplinary Operating Room simulation-based team training to reduce treatment errors: a feasibility study in New Zealand hospitals. N Z Med J. 2015;128:40-51.
  13. Mazzocco K, Petitti DB, Fong KT, et al. Surgical team behaviors and patient outcomes. Am J Surg. 2009;197:678-85.
  14. ASA Physical Status Classification System. 2014. (Accessed 29 May 2015, at http://www.asahq.org/resources/clinical-information/asa-physical-status-classification-system.)
  15. Carthey J, The role of structured observational research in healthcare. Qual Saf Health Care. 2003;12:3-6.
  16. Tracy K, Adler LA, Rotrosen J, et al. Interrater reliability issues in multicenter trials, part I: theoretical concepts and operational procedures used in Department of Veterans Affairs Cooperative Study #394. Psychopharmacol Bull. 1997;33:53-67.
  17. Castorr AH, Thompson KO, Ryan JW, et al. The process of rather training for observartional instruments: implications for interrater reliability. Res Nurs Health. 1990;13:311-8.
  18. Stevens LM, Cooper JB, Raemer DB, et al. Educational program in crisis management for cardiac surgery teams including high realism simulation. J Thorac Cardiovasc Surg. 2012;144:17-24.
  19. Arriaga AF, Gawande AA, Raemer DB, et al. Pilot testing of a model for insurer-driven, large-scale multicenter simulation training for operating room teams. Ann Surg. 2014;259:403-10.
  20. Neily J, Mills PD, Young-Xu Y, et al. Association between implementation of a medical team training program and surgical mortality. JAMA. 2010;304:1693-700.