  |
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Journal of the New Zealand Medical Association,
22-October-2004, Vol 117 No 1204
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Simulation-based training to improve acute care skills in
medical undergraduates
Jennifer Weller, Brian Robinson, Peter Larsen, Catherine
Caldwell
One of the expected outcomes of medical undergraduate
training in New Zealand is the ability to manage a range of clinical
emergencies, including the initial management of a shocked patient.
There is considerable evidence that clinical skills may be
poorly taught and students may be ill-equipped to deal with the demands of a
house surgeon year.1 Training for medical
emergencies during undergraduate years traditionally includes reading, lectures,
tutorials, and clinical experience. Only clinical experience is aligned with the
desired learning outcome, but is limited by the unpredictable occurrence of
emergencies and the ethical problems of allowing medical undergraduates to learn
on acutely ill patients.
Patient simulation provides a safe learning environment
(where events can be scheduled, repeated, and observed) offering the potential
for greater efficiency and rigour over traditional methods. Simulation has in
fact been described as an ethical imperative, but has not yet been widely
incorporated into the medical undergraduate
curriculum2–4.
Recent studies have provided evidence on the reliability and
validity of simulation in assessment of both anaesthesia
skills5,6 and acute care
skills.7 In terms of effectiveness, there is
some evidence that anaesthetists’ performance improves following
training,8,9 and improved anaesthesia skills in
medical undergraduates have been demonstrated following
training.6 There is currently little evidence
on the effectiveness of simulation in the context of emergency management in
medical undergraduates.
We hypothesised that new medical graduates may not be
competent to initiate emergency care in unstable patients and that this
deficiency could be addressed through simulation-based education.
MethodOver a 6-month period, 71 medical undergraduates in
their 4th and 6th (final) year were scheduled to attend a 3-hour
simulation-based workshop.
Ethics approval was obtained, information provided to
participants and written consent obtained.
We used the Laerdal SimMan full-body computerised
manikin, integrated with monitoring devices, airway, and resuscitation
equipment. The manikin is capable of simulating speech and a range of clinical
signs including chest movement, breath sounds, heart sounds, and pulses. Airway
interventions, fluid, and drug administration, cardioversion, and defibrillation
are possible. The model will generate an ECG, blood pressure, pulse oximetry,
and capnography trace.
Students underwent an initial period of familiarisation
with the simulator and equipment. In the workshop, students worked in teams in
one of three scenarios, each team completing the same scenario twice (baseline
and repeat). The scenarios focussed on a theme of post-operative shock and were
set in a simulated surgical ward. A faculty nurse was present during the
scenarios and provided additional cues on ‘patient’ appearance and
could assist with monitoring tasks. Information was available in patient notes
and charts when requested.
The first 5-minute period of each scenario was
standardised and scripted to allow consistent scoring and comparison between
baseline and repeat performances. Scenarios were designed that required
immediate action within this 5-minute timeframe. After the initial test period,
if required, the faculty nurse could offer suggestions to students to direct
them towards appropriate management. This was to ensure that correct treatment
was eventually given, the ‘patient’ survived, and the simulation
experience was positive for the students.
The baseline scenario (5–7 minutes duration)
tested entry level skills. Workshop training consisted of participation in
simulations, observation of peers, feedback following simulations and a
facilitated discussion during which students developed a systematic approach to
the shocked patient. The repeat scenarios were of 10–15 minutes duration,
with the initial 5 minutes identical to the baseline.
Students gave written statements in response to a
questionnaire seeking their views on learning processes in simulation and use of
simulation in their assessment.
The videos of the baseline and repeat simulations were
randomised then assessed independently by two expert examiners not involved with
the workshop; these examiners scored the 5-minute test period of the scenario.
One examiner was a specialist anaesthetist and experienced simulation centre
instructor, and the second examiner was a senior New Zealand Resuscitation
Council instructor.
The examiners were blind to the year level of the
student team and the order of the scenarios. One instructor had no prior
knowledge of the students. An anchored five-point rating scale was used to score
three dimensions of performance: systematic approach to the problem, clear
leadership and division of tasks between team members. The scores for the three
dimensions of performance were averaged, and the mean of the two
examiners’ scores taken to give a global score for performance in each
baseline and repeat scenario. In addition, a checklist score was generated from
a list of key clinical management tasks.
Statistical
analyses—Quantitative data from questionnaires were analysed using
descriptive statistics and written responses were coded and grouped into themes.
The Wilcoxon matched-pairs signed-ranks test was used
to compare global scores in the baseline and repeat scenarios and to compare 4th
year and 6th year scores.
The Mann Whitney U test was used to compare checklist
scores of 4th and 6th year students.
ResultsAll 71 students who attended the
workshops agreed to take part in the study (45 fourth-year students, 26
sixth-year students). All students completed the questionnaire. A total of 21
pairs of baseline and repeat scenarios were scored by the two examiners (Table
1).
Table 1. Scores for
4th- and
6th-year students for the three baseline and
repeat scenarios
*Median score of all the teams
at that year level; n=number of teams rated on a rating scale of 1 to 5 where 1
is poor and 5 is good.
Results of video ratings of
performance—On a scale of 1–5, where a score of three equated
with adequate performance, the median score in the baseline scenario was 1.83
for the 4th-year students and 2.3 for the
6th-year students. In the repeat scenarios,
median scores improved to 2.67 and 3.33 for the
4th- and
6th-year students, respectively.
The median scores for the two groups in each of the three
scenarios (Table 1) shows evidence of improvement in each of the three
scenarios. Combining all pre-test and all post-test scores, overall there was a
significant improvement from baseline to repeat (p<0.001). Scores were
compared to see if there was a difference between
4th- and
6th-year students. The median scores for
6th-year students were significantly higher
than for 4th-year students for both baseline
(p<0.01) and repeat scenarios (p<0.001).
The students performed significantly more (p<0.05) key
tasks in the repeat scenarios than the baseline scenarios (Table 2).
Table 2. Comparison of average checklist scores for
baseline and repeat scenarios (total possible scores 21, 20, 21 for scenarios 1,
2, and 3 respectively)
Students compared the process of learning in the simulation
workshop with their traditional education in this domain. All 71 students
identified advantages, and a number of themes emerged from the written
statements (Table 3). Active engagement with the material, and the need to make
decisions and to commit to action in a realistic timeframe were seen as
effective ways of learning and remembering. Students felt they had to
‘make decisions quickly that really matter’ and ‘look at the
findings because you understand the importance’.
Students valued hands-on practice of clinical skills, the
use of equipment ‘to see how things would be done in practice’, and
discovered how difficult it was to actually manage a case compared to discussing
what should be done, as illustrated by the following student.
‘Theory’s all
very well but you gotta (sic) know how to turn on the oxygen before you can
administer it’
The transfer of theoretical knowledge to practice was
recognised as an essential step, which was facilitated by the simulation
workshop:
‘Even though we may
know the theory, it is much different in practice’
For many students, the simulations provided a memorable and
realistic experience from which to learn:
‘It’s a
wonderful way to learn because you remember the situation and the devised plan
of attack’; ‘[The] experience sticks in your mind because you are
using all 5 senses’
Simulation helped students see the relevance of their
theoretical knowledge and sort out what was important:
‘With reading, it is
difficult to pick out the key points, here it is obvious’
The simulations identified gaps in students’ knowledge
and motivated learning, characterised by this student’s
thoughts:
‘It motivates me to
thoroughly learn the theory as I have realised the seriousness of emergencies /
resuscitation and the implications of being ignorant’
Seventy out of the 71 students (99%) agreed that they had
identified areas they needed to learn more about, and 67 identified specific
areas they intended to address. Eleven students commented on the stressful
nature of the experience. They felt ‘thrown in at the deep end’, and
felt under pressure. However, without exception, these same students linked this
feeling to a positive learning experience, describing the value of ‘hands
on experience under pressure’. The stress made the experience more
memorable, and in fact, was described as ‘a good way to get over
it’.
Table 3. Thematic analysis of written responses on the
process
*N=number of students
responding in each theme.
Most students thought that the simulator could provide a
fair measure of their ability to manage a critically ill patient and 91% thought
it should be included in a hypothetical end of year assessment (Table 4).
Sixty-five students gave written responses on the subject of assessment. The
realism of the simulation assessment emerged as a prominent theme (n=27). It was
“obviously testing what one needs to know,” and it tested teamwork
and the ability to solve problems under pressure. Students recognised limits to
fidelity (n=12), but also commented that simulation was more valid than written
tests. They considered prior training in the simulator to be a prerequisite of
any assessment (n=13). Assessment in a team was considered as potentially
unfair; but on the other hand, the ability to work in a team was important and
reduced the stress of the simulation (n=13). (Numbers in brackets are number of
responses in that category)
Table 4. Students’ opinions of simulation as an
assessment tool.
DiscussionIn this study, we found that the
ability of final year medical students was assessed as less than satisfactory in
the domain of emergency care, and that performance was improved following a
simulation-based workshop. Students saw simulation as a credible and acceptable
assessment tool, and the consistently higher scores achieved by more senior
students support construct validity of simulation-based assessment. The learning
processes described by students in the simulation workshops were considered more
effective than traditional educational methods in this context and were
consistent with known principles of effective
learning.10
Simulation-based education enables the direct application of
learning theory to practice. The theory of constructivism sees the teacher as
the facilitator who provides students with relevant experiences targeted to
their level of understanding. Learning occurs where students engage actively
with the task provided, and is consolidated by in-depth examination of the new
experience. The theory of reflective practice argues that professional
competence cannot be achieved through formal teaching, but requires exposure to
the ‘messy’ problems of real life. Unexpected events or surprises
trigger reflection during the event, so called ‘learning in action’,
while subsequent thinking back on what happened, or ‘reflection on
action’ relates the event to prior experiences and consideration of how
this may affect future practice.
Demonstrating outcomes of educational interventions can be
problematic. A number of confounding variables influence results, and unlimited
access to students for randomised controlled trials is not feasible due to
ethical and resource constraints. A study by Morgan et al
6 compared simulation with video-assisted
learning and found no difference in learning effect. However, neither method is
standard practice, and in fact there may be little difference in resource
requirements. A more useful approach would be to compare simulation with
currently used methods. In this study, we compared simulation with the status
quo. We considered the baseline performance of final year students nearing
completion of their training as a measure of the effectiveness of the
traditional approach over their 6 years of training.
A previous study explored what it was that students learnt
in simulation workshops in the domain of emergency
care.11 Rather than focussing on specific
aspects of medical management, they learnt how to be more systematic in their
approach to a problem, how to work together in teams, and how to communicate
more effectively. We have taken this further to explore the underlying learning
processes in simulation-based education. It is interesting that students
identified a number of accepted principles of effective
learning,10,12,13 supporting the sound
educational basis of simulation.
Simulation can help students identify gaps in their
learning, and motivate them to learn more. These are key factors in promoting
self-directed, life-long learning. Kaufman10
proposed a number of principles to guide teaching practice. These include
actively engaging the learner, solving real-life problems, providing
opportunities for practice, giving feedback, and facilitating reflection on
practice through analysing performance and developing new perspectives and
opinions. Clearly these are applicable to teaching in many contexts, but the
ease with which they can be applied in simulation based education underlies the
power of this innovative teaching tool.
Patient simulators may be useful to assess competence of
medical students, and previous studies show that reliable scores can be
generated. Boulet et al7 demonstrated
reliability and construct validity of simulation-based assessment using highly
specified checklists to score acute care skills.
Morgan5 used checklists to score anaesthesia
skills in medical undergraduates. Global scores by experts have been shown to be
more valid and reliable than checklist scores to assess complex performance, as
checklists tend to reward thoroughness rather than competence, and may not allow
for alternative approaches to a problem.14
For assessing overall performance in these highly
contextualised scenarios, we considered global scores more appropriate than
checklists. We demonstrated that senior students scored more highly, supporting
construct validity of simulation-based assessment.
Morgan and Cleave-Hogg15
found students had a positive attitude to simulation-based assessment of basic
anaesthesia skills. We have demonstrated this is ‘generalisable’ to
assessment of emergency management, where 91% of students thought it should be
part of their formal assessment.
Limitations—It
is possible that scores may improve in repeated scenarios through increased
familiarity with the simulator and the environment. We attempted to minimise
this effect through a period of familiarisation with the simulator prior to the
scenarios. Increased student exposure to simulation, or cross-over designs with
different test scenarios (but similar exposure to simulation) could address this
problem, but were not feasible within the time constraints of the student
curriculum.
For a high stakes assessment, a large number of cases would
be required,16 and the format used in this
study could not reliably rank individual student teams. The combined scores,
however, provide a meaningful comparison between the two student groups and
between baseline and repeat scenarios.
We demonstrated improvement in performance by the end of the
workshop. We are unable to say how long this would persist. Students’
comments suggest simulation aids retention of new learning. It would be
interesting to know if the acquired knowledge was durable, and if refresher
courses were of benefit. This could be a productive area for further study.
Author information:
Jennifer Weller, Senior Lecturer, Department of Anaesthesiology, Faculty of
Medicine and Health Sciences, University of Auckland, Auckland; Brian Robinson,
Director National Patient Simulation Centre, Capital and Coast Health,
Wellington; Peter Larsen, Department of Surgery and Anaesthesia, Wellington
School of Medicine, University of Otago, Wellington; Catherine Caldwell,
Specialist Anaesthetist, Department of Anaesthesia, Capital and Coast Health,
Wellington
Acknowledgements:
The author is grateful for the support and assistance of Gabrielle Davie,
Biostatistician, Department of Public Health, Wellington School of
Medicine.
Correspondence:
Jennifer Weller, Faculty of Medicine and Health Sciences, University of
Auckland, PO Box 92019, Auckland. Fax (09) 623 6587; email: j.weller@auckland.ac.nz
References:
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| | Current
issue | Search journal |
Archived issues | Classifieds
| Hotline (free ads) Subscribe | Contribute | Advertise | Contact Us | Copyright | Other Journals |
|