  |
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Journal of the New Zealand Medical Association,
17-February-2006, Vol 119 No 1229
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Procedural skills of first-year postgraduate doctors at
Auckland District Health Board, New Zealand
Andrew Old, Gill Naden, Stephen Child
The Medical Council of New Zealand (MCNZ) is responsible,
through its education committee and hospital-based intern supervisors, for the
educational experience of doctors in their first postgraduate year (PGY1).
While there is currently no prescribed set of clinical
skills that must be attained and accredited during this time, the MCNZ does
publish an ‘Indicative List of Skills’1 designed to, “ensure
they [PGY1 doctors] achieve an appropriate breadth of experience during the
year”.
In 2002, a Danish study2 asked pre-registration house
officers (PRHOs), the equivalent of PGY1, to rate themselves in 210 practical
clinical skills that had been identified previously using a Delphi technique.3,4
Using this method, an expert panel assessed a range of clinical skills and
identified, by consensus, the expected level of competence of medical graduates
at time of graduation. With a response rate in the survey of 80%, it was
discovered that none of the PRHO respondents met the minimum level for all the
skills, with only 8% meeting the minimum requirements for at least 90%. On
average, the respondents met the minimum level of competence as defined by the
expert panel for only 74% of the skills.2
In this study, we surveyed a group of junior doctors at the
beginning and the end of their first postgraduate year to gauge their experience
with a pre-defined set of skills, paying particular attention to the sub-set
contained in the “Indicative List of Skills” published by the
MCNZ.
MethodsStudy
population—Auckland District Health Board (ADHB) is the largest
public healthcare provider in New Zealand, providing regional services for
approximately 415,000 people along with some national specialty services. There
are more than 7500 staff including approximately 500 resident medical officers
and nearly 3000 nurses.5
In November 2002, incoming postgraduate year 1 (PGY1)
doctors about to start work at ADHB were surveyed at their orientation session
and the survey was repeated at the end of their PGY1 year (November 2003).
Questionnaire—The
survey comprised 134 questions relating to personal experience and expertise
with various clinical skills, including procedural and communication skills (86
questions) as well as management of common conditions (48 questions).
Participants were asked to rate their experience or confidence with each skill
on a five or six-point scale (Table 1). A score of ‘3’ or more
identified clinical experience with the skill and was defined as
‘sufficient performance’ for the purposes of comparison. Demographic
data were collected and there were no personal identifiers on the
questionnaires.
Table 1. Survey rating scale
Questions for the survey were drawn from the skill set
used in the Danish study,2 and the MCNZ Indicative List of Skills.1 Only results
from the questions relating to the MCNZ Indicative List of Skills are reported
here.
Statistical
methods—Results are reported as mean plus or minus standard
deviation unless otherwise specified. Differences in MCNZ skill levels were
compared using an unpaired t-test as there was no identifier available to match
the pre-PGY1 and post-PGY1 questionnaires. All analysis was performed on SAS
statistical software and a p-value less than 0.05 was considered statistically
significant.
ResultsThirty-eight PGY1 house officers started at ADHB in November
2002. Of the 36 questionnaires distributed (two eligible participants did not
attend orientation), 30 were completed and returned (79% overall response rate).
Of the 30 doctors who originally completed the survey, 25 completed the repeat
in November 2003, with 5 doctors lost to follow-up. Reasons for attrition
included rotation away from ADHB and non-attendance at the scheduled feedback
session.
Due to ambiguity in the rating scale, the 48 questions
relating to management skills were unable to be meaningfully evaluated. The
sub-set of skills (28) that relate to the indicative list of skills published by
the MCNZ are reported. Table 2 summarises the characteristics of the study
population.
Table 2. Participant summary data (% of sample)
Table 3. MCNZ indicative list of skills
Results reported as mean± standard deviation. Scores
range from 1.7 to 4.9 where a score of 3 or more indicates clinical experience
with the procedure (refer to Table 1). GCS=Glasgow Coma Score.
Table 4. Percentage of participants who reported
performing the named skill in clinical practice
(identified as those rating the skill as 3 or
greater, refer Table 1)
Increases in the perceived level of experience were observed
in 19 (68%) of 28 skills from the MCNZ indicative list however only nine of
these increases reached statistical significance (Table 3). There were decreases
in the perceived level of experience in eight skills but none with statistical
significance, and seven of these were skills associated with Obstetrics and
Gynaecology which are not offered during PGY1 at ADHB (Table 3).
By the end of the PGY1 year, 100% of doctors had performed
in practice only 6 of 28 (21%) of the skills listed by the MCNZ, compared to 1
of 28 at the beginning of the year (Table 4).
The number of skills performed by at least 50% of the
doctors did not change significantly over the year (16 at the beginning, 17 at
the end). Put another way, 11 out of 28 skills (39%) defined as important to
achieve during PGY1 by the MCNZ, had been performed in practice by less than
half our sample at the end of their PGY1 year.
DiscussionOur study found that experience with most procedural skills
deemed important by the Medical Council of New Zealand does not increase
significantly during a doctor’s first postgraduate year working in a major
tertiary hospital. In addition, there is a significant discrepancy between the
skills expected of graduates at the end of PGY1 (as indicated by the MCNZ) and
those attained. Furthermore, most of the skills expected of PGY1 doctors are
present in the medical school curricula and it is interesting to note that of
the 28 skills discussed, just one had been performed by all the respondents at
the time of their graduation.
Obstetric & gynaecology runs are not currently offered
at PGY1 level at ADHB, while paediatrics is only available to two people. This
explains, in part, the relative lack of experience demonstrated in these
disciplines and, despite skills from these disciplines being included in the
MCNZ’s indicative list, general registration is still granted without this
experience. Given that the survey asked about experience, as opposed to
competence or confidence, the reported decline in experience with these skills
can be ignored. This is supported by the lack of statistical significance
associated with this decline.
Internationally, the historical trend has been to not
prescribe minimum skill sets, although many comparable countries (that have
previously issued ‘indicative’ lists similar to the MCNZ) are
currently moving to greater regulation. In Australia, individual states have
produced various suggested skill sets,6 while in the United Kingdom, the
revision of the New Doctor sets out “outcomes” rather than
“experience” that must be achieved prior to gaining full
registration.7 In addition, the recently instituted, two-year “Foundation
Programme” in the United Kingdom lists 41 key competencies that will be
assessed during the programme.8
The general theme of our results (i.e. that PGY1 doctors
generally do not meet defined core competencies) is supported by the
international literature with conclusions like “newly qualified doctors do
not feel prepared for PRHO duties and objectively are not competent in basic
clinical skills”,9 and “students...skills are deficient at the time
of graduation”10 being commonplace.
A South African study which tested 58 recent graduates in a
seven station OSCE found that average scores across all 7 stations was just
67.5% and that the candidates were “unduly optimistic” about their
competence 11. In a UK study, Smith et al found knowledge of acute care to be
“unsatisfactory” amongst recent graduates and called on medical
schools to “urgently incorporate training about common aspects of generic
acute care in their curricula”.12 Another study from the UK that tested
PRHOs against a 17 station OSCE of core clinical skills, concluded that
“PRHOs may have deficiencies in basic clinical skills at the time they
enter the medical register”.13 Furthermore, a larger study of 122 medical
students and 84 PRHOs (that asked about experience with 8 core skills) found
that most of the skills had only been performed a few times at qualification and
less than half of the PRHOs questioned could recall further postgraduate
training in any of them.14
What is encouraging, however, are findings such as those of
a 1999 study that found that early postgraduate trainees in a Sydney teaching
hospital did in fact acquire high levels of confidence and experience in most
skill areas after 2 years of training noting that “the first postgraduate
year is particularly significant for the development of clinical
skills”.15 It is important to note, however, that experience and
confidence have been found not to equate to observed competence.13,16
Our study is limited by the small sample size and by the
unfortunate attrition of five participants between the two surveys. Furthermore,
our study looked at self-perceptions of experience which, as outline above, have
been previously shown to bear little correlation to observed competence.
Notwithstanding the small absolute numbers, our study does
represent the experience of the majority of PGY1 medical staff at ADHB with 79%
(30 out of 38) completing the original survey and 66% (25 of 38) the follow-up.
Experience with most skills surveyed did not increase during the first
postgraduate year, and a significant proportion (11 of 28) of the skills
identified by the MCNZ had been performed by less than half the respondents.
Taking our findings in the context of the international
literature raises a number of questions. Are the skills being tested an accurate
reflection of the role of the PGY1 doctor? If so, why are they not being
achieved and should adequate competence with some or all of these skills be
required to be demonstrated before General Registration is granted? If not, are
there deficiencies with the PGY1 experience, or is the list of skills
inappropriate?
We would contend that the answer is likely to be a
combination of the above. Regarding the appropriateness of the lists, some
authors have argued that the most important skills a doctor can possess are the
tools to identify seriously ill patients, something the traditionally prescribed
skill sets tend to ignore.17,18 Furthermore, some skills, such as cannulation
and phlebotomy, are being increasingly taken over by other health professionals,
thus decreasing the exposure of new medical graduates. Finally, there are some
important skills, such as endotracheal intubation, that a PGY1 doctor is
unlikely to perform, but which are important nonetheless.
As previously mentioned, the MCNZ “Indicative List of
Skills” is not prescriptive and General Registration is granted on the
basis of satisfactory completion of four, 3-month rotations in a New Zealand
hospital. The lack of a prescribed standard of what constitutes acceptable
procedural experience poses difficulties for those responsible for skills
education programmes at both undergraduate and graduate level, those consultants
charged with supervision and assessment, as well as for new graduate doctors
themselves.
Given the discrepancy observed between the reported
experience of PGY1 house officers at ADHB and the indicative list of skills
published by the MCNZ, the growing body of medical literature that echo these
findings, and the moves toward greater regulation in comparable Western
countries, these authors propose the establishment of an internationally agreed,
assessed curriculum for PGY1 that would need to be satisfactorily completed
prior to gaining general registration.
Furthermore, given the changing medicolegal environment and
the increasing demand for public accountability, a more systematic approach to
ensure junior doctors are adequately trained to provide safe, competent care is
fast becoming a necessity. Such a programme would require the introduction of
more formal, competence-based education into the PGY1 year and a necessary shift
in focus from the service dominated working model that exists currently.
Proposed suggestions for achieving this include: limiting the number of patients
house officers manage at one time; relieving junior doctors of
“non-educational chores”; improving educational content; and easing
“emotional stresses”.19
Various models exist for the standardising of such
assessments, but two that are commonly discussed in the literature are the use
of standardised patients,20–23 and a method of clinical skills assessment
developed by the American Board of Internal Medicine known as the mini-CEX.24,25
In addition to the use of the mini-CEX, the UK Foundation Programme is employing
three other assessment techniques: ‘Direct Observation of Procedural
Skills’ (DOPS), ‘Case-based Discussion’ (CbD), and
‘Multi-Source Feedback’ (MSF), a form of 360 degree feedback.8 It is
beyond the scope of this paper to comment on the relative merits of each of
these methods, but it is worth considering that the institution of any
internationally agreed, assessed curriculum would also require agreement on
appropriate methods of testing.
Other options that have been suggested to be beneficial in
improving graduate doctors’ skills are an extended, specific orientation
period,9,26 and greater integration between undergraduate and graduate education
programmes 27. In New Zealand, a strength of our system is the trainee intern
year which provides some degree of integration between medical school and
pre-vocational training, although there is a need for greater collaboration on
appropriate skills, and methods of teaching and assessment, between medical
schools, PGY1 & 2 programme providers, and vocational colleges.
In New Zealand, the ‘Indicative List of Skills’
published by the MCNZ is currently the most authoritative guide to the expected
competencies of recently graduated doctors. This paper has highlighted
significant discrepancies between the skills listed and those attained, and we
call on the MCNZ to review this list, in consultation with all stakeholders and
in light of international developments, both to improve the guidance given to
new graduate doctors and their educators, and to ensure that the skills are
appropriate prior to any move to greater prescription of competencies in this
country.
Author information:
Andrew Old, Public Health Medicine Registrar; Gill Naden, Manager, Clinical
Education and Training Unit; Stephen Child, Director of Clinical Training;
Auckland District Health Board, Auckland
Acknowledgement:
We thank Teena West, Biostatistician, Auckland District Health Board for
statistical assistance.
Correspondence: Dr
Andrew Old, Clinical Education & Training Unit (CETU), Auckland District
Health Board, Private Bag 92024, Auckland. Fax: (09) 623 6421; email: andrew.old@woosh.co.nz
References:
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| | Current
issue | Search journal |
Archived issues | Classifieds
| Hotline (free ads) Subscribe | Contribute | Advertise | Contact Us | Copyright | Other Journals |
|