View Article PDF

Graduating doctors have a regulatory obligation to ensure their medical decisions are based on ‘best evidence-based practice’.1 To meet this requirement doctors are expected to combine research evidence, alongside clinical experience and patient preferences, to inform clinical decisions. Studies of medical students and newly qualified doctors demonstrate that the primary source used for healthcare decision-making involves asking colleagues,2, 3 and resident physicians rarely use evidence-based sources to answer clinical questions.4 Barriers such as time, knowledge, skills in reading research and negative attitudes towards evidence-based medicine (EBM) may influence newly graduated doctors use of research informed practice.5,6

Medical schools aim to equip medical students with the necessary skills and competence in using research evidence to inform healthcare decisions. In a New Zealand medical school, research skills are taught explicitly to second- and third-year medical students through didactic teaching and self-study. In the fourth, fifth and sixth years of the programme, learning is case based and clinically driven. The later years rely on students being self-directed in using research evidence to answer clinically derived questions. This migration towards independently driven, clinically orientated EBM is necessary given these are the conditions under which graduate doctors are expected to practise medicine. However, international work has demonstrated research-informed practice may not improve in the senior years of medical training,7 and it is not clear whether a self-directed approach serves medical students well in developing the requisite skills and competence.

It is recognised that the practice of newly graduated doctors is significantly influenced by the practice of their colleagues.2 Given that senior doctors are role models and may be viewed as experts of medical practice by medical trainees, their attitudes and use of research in clinical decision making may significantly influence the practice of trainees.8 International literature has reported that clinicians’ adherence to the EBM process may vary depending on medical sub-specialty and stage of career, and senior clinicians have been found to be resistant to changing practice based on literature.8 Sub-optimal role models have been noted as a barrier to EBM in a number of studies.8–10 There is currently no empirical data in a New Zealand healthcare context on whether medical students observe senior clinicians undertaking EBM in clinical practice.

It is proposed in social cognitive theory that levels of self-efficacy with an activity influence participation in the given activity.11 It follows that medical students who perceive themselves competent with use of a skill may be more likely to use that skill in practice. Medical educators have been increasingly interested in their students’ self-efficacy with clinical skills for this reason.12 There is no New Zealand-based data that identify whether medical students perceive themselves as competent with EBM or whether perceptions of competence change during the senior years of medical training. Furthermore, no New Zealand-based studies have explored whether students’ subjective judgements of their own EBM competence are accurate in comparison to a validated competency measurement tool.

This study aims to identify whether medical students’ self-perceived and objectively measured competence with EBM changes during the senior years of medical training. A secondary aim is to establish whether medical students report observing and participating in the EBM process in clinical practice environments.

Methodology

Design

A cross-sectional survey was undertaken with students undertaking the Bachelor of Medicine and Bachelor of Surgery (MB ChB) degree at Otago University between February and April 2018.

Sample

A convenience sample of 99 medical students were asked to participate in the investigation. Students from one of the three regional campuses that deliver the programme at Otago University were recruited. The survey sampled fourth (n=37), fifth (n=32) and sixth (n=30) year students who were undertaking a module with the Department of Surgery and Anaesthesia.

Outcome measures

Self-perceived competence with EBM was measured using a questionnaire from a previous study.13 This questionnaire has demonstrated high internal consistency with one question removed.13 In the current investigation, this question was removed. Due to the investigation being undertaken at the start of the academic year, fourth-year students had not been exposed to clinical experiences by that time. Therefore, questions relating to the observation and practice of EBM in clinical environments were obtained from fifth- and sixth-year students only.

Using the method described by Lai et al. (2011)13, a self-perceived EBM competence score was calculated for each participant by adding scores from question two and questions four to seven. These five questions contained 15 items that were scored from 1–4 or 1–5 on a Likert scale based on the participants level of agreement with statements. Combined, all items provided an overall score out of 62. Higher scores indicated greater self-perceived competence with EBM.

An additional five questions were added, which investigated students’ training, exposure and attitudes towards EBM. The additional questions were not used to calculate final scores of self-perceived EBM competence. The final self-perceived competence questionnaire was made up of 10 questions and 22 discrete items, alongside three demographic questions (Appendix Figure 1).

The Assessing Competence in Evidence-Based Medicine (ACE) tool14 was used to quantify true competence. This tool has previously been demonstrated to have discriminant validity in identifying levels of EBM competence in medical students, alongside suitable internal consistency and content validity.14 The ACE tool consists of 15 items over four EBM domains: formulating a question, literature searching, appraising the evidence and applying the evidence. Items are based on a patient scenario, and each item requires a yes/no response. The questionnaire takes approximately 13 minutes to complete and is scored on a linear scale from 0 (low score) to 15 (high score).

Survey administration

Questionnaire surveys were distributed at the beginning of the academic year in an attempt to reduce inconsistencies arising from learning experiences occurring throughout the year and to maximise response rate by selecting a period not in proximity to examinations. Students were introduced to the aims of the investigation and asked to participate via an information leaflet within tutorial sessions. Time was allocated within the module to complete the questionnaire.

Ethical consideration

Ethical oversight was provided by the department curriculum committee and head of department. Formal ethics committee approval was not required for the current investigation as it represented part of a curriculum evaluation undertaken as a quality improvement activity. All surveys were completed anonymously. It was emphasised to students that selecting to complete the survey was voluntary and choosing not to complete the survey would not influence their standing at the university.

Data analysis

Data was inputted into the Statistical Package for the Social Sciences (SPSS 24.0, IBM Corp) and analysed descriptively using summary data. A p-value of <0.05 was determined to represent statistical significance. Sum competence scores were considered ratio level data and therefore parametric tests were used to analyse this data. However, in the case of parametric assumptions being violated (eg, normality), the non-parametric equivalent tests were used. The Shapiro–Wilk test and Levene’s test were used to assess the distribution and variance of ratio data respectively. Self-perceived competence scores were considered ordinal level data therefore non-parametric tests were used in analysis of this data. Homogeneity of variance of non-parametric data was assessed using visual inspection of histograms.

Results

Three students declined to participate, leaving 96 medical students who completed the final questionnaires. The survey response rate was 97%.

Demographics

The majority of students (84%, n=81) were between 20 and 25 years of age. Fifty-nine percent of the participants were female.

In answering a question about research experience, 47% (45) of participants indicated they had previous research experience including a postgraduate qualification, PhD/MSc or a summer studentship, or they had previously published a paper (Table 1).

Table 1: Participants research experience.

Self-perceived competence with EBM

Forty percent (n=40) of students were satisfied with their results from literature searches the ‘majority, most, or all of the time’. Just over half the participants (52%, n=50) indicated that they could tell apart a good study from a not so good study ‘often, most, or all of the time’.

Participants reported being able to understand the introduction and conclusion section of research articles ‘sufficiently’ or ‘fully’ in 97% of cases (n=93). The results and methods section were reported to be understood ‘sufficiently’ or ‘fully’ by 78% (n=75) and 60% (n=58) of participants, respectively (Table 2).

Seventy percent (n=67) of participants reported they were confident with appraising ‘all’ or ‘certain types of study’. Thirty percent (n=29) indicated they either ‘need a lot of guidance appraising all types of studies’ or they ‘haven’t a clue about critical appraisal’.

Median self-perceived competence scores are reported in Table 3. A Kruskal–Wallis test demonstrated significant differences in self perceived competence with EBM between year groups four, five and six: X2(2)=8.922, p=.012. Post hoc pairwise comparisons using Dunn’s (1964) procedure with Bonferroni correction found significant differences between years four and six (p=.019) and no significant differences between years four and five (p=.063) and five and six (p=1.0). A Mann–Whitney U test demonstrated median self-perceived competence with EBM was significantly higher in students who reported EBM experience (56.89) compared to those that did not (41.10): U=770, Z=-2.776, p=.006.

Competence with EBM

Mean EBM competence scores are summarised in Table 3. A Kruskal–Wallis test found no significant difference in EBM competence between year groups (X2=5.54, (2,93) p=0.063).

A Spearman’s correlation coefficient identified a weak positive association between scores of self-perceived competence and scores of true competence (rs=.246, N=96, p=.016).

An independent t-test found no significant difference between students with and without previous EBM experience and true competence scores: t(94)=1.372, p=.173.

Table 2: Participants self-perceived competence with understanding the different components of a research article.

Table 3: Median self-perceived competence (out of a total of 62) (IQR) and mean competence with EBM by year group (±SD).

Observation of, and participation in, EBM in clinical practice environments

Thirty one percent (n=18) of participants reported observing EBM ‘many times’ in clinical practice. A larger number (47%, n=28) reported observing EBM ‘once or twice’. A smaller number indicated they have ‘never’ observed EBM (22%, n=13) in clinical practice (Figure 1).

Twelve percent of participants (n=7) reported to having participated in EBM ‘many times’ in clinical practice. The largest number (47%, n=27) reported to have participated ‘once or twice’ with EBM in clinical practice. Forty-one percent (n=24) reported ‘never’ having participated in EBM in clinical practice (Figure 1).

Figure 1: Number of participants who report having observed and participated in EBM in clinical practice.

A Spearman’s correlation found a moderate positive association between participants exposure to observing EBM in clinical practice, and the frequency of participation in EBM in their own clinical practice (rs=.486, N=58, p=.0001). A weak association was found between students perceived competence with EBM and their reported participation in EBM in clinical practice (rs=.337, N=58, p=.01).

Attitudes towards EBM

Only 23% (n=22) of participants agreed ‘Strongly’ or ‘Somewhat’ that EBM was too time consuming to be done during day-to-day patient care. Very few participants ‘Strongly agreed’ or ‘Somewhat agreed’ that EBM is ‘cookbook’ medicine (7%, n=7) or that EBM relies too much on statistics (8%, n=8) (Table 4).

Table 4: Participants agreement with statements around the relevance of EBM.

Discussion

Findings of the current investigation demonstrate a gap between self-perception and the performance of medical students with competence in evidence-based medicine (EBM). Although students perceived that their competence in EBM improved through some of the senior years of medical training, measures on a validated competency tool do not show this to be true. Comparable findings have been demonstrated where competence with EBM did not improve during the mid to senior years of medical training.7 Students with research experience rated their self-perceived competence with EBM significantly higher than those with no research experience. However, it is interesting to note that this didn’t influence their overall competence with EBM, as measured with a validated tool. The mean competence score on the ACE tool achieved by students within this investigation was similar to a larger sample of senior medical students in Australia14 and lower than third-year medical students following focused EBM training.15 This suggests that the lack of improvement in EBM competence in the current students was not observed due to a high existing performance and ceiling effect. Results demonstrate only a weak association between students’ self-perceived competence with EBM and measures of true competence. Previous research has noted a lack of correlation between students’ judgements of competence with EBM and their performance on competency tools.16 Findings of the current investigation are consistent with international research that demonstrates students may not be able to accurately self-assess their EBM competence.13

An important finding in the current investigation was that a substantial number of senior medical students (41%) report never having followed the process of EBM in clinical practice environments. Furthermore, nearly two thirds of students reported infrequently observing EBM in clinical practice. The lack of explicit role modelling from senior clinicians in incorporating research evidence into decision-making may have influenced students’ perception of the value of using research evidence to inform clinical decisions. Previous research has demonstrated in medical residents that the practice of their supervisors towards EBM had an enduring career impact on residents’ attitudes towards EBM.17 An interesting finding in the current investigation was the moderate positive association between students that observed EBM in clinical practice being more likely to have participated in the process of EBM in clinical environments. Previous qualitative research from a large international study found that sub-optimal role models were the most frequently cited barrier to EBM in medical programmes.18 Other work has suggested senior medical staff do not use research informed practice and do not respond well to having their historical beliefs about treatments challenged with new research evidence.10 It is possible that students in the current study witnessed the application of EBM within clinical management, but that this was concealed due to the process of information seeking and appraisal occurring outside the clinical environment. The current findings suggest that, as in other countries, explicit role modelling of EBM within clinical practice environments in New Zealand is sub-optimal and may be a barrier to students participating in EBM.

Despite other work indicating that students possess negative attitudes towards EBM6,10 and that lack of time is a barrier to research informed clinical practice,8,9 the current results do not support these assertions. Participants in this cohort did not appear to have negative attitudes towards EBM. Only a minority of students agreed with the statement that EBM was ‘cookbook medicine’ or ‘relied too much on statistics’. Furthermore, less than a quarter of participants in the current investigation agreed that ‘EBM is time consuming and cannot be done during day-to-day patient care’, suggesting the time to identify, appraise and apply research evidence was seen as a challenge less frequently by New Zealand medical students, in comparison to students internationally.9 These findings may reflect this cohort’s positive attitude towards EBM and suggest that students who perceive value in undertaking EBM may prioritise allocating time to this aspect of practice as a priority. An explanation for these results may be the focus of EBM at the given medical school. At the current institution, EBM is purposely interwoven within all areas of the curricula, particularly within case discussions and problem-based learning. This approach is different from standalone teaching of EBM topics and positions EBM as an important fabric of medical practice.18 Given medical students are conscious of the exponential increases in information available in modern clinical practice19, it is possible they are considerate of the importance of refining skills that assist them in identifying high-quality information and developing the ability to adequately appraise material prior to its integration within decision-making. Triangulating the questionnaire data with student interviews would have provided richer information and would be useful in future work.

Although self-perceived EBM competence scores increased throughout the senior years of medical training (Table 3), this was only significant between the fourth and sixth years. This finding may be attributable to an accumulation of research experience that students accrue as they progress through training. Interestingly, our findings demonstrated only a weak association between students’ self-perceived competence and participation in EBM in clinical environments. This suggests self-confidence with the process of EBM may not be a strong mediating factor to determine whether students participate with EBM in clinical environments.

A strength of the current investigation was the excellent questionnaire response rate (96%), which reduced the risk of selection bias. The tools used to measure true and self-perceived competence with EBM were previously validated and were found to have satisfactory psychometric properties. The dichotomous nature of responses on scores meant results were not contaminated by issues of inter-rater reliability or interpretation. Limitations to findings of the current investigation are that participants were from one campus of a New Zealand medical school, and so the results may reflect the local academic or clinical environments only. Results may not be generalisable to other medical programmes, particularly to other programmes internationally. However, the consistency of these findings with other literature suggests that these results may have wider applicability. Using questionnaire surveys to understand the attitudes of a particular group have limitations that themes cannot be explored and data may lack depth and rigour. By incorporating yes/no answers, the ACE tool has attracted criticism for not adequately capturing the full spectrum of thought processes needed to measure competence in a skill as complex as research-informed decision-making.20 Questionnaires in general have been criticised for measuring competence with EBM as most tools typically focus on one domain of EBM; critical appraisal.15 The ACE tool assesses multiple EBM domains and has been validated in a group of medical students, demonstrating discriminant validity between novice, intermediate and advanced EBM students,14 so it was determined appropriate for the current investigation.

Conclusion

This investigation found that, in a New Zealand undergraduate medical programme, students’ competence with EBM failed to improve significantly in their final years of medical training. Lack of explicit role modelling of EBM in clinical practice is likely to be a barrier to students implementing EBM skills. Practising medical doctors who supervise students may reflect on how they are modelling EBM and supporting research-informed practice. Time taken to implement EBM was not a barrier for students in this investigation, but it may be a barrier for clinical doctors in role modelling EBM.

Appendix

Appendix Figure 1: The Self-Perceived Evidence-Based Medicine Competence Questionnaire. See the complete questionnaire online: https://assets-global.website-files.com/5e332a62c703f6340a2faf44/6047cac433311e0f6d6d69f3_4839%20-%20appendix.pdf

Summary

Abstract

AIMS: To identify whether medical students’ self-perception of competence with evidence-based medicine (EBM) increases throughout their senior years of medical training. Furthermore, to identify whether their self-perception aligns with their true competence measured using a validated tool. This investigation also outlines whether students report observation of and participation in the process of EBM in clinical practice. METHODS: A cross-sectional survey was undertaken with a convenience sample of medical students in their fourth, fifth and sixth years of training at one campus site of Otago Medical School between February and April 2018. Self-perceived competence with EBM was measured using a 10-item questionnaire. True competence was measured using the Assessing Competency in Evidence-Based Medicine (ACE) tool. Students were asked to self-report their observation of and participation in the process of EBM in clinical settings. RESULTS: Out of 99 students invited to participate, we received a response rate of 97%. Participants included 37 fourth-year, 32 fifth-year and 27 sixth-year students. Mean self-perceived EBM competence was higher in sixth-year compared to fourth-year students. True competence was not significantly different between year groups. Medical students reported little observation of EBM in clinical settings, and few students reported to have participated in the process of EBM during clinical encounters. CONCLUSION: The lack of explicit role modelling of EBM in clinical environments may be a barrier to students improving EBM competence in the senior years of medical training.

Aim

Method

Results

Conclusion

Author Information

Nicholas J Clode: Department of Surgery and Anaesthesia, University of Otago Wellington, Wellington, New Zealand; Therapies Department, Hutt Valley DHB, Lower Hutt, New Zealand. Dr Kirsty Danielson: Department of Surgery and Anaesthesia, University of Otago Wellington, Wellington, New Zealand. Associate Professor Elizabeth Dennett: Department of Surgery and Anaesthesia, University of Otago Wellington, Wellington, New Zealand.

Acknowledgements

Correspondence

Nicholas J Clode, Therapies Department, Hutt Valley DHB, 638 High Street, Lower Hutt, 04 566 6999

Correspondence Email

nick.clode@huttvalleydhb.org.nz

Competing Interests

Nil.

1. Medical Council of New Zealand. Good Medical Practice. 2016.

2. McCord G, Smucker WD, Selius BA, et al. Answering Questions at the Point of Care: Do Residents Practice EBM or Manage Information Sources? Academic Medicine. 2007; 82:298-303.

3. Lai NM, Nalliah S. Information-seeking practices of senior medical students: the impact of an evidence-based medicine training programme. Educ Heal. 2010; 23.

4. Ramos K, Linscheid R, Schafer S. Real-time information-seeking behavior of residency physicians. Fam Med. 2003; 35:257-60.

5. Montori VM, Tabini CC, Ebbert JO. A qualitative assessment of 1st-year internal medicine residents' perceptions of evidence-based clinical decision making. Teaching and learning in medicine. 2002; 14:114-8.

6. Green ML, Ruff TR. Why do residents fail to answer their clinical questions? A qualitative study of barriers to practicing evidence-based medicine. Academic medicine : journal of the Association of American Medical Colleges. 2005; 80:176-82.

7. Smith AB, Semler L, Rehman EA, et al. A Cross-Sectional Study of Medical Student Knowledge of Evidence-Based Medicine as Measured by the Fresno Test of Evidence-Based Medicine. The Journal of emergency medicine. 2016; 50:759-64.

8. Ilic D, Forbes K. Undergraduate medical student perceptions and use of Evidence Based Medicine: A qualitative study. BMC Medical Education. 2010; 10:58.

9. van Dijk N, Hooft L, Wieringa-de Waard M. What Are the Barriers to Residents' Practicing Evidence-Based Medicine? A Systematic Review. Academic Medicine. 2010; 85:1163-70.

10. Bhandari M, Montori V, Devereaux PJ, et al. Challenges to the practice of evidence-based medicine during residents' surgical training: a qualitative study using grounded theory. Academic medicine : journal of the Association of American Medical Colleges. 2003; 78:1183-90.

11. Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychological review. 1977; 84:191-215.

12. Klassen RM, Klassen JRL. Self-efficacy beliefs of medical students: a critical review. Perspect Med Educ. 2018; 7:76-82.

13. Lai NM, Teng CL. Self-perceived competence correlates poorly with objectively measured competence in Evidence Based Medicine among medical students. BMC Medical Education. 2011; 11:25.

14. Ilic D, Nordin RB, Glasziou P, et al. Development and validation of the ACE tool: assessing medical trainees’ competency in evidence based medicine. BMC Medical Education. 2014; 14:114.

15. Buljan I, Jerončić A, Malički M, et al. How to choose an evidence-based medicine knowledge test for medical students? Comparison of three knowledge measures. BMC Medical Education. 2018; 18:290.

16. Keddis MT, Beckman TJ, Cullen MW, et al. Senior internal medicine residents' confidence with essential topics in evidence-based medicine taught during internship. Journal of graduate medical education. 2011; 3:490-6.

17. Swennen MH, van der Heijden GJ, Boeije HR, et al. Doctors' perceptions and use of evidence-based medicine: a systematic review and thematic synthesis of qualitative studies. Acad Med. 2013; 88:1384-96.

18. Maggio LA, ten Cate O, Chen HC, et al. Challenges to Learning Evidence-Based Medicine and Educational Approaches to Meet These Challenges: A Qualitative Study of Selected EBM Curricula in U.S. and Canadian Medical Schools. Academic medicine : journal of the Association of American Medical Colleges. 2016; 91:101-6.

19. Ghojazadeh M, Hajebrahimi S, Azami-Aghdash S, et al. Medical students' attitudes on and experiences with evidence-based medicine: a qualitative study. J Eval Clin Pract. 2014; 20:779-85.

20. Patell R, Raska P, Lee N, et al. Development and Validation of a Test for Competence in Evidence-Based Medicine. J Gen Intern Med. 2020; 35:1530-1536.

Contact diana@nzma.org.nz
for the PDF of this article

View Article PDF

Graduating doctors have a regulatory obligation to ensure their medical decisions are based on ‘best evidence-based practice’.1 To meet this requirement doctors are expected to combine research evidence, alongside clinical experience and patient preferences, to inform clinical decisions. Studies of medical students and newly qualified doctors demonstrate that the primary source used for healthcare decision-making involves asking colleagues,2, 3 and resident physicians rarely use evidence-based sources to answer clinical questions.4 Barriers such as time, knowledge, skills in reading research and negative attitudes towards evidence-based medicine (EBM) may influence newly graduated doctors use of research informed practice.5,6

Medical schools aim to equip medical students with the necessary skills and competence in using research evidence to inform healthcare decisions. In a New Zealand medical school, research skills are taught explicitly to second- and third-year medical students through didactic teaching and self-study. In the fourth, fifth and sixth years of the programme, learning is case based and clinically driven. The later years rely on students being self-directed in using research evidence to answer clinically derived questions. This migration towards independently driven, clinically orientated EBM is necessary given these are the conditions under which graduate doctors are expected to practise medicine. However, international work has demonstrated research-informed practice may not improve in the senior years of medical training,7 and it is not clear whether a self-directed approach serves medical students well in developing the requisite skills and competence.

It is recognised that the practice of newly graduated doctors is significantly influenced by the practice of their colleagues.2 Given that senior doctors are role models and may be viewed as experts of medical practice by medical trainees, their attitudes and use of research in clinical decision making may significantly influence the practice of trainees.8 International literature has reported that clinicians’ adherence to the EBM process may vary depending on medical sub-specialty and stage of career, and senior clinicians have been found to be resistant to changing practice based on literature.8 Sub-optimal role models have been noted as a barrier to EBM in a number of studies.8–10 There is currently no empirical data in a New Zealand healthcare context on whether medical students observe senior clinicians undertaking EBM in clinical practice.

It is proposed in social cognitive theory that levels of self-efficacy with an activity influence participation in the given activity.11 It follows that medical students who perceive themselves competent with use of a skill may be more likely to use that skill in practice. Medical educators have been increasingly interested in their students’ self-efficacy with clinical skills for this reason.12 There is no New Zealand-based data that identify whether medical students perceive themselves as competent with EBM or whether perceptions of competence change during the senior years of medical training. Furthermore, no New Zealand-based studies have explored whether students’ subjective judgements of their own EBM competence are accurate in comparison to a validated competency measurement tool.

This study aims to identify whether medical students’ self-perceived and objectively measured competence with EBM changes during the senior years of medical training. A secondary aim is to establish whether medical students report observing and participating in the EBM process in clinical practice environments.

Methodology

Design

A cross-sectional survey was undertaken with students undertaking the Bachelor of Medicine and Bachelor of Surgery (MB ChB) degree at Otago University between February and April 2018.

Sample

A convenience sample of 99 medical students were asked to participate in the investigation. Students from one of the three regional campuses that deliver the programme at Otago University were recruited. The survey sampled fourth (n=37), fifth (n=32) and sixth (n=30) year students who were undertaking a module with the Department of Surgery and Anaesthesia.

Outcome measures

Self-perceived competence with EBM was measured using a questionnaire from a previous study.13 This questionnaire has demonstrated high internal consistency with one question removed.13 In the current investigation, this question was removed. Due to the investigation being undertaken at the start of the academic year, fourth-year students had not been exposed to clinical experiences by that time. Therefore, questions relating to the observation and practice of EBM in clinical environments were obtained from fifth- and sixth-year students only.

Using the method described by Lai et al. (2011)13, a self-perceived EBM competence score was calculated for each participant by adding scores from question two and questions four to seven. These five questions contained 15 items that were scored from 1–4 or 1–5 on a Likert scale based on the participants level of agreement with statements. Combined, all items provided an overall score out of 62. Higher scores indicated greater self-perceived competence with EBM.

An additional five questions were added, which investigated students’ training, exposure and attitudes towards EBM. The additional questions were not used to calculate final scores of self-perceived EBM competence. The final self-perceived competence questionnaire was made up of 10 questions and 22 discrete items, alongside three demographic questions (Appendix Figure 1).

The Assessing Competence in Evidence-Based Medicine (ACE) tool14 was used to quantify true competence. This tool has previously been demonstrated to have discriminant validity in identifying levels of EBM competence in medical students, alongside suitable internal consistency and content validity.14 The ACE tool consists of 15 items over four EBM domains: formulating a question, literature searching, appraising the evidence and applying the evidence. Items are based on a patient scenario, and each item requires a yes/no response. The questionnaire takes approximately 13 minutes to complete and is scored on a linear scale from 0 (low score) to 15 (high score).

Survey administration

Questionnaire surveys were distributed at the beginning of the academic year in an attempt to reduce inconsistencies arising from learning experiences occurring throughout the year and to maximise response rate by selecting a period not in proximity to examinations. Students were introduced to the aims of the investigation and asked to participate via an information leaflet within tutorial sessions. Time was allocated within the module to complete the questionnaire.

Ethical consideration

Ethical oversight was provided by the department curriculum committee and head of department. Formal ethics committee approval was not required for the current investigation as it represented part of a curriculum evaluation undertaken as a quality improvement activity. All surveys were completed anonymously. It was emphasised to students that selecting to complete the survey was voluntary and choosing not to complete the survey would not influence their standing at the university.

Data analysis

Data was inputted into the Statistical Package for the Social Sciences (SPSS 24.0, IBM Corp) and analysed descriptively using summary data. A p-value of <0.05 was determined to represent statistical significance. Sum competence scores were considered ratio level data and therefore parametric tests were used to analyse this data. However, in the case of parametric assumptions being violated (eg, normality), the non-parametric equivalent tests were used. The Shapiro–Wilk test and Levene’s test were used to assess the distribution and variance of ratio data respectively. Self-perceived competence scores were considered ordinal level data therefore non-parametric tests were used in analysis of this data. Homogeneity of variance of non-parametric data was assessed using visual inspection of histograms.

Results

Three students declined to participate, leaving 96 medical students who completed the final questionnaires. The survey response rate was 97%.

Demographics

The majority of students (84%, n=81) were between 20 and 25 years of age. Fifty-nine percent of the participants were female.

In answering a question about research experience, 47% (45) of participants indicated they had previous research experience including a postgraduate qualification, PhD/MSc or a summer studentship, or they had previously published a paper (Table 1).

Table 1: Participants research experience.

Self-perceived competence with EBM

Forty percent (n=40) of students were satisfied with their results from literature searches the ‘majority, most, or all of the time’. Just over half the participants (52%, n=50) indicated that they could tell apart a good study from a not so good study ‘often, most, or all of the time’.

Participants reported being able to understand the introduction and conclusion section of research articles ‘sufficiently’ or ‘fully’ in 97% of cases (n=93). The results and methods section were reported to be understood ‘sufficiently’ or ‘fully’ by 78% (n=75) and 60% (n=58) of participants, respectively (Table 2).

Seventy percent (n=67) of participants reported they were confident with appraising ‘all’ or ‘certain types of study’. Thirty percent (n=29) indicated they either ‘need a lot of guidance appraising all types of studies’ or they ‘haven’t a clue about critical appraisal’.

Median self-perceived competence scores are reported in Table 3. A Kruskal–Wallis test demonstrated significant differences in self perceived competence with EBM between year groups four, five and six: X2(2)=8.922, p=.012. Post hoc pairwise comparisons using Dunn’s (1964) procedure with Bonferroni correction found significant differences between years four and six (p=.019) and no significant differences between years four and five (p=.063) and five and six (p=1.0). A Mann–Whitney U test demonstrated median self-perceived competence with EBM was significantly higher in students who reported EBM experience (56.89) compared to those that did not (41.10): U=770, Z=-2.776, p=.006.

Competence with EBM

Mean EBM competence scores are summarised in Table 3. A Kruskal–Wallis test found no significant difference in EBM competence between year groups (X2=5.54, (2,93) p=0.063).

A Spearman’s correlation coefficient identified a weak positive association between scores of self-perceived competence and scores of true competence (rs=.246, N=96, p=.016).

An independent t-test found no significant difference between students with and without previous EBM experience and true competence scores: t(94)=1.372, p=.173.

Table 2: Participants self-perceived competence with understanding the different components of a research article.

Table 3: Median self-perceived competence (out of a total of 62) (IQR) and mean competence with EBM by year group (±SD).

Observation of, and participation in, EBM in clinical practice environments

Thirty one percent (n=18) of participants reported observing EBM ‘many times’ in clinical practice. A larger number (47%, n=28) reported observing EBM ‘once or twice’. A smaller number indicated they have ‘never’ observed EBM (22%, n=13) in clinical practice (Figure 1).

Twelve percent of participants (n=7) reported to having participated in EBM ‘many times’ in clinical practice. The largest number (47%, n=27) reported to have participated ‘once or twice’ with EBM in clinical practice. Forty-one percent (n=24) reported ‘never’ having participated in EBM in clinical practice (Figure 1).

Figure 1: Number of participants who report having observed and participated in EBM in clinical practice.

A Spearman’s correlation found a moderate positive association between participants exposure to observing EBM in clinical practice, and the frequency of participation in EBM in their own clinical practice (rs=.486, N=58, p=.0001). A weak association was found between students perceived competence with EBM and their reported participation in EBM in clinical practice (rs=.337, N=58, p=.01).

Attitudes towards EBM

Only 23% (n=22) of participants agreed ‘Strongly’ or ‘Somewhat’ that EBM was too time consuming to be done during day-to-day patient care. Very few participants ‘Strongly agreed’ or ‘Somewhat agreed’ that EBM is ‘cookbook’ medicine (7%, n=7) or that EBM relies too much on statistics (8%, n=8) (Table 4).

Table 4: Participants agreement with statements around the relevance of EBM.

Discussion

Findings of the current investigation demonstrate a gap between self-perception and the performance of medical students with competence in evidence-based medicine (EBM). Although students perceived that their competence in EBM improved through some of the senior years of medical training, measures on a validated competency tool do not show this to be true. Comparable findings have been demonstrated where competence with EBM did not improve during the mid to senior years of medical training.7 Students with research experience rated their self-perceived competence with EBM significantly higher than those with no research experience. However, it is interesting to note that this didn’t influence their overall competence with EBM, as measured with a validated tool. The mean competence score on the ACE tool achieved by students within this investigation was similar to a larger sample of senior medical students in Australia14 and lower than third-year medical students following focused EBM training.15 This suggests that the lack of improvement in EBM competence in the current students was not observed due to a high existing performance and ceiling effect. Results demonstrate only a weak association between students’ self-perceived competence with EBM and measures of true competence. Previous research has noted a lack of correlation between students’ judgements of competence with EBM and their performance on competency tools.16 Findings of the current investigation are consistent with international research that demonstrates students may not be able to accurately self-assess their EBM competence.13

An important finding in the current investigation was that a substantial number of senior medical students (41%) report never having followed the process of EBM in clinical practice environments. Furthermore, nearly two thirds of students reported infrequently observing EBM in clinical practice. The lack of explicit role modelling from senior clinicians in incorporating research evidence into decision-making may have influenced students’ perception of the value of using research evidence to inform clinical decisions. Previous research has demonstrated in medical residents that the practice of their supervisors towards EBM had an enduring career impact on residents’ attitudes towards EBM.17 An interesting finding in the current investigation was the moderate positive association between students that observed EBM in clinical practice being more likely to have participated in the process of EBM in clinical environments. Previous qualitative research from a large international study found that sub-optimal role models were the most frequently cited barrier to EBM in medical programmes.18 Other work has suggested senior medical staff do not use research informed practice and do not respond well to having their historical beliefs about treatments challenged with new research evidence.10 It is possible that students in the current study witnessed the application of EBM within clinical management, but that this was concealed due to the process of information seeking and appraisal occurring outside the clinical environment. The current findings suggest that, as in other countries, explicit role modelling of EBM within clinical practice environments in New Zealand is sub-optimal and may be a barrier to students participating in EBM.

Despite other work indicating that students possess negative attitudes towards EBM6,10 and that lack of time is a barrier to research informed clinical practice,8,9 the current results do not support these assertions. Participants in this cohort did not appear to have negative attitudes towards EBM. Only a minority of students agreed with the statement that EBM was ‘cookbook medicine’ or ‘relied too much on statistics’. Furthermore, less than a quarter of participants in the current investigation agreed that ‘EBM is time consuming and cannot be done during day-to-day patient care’, suggesting the time to identify, appraise and apply research evidence was seen as a challenge less frequently by New Zealand medical students, in comparison to students internationally.9 These findings may reflect this cohort’s positive attitude towards EBM and suggest that students who perceive value in undertaking EBM may prioritise allocating time to this aspect of practice as a priority. An explanation for these results may be the focus of EBM at the given medical school. At the current institution, EBM is purposely interwoven within all areas of the curricula, particularly within case discussions and problem-based learning. This approach is different from standalone teaching of EBM topics and positions EBM as an important fabric of medical practice.18 Given medical students are conscious of the exponential increases in information available in modern clinical practice19, it is possible they are considerate of the importance of refining skills that assist them in identifying high-quality information and developing the ability to adequately appraise material prior to its integration within decision-making. Triangulating the questionnaire data with student interviews would have provided richer information and would be useful in future work.

Although self-perceived EBM competence scores increased throughout the senior years of medical training (Table 3), this was only significant between the fourth and sixth years. This finding may be attributable to an accumulation of research experience that students accrue as they progress through training. Interestingly, our findings demonstrated only a weak association between students’ self-perceived competence and participation in EBM in clinical environments. This suggests self-confidence with the process of EBM may not be a strong mediating factor to determine whether students participate with EBM in clinical environments.

A strength of the current investigation was the excellent questionnaire response rate (96%), which reduced the risk of selection bias. The tools used to measure true and self-perceived competence with EBM were previously validated and were found to have satisfactory psychometric properties. The dichotomous nature of responses on scores meant results were not contaminated by issues of inter-rater reliability or interpretation. Limitations to findings of the current investigation are that participants were from one campus of a New Zealand medical school, and so the results may reflect the local academic or clinical environments only. Results may not be generalisable to other medical programmes, particularly to other programmes internationally. However, the consistency of these findings with other literature suggests that these results may have wider applicability. Using questionnaire surveys to understand the attitudes of a particular group have limitations that themes cannot be explored and data may lack depth and rigour. By incorporating yes/no answers, the ACE tool has attracted criticism for not adequately capturing the full spectrum of thought processes needed to measure competence in a skill as complex as research-informed decision-making.20 Questionnaires in general have been criticised for measuring competence with EBM as most tools typically focus on one domain of EBM; critical appraisal.15 The ACE tool assesses multiple EBM domains and has been validated in a group of medical students, demonstrating discriminant validity between novice, intermediate and advanced EBM students,14 so it was determined appropriate for the current investigation.

Conclusion

This investigation found that, in a New Zealand undergraduate medical programme, students’ competence with EBM failed to improve significantly in their final years of medical training. Lack of explicit role modelling of EBM in clinical practice is likely to be a barrier to students implementing EBM skills. Practising medical doctors who supervise students may reflect on how they are modelling EBM and supporting research-informed practice. Time taken to implement EBM was not a barrier for students in this investigation, but it may be a barrier for clinical doctors in role modelling EBM.

Appendix

Appendix Figure 1: The Self-Perceived Evidence-Based Medicine Competence Questionnaire. See the complete questionnaire online: https://assets-global.website-files.com/5e332a62c703f6340a2faf44/6047cac433311e0f6d6d69f3_4839%20-%20appendix.pdf

Summary

Abstract

AIMS: To identify whether medical students’ self-perception of competence with evidence-based medicine (EBM) increases throughout their senior years of medical training. Furthermore, to identify whether their self-perception aligns with their true competence measured using a validated tool. This investigation also outlines whether students report observation of and participation in the process of EBM in clinical practice. METHODS: A cross-sectional survey was undertaken with a convenience sample of medical students in their fourth, fifth and sixth years of training at one campus site of Otago Medical School between February and April 2018. Self-perceived competence with EBM was measured using a 10-item questionnaire. True competence was measured using the Assessing Competency in Evidence-Based Medicine (ACE) tool. Students were asked to self-report their observation of and participation in the process of EBM in clinical settings. RESULTS: Out of 99 students invited to participate, we received a response rate of 97%. Participants included 37 fourth-year, 32 fifth-year and 27 sixth-year students. Mean self-perceived EBM competence was higher in sixth-year compared to fourth-year students. True competence was not significantly different between year groups. Medical students reported little observation of EBM in clinical settings, and few students reported to have participated in the process of EBM during clinical encounters. CONCLUSION: The lack of explicit role modelling of EBM in clinical environments may be a barrier to students improving EBM competence in the senior years of medical training.

Aim

Method

Results

Conclusion

Author Information

Nicholas J Clode: Department of Surgery and Anaesthesia, University of Otago Wellington, Wellington, New Zealand; Therapies Department, Hutt Valley DHB, Lower Hutt, New Zealand. Dr Kirsty Danielson: Department of Surgery and Anaesthesia, University of Otago Wellington, Wellington, New Zealand. Associate Professor Elizabeth Dennett: Department of Surgery and Anaesthesia, University of Otago Wellington, Wellington, New Zealand.

Acknowledgements

Correspondence

Nicholas J Clode, Therapies Department, Hutt Valley DHB, 638 High Street, Lower Hutt, 04 566 6999

Correspondence Email

nick.clode@huttvalleydhb.org.nz

Competing Interests

Nil.

1. Medical Council of New Zealand. Good Medical Practice. 2016.

2. McCord G, Smucker WD, Selius BA, et al. Answering Questions at the Point of Care: Do Residents Practice EBM or Manage Information Sources? Academic Medicine. 2007; 82:298-303.

3. Lai NM, Nalliah S. Information-seeking practices of senior medical students: the impact of an evidence-based medicine training programme. Educ Heal. 2010; 23.

4. Ramos K, Linscheid R, Schafer S. Real-time information-seeking behavior of residency physicians. Fam Med. 2003; 35:257-60.

5. Montori VM, Tabini CC, Ebbert JO. A qualitative assessment of 1st-year internal medicine residents' perceptions of evidence-based clinical decision making. Teaching and learning in medicine. 2002; 14:114-8.

6. Green ML, Ruff TR. Why do residents fail to answer their clinical questions? A qualitative study of barriers to practicing evidence-based medicine. Academic medicine : journal of the Association of American Medical Colleges. 2005; 80:176-82.

7. Smith AB, Semler L, Rehman EA, et al. A Cross-Sectional Study of Medical Student Knowledge of Evidence-Based Medicine as Measured by the Fresno Test of Evidence-Based Medicine. The Journal of emergency medicine. 2016; 50:759-64.

8. Ilic D, Forbes K. Undergraduate medical student perceptions and use of Evidence Based Medicine: A qualitative study. BMC Medical Education. 2010; 10:58.

9. van Dijk N, Hooft L, Wieringa-de Waard M. What Are the Barriers to Residents' Practicing Evidence-Based Medicine? A Systematic Review. Academic Medicine. 2010; 85:1163-70.

10. Bhandari M, Montori V, Devereaux PJ, et al. Challenges to the practice of evidence-based medicine during residents' surgical training: a qualitative study using grounded theory. Academic medicine : journal of the Association of American Medical Colleges. 2003; 78:1183-90.

11. Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychological review. 1977; 84:191-215.

12. Klassen RM, Klassen JRL. Self-efficacy beliefs of medical students: a critical review. Perspect Med Educ. 2018; 7:76-82.

13. Lai NM, Teng CL. Self-perceived competence correlates poorly with objectively measured competence in Evidence Based Medicine among medical students. BMC Medical Education. 2011; 11:25.

14. Ilic D, Nordin RB, Glasziou P, et al. Development and validation of the ACE tool: assessing medical trainees’ competency in evidence based medicine. BMC Medical Education. 2014; 14:114.

15. Buljan I, Jerončić A, Malički M, et al. How to choose an evidence-based medicine knowledge test for medical students? Comparison of three knowledge measures. BMC Medical Education. 2018; 18:290.

16. Keddis MT, Beckman TJ, Cullen MW, et al. Senior internal medicine residents' confidence with essential topics in evidence-based medicine taught during internship. Journal of graduate medical education. 2011; 3:490-6.

17. Swennen MH, van der Heijden GJ, Boeije HR, et al. Doctors' perceptions and use of evidence-based medicine: a systematic review and thematic synthesis of qualitative studies. Acad Med. 2013; 88:1384-96.

18. Maggio LA, ten Cate O, Chen HC, et al. Challenges to Learning Evidence-Based Medicine and Educational Approaches to Meet These Challenges: A Qualitative Study of Selected EBM Curricula in U.S. and Canadian Medical Schools. Academic medicine : journal of the Association of American Medical Colleges. 2016; 91:101-6.

19. Ghojazadeh M, Hajebrahimi S, Azami-Aghdash S, et al. Medical students' attitudes on and experiences with evidence-based medicine: a qualitative study. J Eval Clin Pract. 2014; 20:779-85.

20. Patell R, Raska P, Lee N, et al. Development and Validation of a Test for Competence in Evidence-Based Medicine. J Gen Intern Med. 2020; 35:1530-1536.

Contact diana@nzma.org.nz
for the PDF of this article

View Article PDF

Graduating doctors have a regulatory obligation to ensure their medical decisions are based on ‘best evidence-based practice’.1 To meet this requirement doctors are expected to combine research evidence, alongside clinical experience and patient preferences, to inform clinical decisions. Studies of medical students and newly qualified doctors demonstrate that the primary source used for healthcare decision-making involves asking colleagues,2, 3 and resident physicians rarely use evidence-based sources to answer clinical questions.4 Barriers such as time, knowledge, skills in reading research and negative attitudes towards evidence-based medicine (EBM) may influence newly graduated doctors use of research informed practice.5,6

Medical schools aim to equip medical students with the necessary skills and competence in using research evidence to inform healthcare decisions. In a New Zealand medical school, research skills are taught explicitly to second- and third-year medical students through didactic teaching and self-study. In the fourth, fifth and sixth years of the programme, learning is case based and clinically driven. The later years rely on students being self-directed in using research evidence to answer clinically derived questions. This migration towards independently driven, clinically orientated EBM is necessary given these are the conditions under which graduate doctors are expected to practise medicine. However, international work has demonstrated research-informed practice may not improve in the senior years of medical training,7 and it is not clear whether a self-directed approach serves medical students well in developing the requisite skills and competence.

It is recognised that the practice of newly graduated doctors is significantly influenced by the practice of their colleagues.2 Given that senior doctors are role models and may be viewed as experts of medical practice by medical trainees, their attitudes and use of research in clinical decision making may significantly influence the practice of trainees.8 International literature has reported that clinicians’ adherence to the EBM process may vary depending on medical sub-specialty and stage of career, and senior clinicians have been found to be resistant to changing practice based on literature.8 Sub-optimal role models have been noted as a barrier to EBM in a number of studies.8–10 There is currently no empirical data in a New Zealand healthcare context on whether medical students observe senior clinicians undertaking EBM in clinical practice.

It is proposed in social cognitive theory that levels of self-efficacy with an activity influence participation in the given activity.11 It follows that medical students who perceive themselves competent with use of a skill may be more likely to use that skill in practice. Medical educators have been increasingly interested in their students’ self-efficacy with clinical skills for this reason.12 There is no New Zealand-based data that identify whether medical students perceive themselves as competent with EBM or whether perceptions of competence change during the senior years of medical training. Furthermore, no New Zealand-based studies have explored whether students’ subjective judgements of their own EBM competence are accurate in comparison to a validated competency measurement tool.

This study aims to identify whether medical students’ self-perceived and objectively measured competence with EBM changes during the senior years of medical training. A secondary aim is to establish whether medical students report observing and participating in the EBM process in clinical practice environments.

Methodology

Design

A cross-sectional survey was undertaken with students undertaking the Bachelor of Medicine and Bachelor of Surgery (MB ChB) degree at Otago University between February and April 2018.

Sample

A convenience sample of 99 medical students were asked to participate in the investigation. Students from one of the three regional campuses that deliver the programme at Otago University were recruited. The survey sampled fourth (n=37), fifth (n=32) and sixth (n=30) year students who were undertaking a module with the Department of Surgery and Anaesthesia.

Outcome measures

Self-perceived competence with EBM was measured using a questionnaire from a previous study.13 This questionnaire has demonstrated high internal consistency with one question removed.13 In the current investigation, this question was removed. Due to the investigation being undertaken at the start of the academic year, fourth-year students had not been exposed to clinical experiences by that time. Therefore, questions relating to the observation and practice of EBM in clinical environments were obtained from fifth- and sixth-year students only.

Using the method described by Lai et al. (2011)13, a self-perceived EBM competence score was calculated for each participant by adding scores from question two and questions four to seven. These five questions contained 15 items that were scored from 1–4 or 1–5 on a Likert scale based on the participants level of agreement with statements. Combined, all items provided an overall score out of 62. Higher scores indicated greater self-perceived competence with EBM.

An additional five questions were added, which investigated students’ training, exposure and attitudes towards EBM. The additional questions were not used to calculate final scores of self-perceived EBM competence. The final self-perceived competence questionnaire was made up of 10 questions and 22 discrete items, alongside three demographic questions (Appendix Figure 1).

The Assessing Competence in Evidence-Based Medicine (ACE) tool14 was used to quantify true competence. This tool has previously been demonstrated to have discriminant validity in identifying levels of EBM competence in medical students, alongside suitable internal consistency and content validity.14 The ACE tool consists of 15 items over four EBM domains: formulating a question, literature searching, appraising the evidence and applying the evidence. Items are based on a patient scenario, and each item requires a yes/no response. The questionnaire takes approximately 13 minutes to complete and is scored on a linear scale from 0 (low score) to 15 (high score).

Survey administration

Questionnaire surveys were distributed at the beginning of the academic year in an attempt to reduce inconsistencies arising from learning experiences occurring throughout the year and to maximise response rate by selecting a period not in proximity to examinations. Students were introduced to the aims of the investigation and asked to participate via an information leaflet within tutorial sessions. Time was allocated within the module to complete the questionnaire.

Ethical consideration

Ethical oversight was provided by the department curriculum committee and head of department. Formal ethics committee approval was not required for the current investigation as it represented part of a curriculum evaluation undertaken as a quality improvement activity. All surveys were completed anonymously. It was emphasised to students that selecting to complete the survey was voluntary and choosing not to complete the survey would not influence their standing at the university.

Data analysis

Data was inputted into the Statistical Package for the Social Sciences (SPSS 24.0, IBM Corp) and analysed descriptively using summary data. A p-value of <0.05 was determined to represent statistical significance. Sum competence scores were considered ratio level data and therefore parametric tests were used to analyse this data. However, in the case of parametric assumptions being violated (eg, normality), the non-parametric equivalent tests were used. The Shapiro–Wilk test and Levene’s test were used to assess the distribution and variance of ratio data respectively. Self-perceived competence scores were considered ordinal level data therefore non-parametric tests were used in analysis of this data. Homogeneity of variance of non-parametric data was assessed using visual inspection of histograms.

Results

Three students declined to participate, leaving 96 medical students who completed the final questionnaires. The survey response rate was 97%.

Demographics

The majority of students (84%, n=81) were between 20 and 25 years of age. Fifty-nine percent of the participants were female.

In answering a question about research experience, 47% (45) of participants indicated they had previous research experience including a postgraduate qualification, PhD/MSc or a summer studentship, or they had previously published a paper (Table 1).

Table 1: Participants research experience.

Self-perceived competence with EBM

Forty percent (n=40) of students were satisfied with their results from literature searches the ‘majority, most, or all of the time’. Just over half the participants (52%, n=50) indicated that they could tell apart a good study from a not so good study ‘often, most, or all of the time’.

Participants reported being able to understand the introduction and conclusion section of research articles ‘sufficiently’ or ‘fully’ in 97% of cases (n=93). The results and methods section were reported to be understood ‘sufficiently’ or ‘fully’ by 78% (n=75) and 60% (n=58) of participants, respectively (Table 2).

Seventy percent (n=67) of participants reported they were confident with appraising ‘all’ or ‘certain types of study’. Thirty percent (n=29) indicated they either ‘need a lot of guidance appraising all types of studies’ or they ‘haven’t a clue about critical appraisal’.

Median self-perceived competence scores are reported in Table 3. A Kruskal–Wallis test demonstrated significant differences in self perceived competence with EBM between year groups four, five and six: X2(2)=8.922, p=.012. Post hoc pairwise comparisons using Dunn’s (1964) procedure with Bonferroni correction found significant differences between years four and six (p=.019) and no significant differences between years four and five (p=.063) and five and six (p=1.0). A Mann–Whitney U test demonstrated median self-perceived competence with EBM was significantly higher in students who reported EBM experience (56.89) compared to those that did not (41.10): U=770, Z=-2.776, p=.006.

Competence with EBM

Mean EBM competence scores are summarised in Table 3. A Kruskal–Wallis test found no significant difference in EBM competence between year groups (X2=5.54, (2,93) p=0.063).

A Spearman’s correlation coefficient identified a weak positive association between scores of self-perceived competence and scores of true competence (rs=.246, N=96, p=.016).

An independent t-test found no significant difference between students with and without previous EBM experience and true competence scores: t(94)=1.372, p=.173.

Table 2: Participants self-perceived competence with understanding the different components of a research article.

Table 3: Median self-perceived competence (out of a total of 62) (IQR) and mean competence with EBM by year group (±SD).

Observation of, and participation in, EBM in clinical practice environments

Thirty one percent (n=18) of participants reported observing EBM ‘many times’ in clinical practice. A larger number (47%, n=28) reported observing EBM ‘once or twice’. A smaller number indicated they have ‘never’ observed EBM (22%, n=13) in clinical practice (Figure 1).

Twelve percent of participants (n=7) reported to having participated in EBM ‘many times’ in clinical practice. The largest number (47%, n=27) reported to have participated ‘once or twice’ with EBM in clinical practice. Forty-one percent (n=24) reported ‘never’ having participated in EBM in clinical practice (Figure 1).

Figure 1: Number of participants who report having observed and participated in EBM in clinical practice.

A Spearman’s correlation found a moderate positive association between participants exposure to observing EBM in clinical practice, and the frequency of participation in EBM in their own clinical practice (rs=.486, N=58, p=.0001). A weak association was found between students perceived competence with EBM and their reported participation in EBM in clinical practice (rs=.337, N=58, p=.01).

Attitudes towards EBM

Only 23% (n=22) of participants agreed ‘Strongly’ or ‘Somewhat’ that EBM was too time consuming to be done during day-to-day patient care. Very few participants ‘Strongly agreed’ or ‘Somewhat agreed’ that EBM is ‘cookbook’ medicine (7%, n=7) or that EBM relies too much on statistics (8%, n=8) (Table 4).

Table 4: Participants agreement with statements around the relevance of EBM.

Discussion

Findings of the current investigation demonstrate a gap between self-perception and the performance of medical students with competence in evidence-based medicine (EBM). Although students perceived that their competence in EBM improved through some of the senior years of medical training, measures on a validated competency tool do not show this to be true. Comparable findings have been demonstrated where competence with EBM did not improve during the mid to senior years of medical training.7 Students with research experience rated their self-perceived competence with EBM significantly higher than those with no research experience. However, it is interesting to note that this didn’t influence their overall competence with EBM, as measured with a validated tool. The mean competence score on the ACE tool achieved by students within this investigation was similar to a larger sample of senior medical students in Australia14 and lower than third-year medical students following focused EBM training.15 This suggests that the lack of improvement in EBM competence in the current students was not observed due to a high existing performance and ceiling effect. Results demonstrate only a weak association between students’ self-perceived competence with EBM and measures of true competence. Previous research has noted a lack of correlation between students’ judgements of competence with EBM and their performance on competency tools.16 Findings of the current investigation are consistent with international research that demonstrates students may not be able to accurately self-assess their EBM competence.13

An important finding in the current investigation was that a substantial number of senior medical students (41%) report never having followed the process of EBM in clinical practice environments. Furthermore, nearly two thirds of students reported infrequently observing EBM in clinical practice. The lack of explicit role modelling from senior clinicians in incorporating research evidence into decision-making may have influenced students’ perception of the value of using research evidence to inform clinical decisions. Previous research has demonstrated in medical residents that the practice of their supervisors towards EBM had an enduring career impact on residents’ attitudes towards EBM.17 An interesting finding in the current investigation was the moderate positive association between students that observed EBM in clinical practice being more likely to have participated in the process of EBM in clinical environments. Previous qualitative research from a large international study found that sub-optimal role models were the most frequently cited barrier to EBM in medical programmes.18 Other work has suggested senior medical staff do not use research informed practice and do not respond well to having their historical beliefs about treatments challenged with new research evidence.10 It is possible that students in the current study witnessed the application of EBM within clinical management, but that this was concealed due to the process of information seeking and appraisal occurring outside the clinical environment. The current findings suggest that, as in other countries, explicit role modelling of EBM within clinical practice environments in New Zealand is sub-optimal and may be a barrier to students participating in EBM.

Despite other work indicating that students possess negative attitudes towards EBM6,10 and that lack of time is a barrier to research informed clinical practice,8,9 the current results do not support these assertions. Participants in this cohort did not appear to have negative attitudes towards EBM. Only a minority of students agreed with the statement that EBM was ‘cookbook medicine’ or ‘relied too much on statistics’. Furthermore, less than a quarter of participants in the current investigation agreed that ‘EBM is time consuming and cannot be done during day-to-day patient care’, suggesting the time to identify, appraise and apply research evidence was seen as a challenge less frequently by New Zealand medical students, in comparison to students internationally.9 These findings may reflect this cohort’s positive attitude towards EBM and suggest that students who perceive value in undertaking EBM may prioritise allocating time to this aspect of practice as a priority. An explanation for these results may be the focus of EBM at the given medical school. At the current institution, EBM is purposely interwoven within all areas of the curricula, particularly within case discussions and problem-based learning. This approach is different from standalone teaching of EBM topics and positions EBM as an important fabric of medical practice.18 Given medical students are conscious of the exponential increases in information available in modern clinical practice19, it is possible they are considerate of the importance of refining skills that assist them in identifying high-quality information and developing the ability to adequately appraise material prior to its integration within decision-making. Triangulating the questionnaire data with student interviews would have provided richer information and would be useful in future work.

Although self-perceived EBM competence scores increased throughout the senior years of medical training (Table 3), this was only significant between the fourth and sixth years. This finding may be attributable to an accumulation of research experience that students accrue as they progress through training. Interestingly, our findings demonstrated only a weak association between students’ self-perceived competence and participation in EBM in clinical environments. This suggests self-confidence with the process of EBM may not be a strong mediating factor to determine whether students participate with EBM in clinical environments.

A strength of the current investigation was the excellent questionnaire response rate (96%), which reduced the risk of selection bias. The tools used to measure true and self-perceived competence with EBM were previously validated and were found to have satisfactory psychometric properties. The dichotomous nature of responses on scores meant results were not contaminated by issues of inter-rater reliability or interpretation. Limitations to findings of the current investigation are that participants were from one campus of a New Zealand medical school, and so the results may reflect the local academic or clinical environments only. Results may not be generalisable to other medical programmes, particularly to other programmes internationally. However, the consistency of these findings with other literature suggests that these results may have wider applicability. Using questionnaire surveys to understand the attitudes of a particular group have limitations that themes cannot be explored and data may lack depth and rigour. By incorporating yes/no answers, the ACE tool has attracted criticism for not adequately capturing the full spectrum of thought processes needed to measure competence in a skill as complex as research-informed decision-making.20 Questionnaires in general have been criticised for measuring competence with EBM as most tools typically focus on one domain of EBM; critical appraisal.15 The ACE tool assesses multiple EBM domains and has been validated in a group of medical students, demonstrating discriminant validity between novice, intermediate and advanced EBM students,14 so it was determined appropriate for the current investigation.

Conclusion

This investigation found that, in a New Zealand undergraduate medical programme, students’ competence with EBM failed to improve significantly in their final years of medical training. Lack of explicit role modelling of EBM in clinical practice is likely to be a barrier to students implementing EBM skills. Practising medical doctors who supervise students may reflect on how they are modelling EBM and supporting research-informed practice. Time taken to implement EBM was not a barrier for students in this investigation, but it may be a barrier for clinical doctors in role modelling EBM.

Appendix

Appendix Figure 1: The Self-Perceived Evidence-Based Medicine Competence Questionnaire. See the complete questionnaire online: https://assets-global.website-files.com/5e332a62c703f6340a2faf44/6047cac433311e0f6d6d69f3_4839%20-%20appendix.pdf

Summary

Abstract

AIMS: To identify whether medical students’ self-perception of competence with evidence-based medicine (EBM) increases throughout their senior years of medical training. Furthermore, to identify whether their self-perception aligns with their true competence measured using a validated tool. This investigation also outlines whether students report observation of and participation in the process of EBM in clinical practice. METHODS: A cross-sectional survey was undertaken with a convenience sample of medical students in their fourth, fifth and sixth years of training at one campus site of Otago Medical School between February and April 2018. Self-perceived competence with EBM was measured using a 10-item questionnaire. True competence was measured using the Assessing Competency in Evidence-Based Medicine (ACE) tool. Students were asked to self-report their observation of and participation in the process of EBM in clinical settings. RESULTS: Out of 99 students invited to participate, we received a response rate of 97%. Participants included 37 fourth-year, 32 fifth-year and 27 sixth-year students. Mean self-perceived EBM competence was higher in sixth-year compared to fourth-year students. True competence was not significantly different between year groups. Medical students reported little observation of EBM in clinical settings, and few students reported to have participated in the process of EBM during clinical encounters. CONCLUSION: The lack of explicit role modelling of EBM in clinical environments may be a barrier to students improving EBM competence in the senior years of medical training.

Aim

Method

Results

Conclusion

Author Information

Nicholas J Clode: Department of Surgery and Anaesthesia, University of Otago Wellington, Wellington, New Zealand; Therapies Department, Hutt Valley DHB, Lower Hutt, New Zealand. Dr Kirsty Danielson: Department of Surgery and Anaesthesia, University of Otago Wellington, Wellington, New Zealand. Associate Professor Elizabeth Dennett: Department of Surgery and Anaesthesia, University of Otago Wellington, Wellington, New Zealand.

Acknowledgements

Correspondence

Nicholas J Clode, Therapies Department, Hutt Valley DHB, 638 High Street, Lower Hutt, 04 566 6999

Correspondence Email

nick.clode@huttvalleydhb.org.nz

Competing Interests

Nil.

1. Medical Council of New Zealand. Good Medical Practice. 2016.

2. McCord G, Smucker WD, Selius BA, et al. Answering Questions at the Point of Care: Do Residents Practice EBM or Manage Information Sources? Academic Medicine. 2007; 82:298-303.

3. Lai NM, Nalliah S. Information-seeking practices of senior medical students: the impact of an evidence-based medicine training programme. Educ Heal. 2010; 23.

4. Ramos K, Linscheid R, Schafer S. Real-time information-seeking behavior of residency physicians. Fam Med. 2003; 35:257-60.

5. Montori VM, Tabini CC, Ebbert JO. A qualitative assessment of 1st-year internal medicine residents' perceptions of evidence-based clinical decision making. Teaching and learning in medicine. 2002; 14:114-8.

6. Green ML, Ruff TR. Why do residents fail to answer their clinical questions? A qualitative study of barriers to practicing evidence-based medicine. Academic medicine : journal of the Association of American Medical Colleges. 2005; 80:176-82.

7. Smith AB, Semler L, Rehman EA, et al. A Cross-Sectional Study of Medical Student Knowledge of Evidence-Based Medicine as Measured by the Fresno Test of Evidence-Based Medicine. The Journal of emergency medicine. 2016; 50:759-64.

8. Ilic D, Forbes K. Undergraduate medical student perceptions and use of Evidence Based Medicine: A qualitative study. BMC Medical Education. 2010; 10:58.

9. van Dijk N, Hooft L, Wieringa-de Waard M. What Are the Barriers to Residents' Practicing Evidence-Based Medicine? A Systematic Review. Academic Medicine. 2010; 85:1163-70.

10. Bhandari M, Montori V, Devereaux PJ, et al. Challenges to the practice of evidence-based medicine during residents' surgical training: a qualitative study using grounded theory. Academic medicine : journal of the Association of American Medical Colleges. 2003; 78:1183-90.

11. Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychological review. 1977; 84:191-215.

12. Klassen RM, Klassen JRL. Self-efficacy beliefs of medical students: a critical review. Perspect Med Educ. 2018; 7:76-82.

13. Lai NM, Teng CL. Self-perceived competence correlates poorly with objectively measured competence in Evidence Based Medicine among medical students. BMC Medical Education. 2011; 11:25.

14. Ilic D, Nordin RB, Glasziou P, et al. Development and validation of the ACE tool: assessing medical trainees’ competency in evidence based medicine. BMC Medical Education. 2014; 14:114.

15. Buljan I, Jerončić A, Malički M, et al. How to choose an evidence-based medicine knowledge test for medical students? Comparison of three knowledge measures. BMC Medical Education. 2018; 18:290.

16. Keddis MT, Beckman TJ, Cullen MW, et al. Senior internal medicine residents' confidence with essential topics in evidence-based medicine taught during internship. Journal of graduate medical education. 2011; 3:490-6.

17. Swennen MH, van der Heijden GJ, Boeije HR, et al. Doctors' perceptions and use of evidence-based medicine: a systematic review and thematic synthesis of qualitative studies. Acad Med. 2013; 88:1384-96.

18. Maggio LA, ten Cate O, Chen HC, et al. Challenges to Learning Evidence-Based Medicine and Educational Approaches to Meet These Challenges: A Qualitative Study of Selected EBM Curricula in U.S. and Canadian Medical Schools. Academic medicine : journal of the Association of American Medical Colleges. 2016; 91:101-6.

19. Ghojazadeh M, Hajebrahimi S, Azami-Aghdash S, et al. Medical students' attitudes on and experiences with evidence-based medicine: a qualitative study. J Eval Clin Pract. 2014; 20:779-85.

20. Patell R, Raska P, Lee N, et al. Development and Validation of a Test for Competence in Evidence-Based Medicine. J Gen Intern Med. 2020; 35:1530-1536.

Contact diana@nzma.org.nz
for the PDF of this article

Subscriber Content

The full contents of this pages only available to subscribers.

LOGINSUBSCRIBE