1st April 2016, Volume 129 Number 1432

Georgina Chan, Louret Bezuidenhout, Logan Walker, Robert Rowan

The process of prioritising patients for elective surgery remains a challenge. The New Zealand health system was restructured in 1992 with the aim of improving assessment and accountability in the publicly funded health sector.1 A National Health Committee (NHC) was given the task of providing independent advice to the Minister of Health. The recommendation was that surgical waiting lists be replaced by a booking system, with the aim that patients with the greatest need and potential to benefit would access surgery first.1 This new booking system was formally introduced in 1998. Patients received scores according to their clinical and social need for treatment, then, if a certain threshold score was reached, the patient would be booked for surgery and placed on the waiting list.2 Currently, there is no universally accepted tool for prioritising patients for orthopaedic surgery in New Zealand.

Patients referred to the orthopaedic service through their general practitioners are seen by a consultant orthopaedic surgeon at their first specialist appointment (FSA). Following assessment and discussion with the patient, they may be recommended for surgery. This is not an indication that they have been placed on the waiting list. A patient is placed on the waiting list once their score reaches a certain threshold, and receive surgery within a given timeframe. There remain far more patients recommended for surgery than funding allows, hence the need for a prioritisation tool.

The Clinical Priority Assessment Criteria (CPAC) was developed by national clinical working parties, which includes three dimensions: clinical, patient-experienced, and social. These dimensions generated a total score, to score patients from 0 to 100 (least to most urgent). The aim of priority criteria tools was to create equity, transparency and consistency to the process of allocating elective services.1,3-5 Evidence from the early 2000s suggests that there is still great variation among priority scoring systems. It was established that methods for calculating CPAC scores were inconsistent, and that surgeons often felt their clinical judgement was more effective in prioritising patients.3,6,7

Doughty and colleagues assessed the reliability of CPAC tools for elective surgery in New Zealand, concluding that vignette-based methods were feasible assessment tools,8 and there was a marked inter-surgeon variability when assessing the reliability of 3 different priority tools based upon clinical vignettes, highlighting the need for a reliable scoring system to ensure equity of access.9

Theis reported that generic priority criteria scores correlated well with quality of life measures and patients selected for surgery had significantly lower scores compared to those who were not, concluding that their tool had the ability to select patients with the highest need and ability to benefit.10 Patient centredness is vital to healthcare provision, as it has been shown to increase patient compliance and satisfaction. The disparity between patient and surgeon perception on how a certain condition affects the patient’s life warrants even more involvement of the patient in the prioritisation process.11

Prioritisation criteria are most effective when objective and subjective measures are included. This means using a condition-specific measure as well as a general health score. The Impact on Life (IoL), therefore, is designed to be used alongside disease-specific measures and does not need to capture condition-specific information.

The Western Canada Waiting List project emphasised the importance of ongoing evaluation of prioritisation tools. Characteristics associated with successful implementation and evaluation initiatives were outlined; for example, receptive management and steering committees as well as familiarity with tools.12 Criticism of prioritising tools was generally focused on the lack of validation prior to implementation.3,4,6,7,13

The IoL (Appendix 1) instrument was developed in conjunction with the Ministry of Health, specifically intended for use in prioritisation for elective surgery in New Zealand. It provides a unique approach to the assessment of impact of health conditions on everyday life. The IoL was first introduced for use in prioritising cataract surgery in 2005 and subsequently in plastic and reconstructive surgery in 2008. The intention of the Ministry is to implement its use across all surgical specialties in New Zealand. The IoL is a one-page, 6-item questionnaire applicable across a range of specialties, using a ‘think-aloud procedure’ completed by patients. It assesses the overall impact of health conditions on a patient’s activities of daily living, assuming that one or more condition/s may affect several domains concurrently. Chamberlain and McGuigan demonstrated that the questionnaire was successful for use in patients with different conditions, of varying age and was generally easy to understand and administer.14

We are not aware of any other tool that assesses impact on life for the purpose of prioritisation. The IoL has not been previously validated for use in orthopaedic surgery.

We examine the IoL and test its reliability and validity by comparing it with the SF-12 and Oxford scores, validated patient-rated health measures in orthopaedic surgery. The SF-12 and Oxford score were not designed as tests for surgical prioritisation, but have been used for this purpose in some hospitals in New Zealand. We chose to compare the IoL with these measures as they are scoring systems used at our hospital, and have been shown to predict patients’ benefit prospectively.6 The SF-12 provides a measure of a patient’s general health and the Oxford assesses how a specific joint affects an individual. This study assessed only patients who have been recommended for surgery, to determine whether the IoL is a reliable and valid prioritisation tool.

Methods

All patients over the age of 18 years, who attended orthopaedic specialist assessments at Wellington Regional Hospital between July and November 2013, and who were recommended for surgery, completed the IoL, SF-12 and, where applicable, hip, knee or shoulder Oxford questionnaires. The surgeon completed a priority scoring tool based on their clinical assessment for each patient.

The IoL questionnaire consists of six dimensions: social interaction, personal interaction, ability to fulfil responsibilities to others, personal care, personal safety, and leisure activities. The surgeon priority scoring tool (Appendix 2) includes three dimensions: potential deterioration, expected benefit and surgeon-assessed severity. The weighted IoL score accounted for 25%, and the surgeon score 75% to the final orthopaedic CPAC score, ranging from 0 to 100 (best to worst possible health).

The SF-12 is a widely-used tool consisting of 12 questions covering 8 general health dimensions: physical functioning, physical role-performance, emotional role performance, mental health, bodily pain, vitality, social functioning and an overall rating of general health.15

The Oxford questionnaire consists of 12 questions which assess a patient’s function and pain of a specific joint.16 Patients with hip, knee and shoulder problems completed the Oxford score and SF-12. Those who were recommended for surgery for other orthopaedic conditions completed only the SF-12. Patients under the age of 18 and those who failed to complete questionnaires were excluded. The worst response was adopted for questions which returned with more than one answer.

Analysis of the data was performed using SPSS Statistics Version 21 (IBM). Reliability testing was completed using Cronbach’s coefficient alpha, which measures the internal consistency of questionnaires, and is commonly used to assess reliability.17 A Cronbach’s alpha of greater than 0.7 indicates good reliability, with 0.9 being the recommended coefficient for clinical application.17 Construct validity was completed using correlation analysis. This was also performed to delineate any relationship between patient-rated score and surgeon-assessed severity. T-tests were completed to determine whether there were any differences between patient- and surgeon-rated scores with regards to those patients that were added to the waiting list compared to those that were not.

We did not measure inter-observer or intra-observer reliability for the IoL as it is a test applied to an individual to measure their disability at one instant in time.

Results

1,324 FSA patients were seen through our clinic between July and November 2013. 324 patients (53% female) were recommended for surgery and completed all questionnaires at their specialist assessment appointment (100% completion rate). The Oxford Score was completed by 146 patients (those who had hip, knee or shoulder problems). Fourteen surgeons were involved (100% departmental involvement), and completed surgeon assessment scores for their patients. The patients’ mean age was 58 years (range 18–88, standard deviation 17). The elective procedures included all orthopaedic subspecialties except paediatric orthopaedics.

Cronbach’s coefficient alpha for the IoL was 0.926. Construct validity was tested using Pearson correlation coefficients (Table 1).

Table 1: Pearson correlation coefficients comparing all questionnaires (p-value). 

 

Surgeon Score

IoL

PCS

MCS

SF-12

Oxford

Surgeon Score

1

 

 

 

 

 

IoL

0.077
(0.177)

1

 

 

 

 

SF-12 PCS

0.067
(0.237)

-0.580 (<0.01)**

1

 

 

 

SF-12 MCS

-0.056 (0.326)

-0.551 (<0.01)**

-0.187 (0.001)**

1

 

 

Total SF-12

0.003
(0.962)

-0.735 (0.00)**

0.734 (0.00)**

0.804 (0.00)**

1

 

Oxford

-0.215 (0.011)*

-0.674 (<0.01)**

0.678 (<0.01)**

0.399 (<0.01)**

0.691 (<0.01)**

1

 *Correlation is significant at the 0.05 level (2-tailed); **Correlation is significant at the 0.01 level (2-tailed)

Significant correlations were found between IoL and SF-12, as well as IoL and Oxford scores (-0.735 and -0.674, respectively). There was no correlation between surgeon scores and IoL or SF-12 scores. All correlations between subscales of the SF-12 and IoL were statistically significant. Table 2 shows that patients who were added to the waiting list had significantly higher surgeon scores (mean difference of 19.59) as well as IoL scores (mean difference of 4.23) compared to patients who were not added (p=0.000).

Table 2: Results of t-tests showing the difference in outcome measure scores between patients that were added to the surgical waiting list compared to those who were not added.  

 

Mean Scores

 

p-value

 

Added to waiting list

Not added to waiting list

 

CPAC

78.13

52.62

0.000**

Surgeon Score

66.61

47.02

0.000**

IoL

10.41

6.18

0.000**

SF-12 PCS

33.74

35.12

0.296

SF-12 MCS

44.98

49.58

0.002**

SF-12 total

78.73

84.70

0.006**

Oxford Score

14.73

21.55

0.000**

 *Mean difference is significant at the 0.05 level; **Mean difference is significant at the 0.01 level 

Discussion

This study found that the IoL questionnaire correlated well with other validated, patient-rated measures. The Cronbach’s coefficient alpha of 0.926 suggests excellent internal consistency and exceeded the minimum value recommended for clinical application.17 Pearson correlation coefficients for all patient-rated questionnaires were good (>0.551) indicating a moderate to strong relationship between IoL and SF-12, as well as IoL and Oxford Scores.

The statistical significance of the reliability and validity measures of the IoL indicates that this is a statistically valid patient-generated assessment tool. The IoL is a measure of the disability caused by the surgical problem we plan to treat. It is more specific to surgical patients than the SF-12 which measures general quality of life and is more generalisable than the joint-specific Oxford score.

Patient-reported health measures are essential in clinical decision making. We must improve our ability to appraise patients’ comfort and performance accurately, reliably and efficiently.18 Based on their justified results, Derrett and Paul suggested that quality of life measures should be used more widely in clinical practice, indicating it would improve quality of care.19 This was further supported by findings that patient-experienced health measures were strong predictors of benefit, and that priority criteria tools appropriately selected patients with highest need and ability to benefit.6,11,20

Our results confirmed that quality of life measures correlate weakly with surgeon scores. This is in agreement with previous studies, where physicians were reported to rate patient quality of life significantly better, while scoring pain intensity significantly lower than patients scored themselves.4, 6

Conclusion

This study provides the evidence to support the reliability and validity of the IoL as a patient reported health measure for the assessment of orthopaedic elective procedures. The IoL is a relevant, comprehensive and user-friendly tool for use in prioritisation for elective surgical services. It is the intention of the Ministry of Health that this tool be used nationwide as a standardised measure for prioritisation for elective surgical services in New Zealand. We recommend its use alongside disease-specific measures for use in orthopaedic prioritisation.

Further investigation is required to assess the validity of the complete priority criteria suggested for orthopaedic surgery (combined surgeon- and patient-rated scores). The need for a universal prioritising tool is evident in order to achieve nationwide consistency in the distribution of elective surgical services. Long-term studies will be vital to provide further evidence of the validity of the proposed orthopaedic priority scoring tool. 


 Appendix 1: Impact on Life Questionnaire

c 

Appendix 2: Surgeon Priority Scoring Tool  

c 

 

Summary

The Ministry of Health has rolled out a new scoring system for prioritising patients for elective surgery in public hospitals across New Zealand. Part of this tool uses the Impact on Life Questionnaire. We compared this with widely-used scores in orthopaedic surgery and found that they correlated well. Therefore, the Impact on Life Questionnaire can be reliably used to prioritise patients for surgery in orthopaedic surgery.

Abstract

Aim

This cohort study tested the reliability and validity of the Impact on Life (IoL) patient-rated questionnaire for use in prioritising orthopaedic procedures.

Method

Three hundred and twenty-four patients completed the questionnaire during specialist orthopaedic assessments over a 5-month period in 2013. The reliability and validity of the IoL were tested against the SF-12 and Oxford scores. Correlation analysis was used to assess patient- and surgeon-rated scores. Internal consistency reliability was assessed using Cronbach’s alpha. Patient- and surgeon-rated scores were further analysed between patients added to the waiting list and those that were not.

Results

Participants’ mean age was 58 years (range 18–88). Reliability analysis showed the IoL had excellent internal consistency with a Cronbach’s alpha of 0.926, reaching the threshold for clinical application. Construct validity of the IoL was confirmed with significant correlation with other validated quality of life measures (p<0.01). T-tests indicated that patients placed on the waiting list had significantly higher surgeon and IoL scores (p<0.001), compared with those not placed on the waiting list.

Conclusion

Our results support the IoL as a valid and reliable method of assessing patient-rated quality of life and recommend its use in the Orthopaedic Clinical Priority Assessment Criteria score.

Author Information

Georgina Chan, Registrar, Department of Orthopaedic Surgery, Wellington Hospital, Wellington; Logan Walker, Registrar, Department of Orthopaedic Surgery, Wellington Hospital, Wellington; Louret Bezuidenhout, Department of Orthopaedic Surgery, Wellington Hospital, Wellington; Robert Rowan, Consultant, Department of Orthopaedic Surgery, Wellington Hospital, Wellington.

Correspondence

Robert Rowan, Wellington Hospital, Riddiford Street, Newtown, Wellington 6035.

Correspondence Email

robert.rowan@ccdhb.org.nz

References

  1. Hadorn D, Holmes A. The New Zealand priority criteria project. Part 1: Overview. BMJ. 1997;314:131-4.
  2. Gauld R, Derrett S. Solving the surgical waiting list problem? New Zealand’s ‘booking system’. Int J Health Plan M. 2000;15:259-72.
  3. McLeod D, Morgan S, McKinlay E, Dew K, et al. Use of, and attitudes to, clinical priority assessment criteria in elective surgery in New Zealand. J Health Serv Res Policy. 2004;9:91-9.
  4. Conner-Spady B, Estey A, Arnett G, Ness K, et al. Prioritization of patients on waiting lists for hip and knee replacement: Validation of a priority criteria tool. Int J Technol Assess. 2004;20:509–15.
  5. Willcox S, Seddon M, Dunn S, Edwards R, et al. Measuring and reducing waiting times: A cross-national comparison of strategies. Health Aff. 2007;26:1078–87.
  6. Derrett S, Devlin N, Hansen P, Herbison P. Prioritizing patients for elective surgery: a prospective study of clinical priority assessment criteria in New Zealand. Int J Technol Assess. 2003;19:91-105.
  7. McLeod D, Morgan S, McKinlay E, Dew K, et al. Clinicians’ reported use of clinical priority assessment criteria and their attitudes to prioritization for elective surgery: A cross-sectional survey. ANZ J Surg. 2004;74:1003-9.
  8. Doughty C, MacCormick AD, Roake J, Fraser J, et al. Prioritisation of elective surgery in New Zealand: the reliability study. NZMJ 2005; 118 (1219): Available online at: https://researchspace.auckland.ac.nz/handle/2292/4717
  9. Doughty C, Fraser J, Kirk R , MacCormick AD, et al. Reliability of clinical priority tools in orthopaedic surgery. JBJS (Br). 2005; 87, (SUPP I):28-28.
  10. Theis J. Clinical priority criteria in orthopaedics: A validation study using the SF-36 quality of life questionnaire. Health Serv Res. 2004;17:59-61.
  11. Epstein RM, Fiscella K, Lesser C, Stange K. Why the nation needs a policy push on patient-centered healthcare. Health Aff. 2010;29(8), 2010;1489-1495.
  12. Western Canada Waiting List Project. Chaos to Order: Making Sense of Waiting Lists in Canada. [PDF on Internet]. Alberta: Canada, [March 31, 2001, cited 11 March 2014]. Available from:.http://www.wcwl.ca/media/pdf/library/final_reports.2.pdf.)
  13. MacCormick A, Macmillan A, Parry B. Identification of criteria for the prioritisation of patients for elective general surgery. J Health Serv Res Policy. 2004;9:28-33.
  14. Chamberlain K, McGuigan K. The impact on life (IoL) questionnaire. Report commissioned by the Ministry of Health, 2014.
  15. Ware J, Kosinski M, Keller S. A 12-item short-form health survey: construction of scales and preliminary tests of reliability and validity. Med Care. 1996;34:220-3.
  16. Kalairajah Y, Azurza K, Hulme C, Molloy S,et al. Health outcome measures in the evaluation of total hip arthroplasties - a comparison between the Harris hip score and the Oxford hip score. J Arthroplasty. 2005;20:1037-41.
  17. Cortina J. What is coefficient alpha? An examination of theory and applications. J Appl Psychol. 1993;78:98-104.
  18. Troidl H, Kusche J, Vestweber K, Eypasch E, et al. Quality of life: An important endpoint both in surgical practice and research. J Chron Dis. 1987;40:523-8.
  19. Derrett S, Paul C, Herbison P, Williams H. Evaluation of explicit prioritisation for elective surgery: a prospective study. Health Serv Res. 2002;7:14-22.
  20. Noseworthy T, McGurran J, Hadorn D. Steering Committee of the Western Canada Waiting List Project (2003). Waiting for scheduled services in Canada: development of priority-setting scoring systems. J Eval Clin Pract. 2002;9:23-31.