View Article PDF

Since the publication of landmark studies detailing the prevalence of adverse events in hospitals,1–3 worldwide attention has focused on the need to monitor the safety and quality of inpatient care. Four main methods have emerged to monitor hospital performance across aspects of structure, processes and outcomes: regulatory inspection, third party evaluations, internal investigations and statistical indicators.4 Taking advantage of the availability of existing administrative data, indicators have become increasingly employed around the world5,6 as objective measures of hospital performance with two main purposes: summative assessments for external accountability and formative measures for internal quality improvement.7Leading this work, the United States-based Agency for Healthcare Research and Quality (AHRQ) has developed a series of quality and patient safety indicators.8 The AHRQ patient safety indicators (PSIs) were formulated to target occurrences likely to represent adverse events, such as foreign bodies left after procedures. They were developed from a rigorous process that involved extensive literature review, detailed analysis of the International Classification of Diseases (ICD-9) codes, and assessments by clinical panels and empirical analyses.9 The usefulness and validity of the indicators have now been well demonstrated in the United States.10 International interest in the indicators has seen them extended for use in other countries including an adaption by an Australian team at Victoria Healthcare (AusPSIs) based on the same iteration of the ICD-10 coding system that is shared with New Zealand.11The work undertaken for the AusPSIs involved electronic mapping of the AHRQ codes followed by detailed review by an expert coder with subsequent testing. The adaption of the AHRQ indicators to ICD10-AM coding provides an opportunity to explore their functionality in the New Zealand setting.The availability of large datasets that comprehensively include all admissions to public hospitals in a country provides an undisputed advantage in efforts to readily and reliably monitor the quality of inpatient care.12,13 Furthermore employing information coded to internationally agreed standards offers a potentially reliable basis for comparative assessment. The large resources needed to undertake internal investigations using medical record review such as were employed in the New Zealand Quality of Health Care Study1 nullify any chance they could function as an ongoing monitoring tool, although targeted options like the Global Trigger Tool offer considerable promise.14 Other alternatives such as using incident reporting data are plagued by uncertainties associated with the awareness and willingness of staff to volunteer information about their errors.15,16New Zealand is well prepared to capitalize on previous PSI work with the availability of a national inpatient dataset, the National Minimum Dataset (NMDS). The NMDS uses the ICD-10-AM coding system and data are obtained from all District Health Boards and hospitals using a common unique national patient identifier (National Health Index, NHI). However, despite the availability of a comprehensive dataset that includes all publicly funded hospitalisations, New Zealand lags behind other countries particularly the United States and the United Kingdom in its monitoring and reporting of quality measures related to hospital performance.17 Dr Foster in the United Kingdom18 and Hospital Compare in the United States,19 provide online up-to-date information about hospital performance using a range of indicators.The primary objective of this study was to explore the feasibility of applying the AHRQ/AusPSIs to New Zealand administrative data over the period 2001–9 and describe rates of positive indicator events related to medical and surgical care across New Zealand hospitals. Secondary objectives were to explore variation over time in positive indicator event rates and variation between hospitals after risk adjustment.MethodsAll data in the NMDS between and including January 2001 and December 2009 were obtained and linked by NHI to mortality records. Admissions were recorded to 91 public hospitals over the study period although not all hospitals recorded admissions in all years. The data from a small number of facilities were combined to allow for the opening and closing of institutions. Admissions to private facilities were excluded.Coding for the numerators and denominators for 16 indicators in ICD-10-AM (version 3) were developed in SAS (version 9.2) software,20 modified from information provided by AHRQ21,22 and AusPSI,23 and then checked with clinical and coding staff to verify their relevance for the New Zealand setting.PSIs were grouped into three categories: surgical (postoperative), medical, and general. Checks were undertaken of the PSI results to ensure there were no logical discrepancies, such as admissions flagged positive for a postoperative complication when a surgical procedure had not occurred. PSI rates were benchmarked against Australian and other international results and found to be generally consistent.Approval for the study was obtained from the Multi Region Ethics Committee (MEC/08/59).Statistical analyses and control charts—Descriptive statistics are presented for the population of all people discharged from New Zealand public hospitals from 2001 to 2009. Crude positive indicator rates are presented with 95% confidence intervals for the binomial estimates. In order to explore trends in the frequency of indicator positive events over time statistical process control charts are presented for the three most common indicators. Control charts present mean and upper and lower limits based on ±3 standard deviations and enable the identification of common or special causes of variation.24Event rates are presented for each 3-month period. To explore variation in the same three PSIs between hospitals event rates were first risk adjusted using methodology employed by AHRQ.21 The risk adjustment accounted for age, gender, ethnicity, rurality, deprivation and comorbidities and was undertaken with SAS20 software. Deprivation was measured according to the New Zealand Deprivation Score (NZDep01 or NZDep06) based on the domicile of the patient.25The NZDep scores were grouped into five quintiles with the most deprived areas being in quintile five. Rural or urban residence was based on the definition provided by Statistics New Zealand26 for the census area unit related to the patient's domicile. Urban areas were identified (as 0), and rural locations with high (as 1), moderate (as 2) or low (as 3) urban influence were identified. Up to 30 comorbidities were listed for each admission based on the modified Elixhauser set.27ResultsDescription of the patient safety indicators and population summary information—A description of the PSIs is included (Table 1). Over the study period there were 7,487,432 admissions to public hospitals. The people admitted to the facilities over the study period were predominantly female, European, and resided in urban locations and lower socio-economic areas (Table 2). Admission numbers rose steadily across the study period. Hypertension was the most frequent comorbidity (8.6%), while AIDS was rare (<0.1%). Table 1. Definitions of the patient safety indicators Indicator Category Cases At-risk population PSI 1 Complications of anaesthesia General / anaesthesia Episodes with anaesthesia complications All adult surgical discharges PSI 2 death in low mortality DRG Medical Patient death Adults in low-mortality DRGs, (DRGs with a total mortality rate less than 0.5% over the previous three years or less than 0.5% in any of the previous 3 years. PSI 3 Decubitus ulcer Medical Episodes with a decubitus ulcer All adult medical and surgical episodes PSI 4 Failure to rescue Medical Patient death Episodes, adults, potential complication of care (i.e. acute renal failure, DVT/PE, pneumonia, sepsis, shock or cardiac arrest, and GI haemorrhage/acute renal failure) PSI 5 Foreign body left during procedure General Episodes with foreign body left during procedure All adult episodes PSI 6 Iatrogenic pneumothorax General Episodes with pneumothorax. All adult episodes PSI 7 Selected infections due to medical care Medical Episodes with selected infection All adult episodes PSI 8 Postoperative hip fracture Post-op Episodes hip fracture All adult surgical episodes PSI 9 Postoperative haemorrhage or haematoma Post-op Episodes with postoperative haemorrhage or haematoma and postoperative control of haemorrhage or drainage of haematoma All adult surgical episodes PSI 10 Postoperative physiologic and metabolic derangement Post-op Episodes with physiologic and metabolic derangement All elective adult surgical episodes PSI 11 Postoperative respiratory failure Post-op Episodes with acute respiratory failure and a reintubation All adult elective surgical episodes PSI 12 Postoperative DVT or PE Post-op Episodes with pulmonary embolism or deep vein thrombosis All adult surgical episodes PSI 13 Postoperative sepsis Post-op Episodes of sepsis All adult elective surgical episodes PSI 14 Postoperative wound dehiscence Post-op Episodes with reclosure of postoperative disruption of abdominal wall All adult abdominopelvic surgical episodes PSI 15 Accidental puncture or laceration General Episodes with accidental puncture, laceration, cut or perforation All adult episodes PSI 16 Transfusion reaction General Episodes with a transfusion reaction All episodes with a transfusion PSI 17 Birth trauma – injury to neonate Obstetric Episodes with birth trauma All newborn PSI 18 Obstetric trauma – vaginal delivery with instrument Obstetric Episodes 3rd and 4th degree obstetric trauma. All vaginal delivery episodes PSI 19 Obstetric trauma – vaginal delivery without instrument Obstetric Episodes with 3rd and 4th degree obstetric All vaginal delivery episodes PSI 20 Obstetric trauma – caesarean delivery Obstetric Episodes with 3rd and 4th degree obstetric trauma All caesarean delivery episodes Table 2. Summary descriptive data for admissions to New Zealand hospitals 2001–9 Table 3. Number of positive cases, denominators and rates for patient safety indicators across New Zealand hospitals 2001–9 Variables Positive cases Admissions at risk Rate per 10,000 admissions (95% confidence interval) Range of rates per 10,000 admissions across NZ hospitals Medical 2 (death low mortality DRG) 2,157 2,024,136 11 (12–13) 0–20 3 (decubitus ulcer) 17,573 995,744 176 (174–179) 0–351 4 (failure to rescue) 10,240 90,920 1126.(1106–1147) 0–1622 7 (infections) 8,812 1,791,261 49 (48–50) 0–141 General 5 (foreign body) 351 4,990,050 0.7 (0.6–0.8) 0–2 6 (pneumothorax) 1,637 4,814,105 3 (3–4) 0–8 15 (puncture) 10,156 4,979,685 20 (20–21) 0–130 16 (transfusion reaction) 25 294,442 0.8 (0.6–1) 0–7 Anaesthesia 1 (complications) 25 1,663,890 0.15 (0.10–0.2) 0–1 Postoperative 8 (hip fracture) 476 1,036,097 4.5 (4–5) 0–12 9 (haemorrhage) 26,745 1,400,503 191 (189–193) 0–311 10 (physiologic derangement) 230 127,989 18 (16–20) 0–44 11 (respiratory failure) 133 92,602 14 (12–17) 0–64 12 (DVT/PE) 4,557 1,426,837 32 (31–33) 0–125 13 (sepsis) 274 21,172 129 (114–146) 0–870 14 (wound dehiscence) 584 123,814 47 (43–51) 0–101 Results for crude PSIs across study period—Some 99,366 potential safety-related events were identified from a combined total denominator of over 26.8 million admissions at risk of at least one PSI (Table 3). Some 12,397 events represented deaths, either in low mortality Diagnostic Related Groups (DRGs) (i.e. PSI2) or related to major complications such as pneumonia, thromboembolism, sepsis, renal failure, shock, cardiac arrest or gastrointestinal haemorrhage (i.e. PSI4). Some indicators were uncommon with half having less than 600 positive numerator events over the study period.By contrast, the denominators, numbers of admissions at risk of each PSI, were generally high (at least 21,172) such that event rates for 15 indicators were less than 0.2%. Positive event rates ranged between 0.15–1126 per 10,000 admissions and considerable variation was evident between hospitals. Inter-hospital variation was greatest for indicators with the highest event rates, particularly PSIs 4 and 13, such that rates ranged between 0–1622 and 0–870 per 10,000 admissions respectively.Among the categories there was least variation exhibited for the general /anaesthesia indicators (PSIs 1, 5, 6, 15, 16) which were generally infrequent and recorded a maximum average rate of 26 per 10,000 admissions over the study period. Three indicators (PSIs 4, 9, 12) were selected based on both their higher number of positive events and their higher rate of positive events per 10,000 admissions to further explore variation over time and between hospitals. Figure 1. Control charts for PSIs quarterly periods from Jan 2001 – Dec 2009 Figure 2: Risk adjusted PSI rates for New Zealand public hospitals 2001-2009Variation in positive event rates over time—Control charts describe variation in crude positive event rates over time. Special cause variation is evident for three indicators where rates occur outside of the control limits. In the case of one indicators (PSI 4) this variation is evident for more recent data. Other guidelines, such as runs of eight points on one side of the centre line, suggest that PSI12 may also be exhibiting special cause variation in later years. Such variation may prompt further investigation to consider possible explanations beginning with changes in the coding or management of the data.Variation in risk adjusted positive indicator event rates between hospitals—The three indicators also exhibit wide variation between hospitals for their risk-adjusted rates for the study period. Variation was highest for PSI4 where rates ranged between 0-907 per 1,000 discharges at 70 different hospitals. For all the PSIs most hospitals exhibited risk adjusted rates that were broadly similar however outliers were also evident. Many of these outliers related to the presence of small numerator and denominators. For example only 20 events were recorded at the hospital with PSI4 rate of 907 per 1,000 discharges, 4 events with the hospital with rate of 163 per 1,000 discharges for PSI 9 and 10 events for the hospital with a rate of 288 per 1,000 discharges for PSI12.DiscussionThe AHRQ/AusPSI indicators were designed to be used with computerised administrative data that have been already collected from hospitals. All publicly funded hospitals provide the data, and advantages for the indicators include comprehensiveness, unobtrusiveness and low cost (Table 6).28 The PSIs do not rely upon the awareness or efforts of health professionals to recognize adverse events under their care.The results from this study suggest that it is readily feasible to employ the AHRQ/AusPSIs in the New Zealand setting. However, it is noteworthy that event rates are relatively low for some of the indicators even over the combined nine years of the study (e.g., PSIs 1, 5, 6, 8 and 16) and their further utility appears limited. Among the three indicators presented in Figures 1 and 2, considerable variation is evident whether results are described across hospitals or over time. International use of the indicators has largely focused on their employment as part of a toolkit to improve patient care at a hospital,29 track progress over time30 and compare performance at the regional or national level.31Potential limitations associated with administrative data and the AHRQ PSIs have been well described (Table 4).32 The concerns need to be tempered by the realisation that indicators act as alerts or flags rather than definitive assessments of the quality of care. The usefulness of the indicators stems from their function as a screening tool and further efforts are usually required to interrogate the findings to ascertain whether patient safety problems actually exist when indicators suggest that patient care has been compromised.The principal concern with the use of PSIs centres on whether administrative data collected for another purpose can be reliably applied to patient safety measurement. A concern is that the range of patient safety issues identified by the PSIs is limited; for example no information is generated about the presence of adverse drug events by the indicators, yet these events make a large contribution to iatrogenic harm.1 Furthermore any identification of injury is further limited by the completeness and accuracy of the ICD-10 coding system. Although principal diagnoses are usually well recorded in most administrative databases doubts remain about the quality of recording for complications and comorbidities.33Some concerns related to these issues are being addressed. Firstly, new indicators have emerged to describe medication related events using Australian administrative data.34 In addition, the timing of any potential adverse event can now be better defined with New Zealand's adoption of the ‘present on admission' flag which ensures that clinical coders record whether a condition was present at admission or alternatively may have occurred as a complication during the inpatient stay.35 Finally, detailed work to validate the individual indicators by means of medical record review is now nearly complete and is producing some promising results.36, 37 Some of the indicators appear to be associated with high positive predictive values underscoring their role as useful screening tools. Work has been undertaken to validate the indicators in New Zealand using similar record review methodology. Table 4. Main advantages of using administrative data and indicators to measure patient safety Preliminary results suggest that the same PSIs are consistently associated with similarly high positive predictive values in the local setting. These findings are reassuring as, unlike some other countries, New Zealand administrative data only includes secondary diagnoses that were materially relevant to patient care during an inpatient stay.38 As the PSIs rely heavily on secondary diagnosis codes it was possible that some indicator events may either not have been identified or some exclusions could not be applied, so indicators may have been less accurate locally. The final results from the validation exercise will be important to further determine the applicability of the indicators in New Zealand.Limitations aside, the application of the PSIs to New Zealand data raises interesting questions about the local epidemiology of adverse events and the impact of quality improvement initiatives. For example, further work is needed to explore whether the decline in rates of PSI 4 (failure to rescue) can reliably be attributed to the increasing adoption of early warning scoring systems and outreach teams in hospitals.39 Further impetus for the use of PSIs to explore the epidemiology of adverse events comes from the suggestion that the indicators may detect different events from those identified by other methods.40The PSIs could be readily adopted by key organisations in New Zealand, such as the Health Quality and Safety Commission (HQSC) and the Health Roundtable, that are already involved with DHBs in the measurement, analysis and reporting of hospital performance information in New Zealand. The PSIs could augment their existing measures and contribute to their provision of customized dashboards that enable each DHB to assess their progress over time, and with suitably adjusted measures, help them compare themselves against others. Ideally these dashboards should monitor data with as close to real time information as possible. Employing the Variable Life Adjusted Display methodology that has been successfully deployed in Australia,41 they could be used to generate case-by case warnings about any significant differences between observed and expected outcomes in conjunction with a guided response that begins with the judicious scrutiny for any changes in coding and escalates when appropriate to the investigation of service delivery.Whilst their most important function would be to foster internal quality improvement by hospitals, on confirmation of their validity some of them could later be considered for an accountability function and incorporated into the Integrated Performance and Incentive Framework42 with the aim of supporting clinical governance and fostering continuous quality improvement for hospitals alongside primary care. Creating incentives for continuous improvement avoids some of the potential constraints associated with health targets that are limited to determining whether a finite level has been achieved rather than charting progress over time and encouraging, ongoing, iterative cycles of improvement.Considerable interest exists in the public reporting of performance information. In New Zealand, hospital performance data are mainly used internally and between DHBs. This contrasts with the United states where The Department of Health and Human Services in the United States now provides patient safety indicator results for named hospitals on the internet for public scrutiny.43 Proponents argue that such reporting generates opportunities for public accountability, scrutiny and competitive improvement between hospitals.Detractors have counseled about the pitfalls inherent in comparative public reporting, citing limitations of using inadequately validated indicators, insufficient adjustments for case mix, spurious findings occurring when hospitals are ranked even though considerable random variation may be present and the potential for gaming that can occur when attention is focused solely on specific measures.44 Limited evidence exists for the premise that public reporting of indicator data sponsors quality improvement45 and until confirmation is provided from rigorous evaluation in the local setting most attention should be given to promoting the use of PSIs by hospitals and DHBs for internal improvement processes possibly as part of a broader scorecard with metrics related to access, effectiveness and efficiency.46In conclusion, the PSIs can be readily applied to New Zealand's administrative data. Used judiciously, in conjunction with other established and emerging forms of quality assessment and monitoring, such as clinical audit, trigger tools and incident reporting, the PSIs offer another valuable tool for hospitals to assist them with their efforts to enhance patient safety.The advantages of New Zealand's rich administrative dataset combined with its small size suggest that promoting the use of PSIs to enhance patient safety should be readily achievable. Furthermore, New Zealand is well placed to take advantage of the work of existing organisations such as the HQSC and new developments in real time monitoring to sponsor their introduction.47With confirmation of validity, the PSIs could be extended to an accountability framework that encourages continuous improvement and perhaps comprises public reporting as long as it includes appropriate evaluation to ensure that undesired outcomes are not fostered.

Summary

Abstract

Introduction Increasing interest has focused on the safety of hospital care. The AusPSIs are a set of indicators developed from Australian administrative data to reliably identify inpatient adverse events in hospitals. The main aim of this study was to explore the application of the AHRQ/AusPSIs to New Zealand administrative hospital data related to medical and surgical care. Variation over time and across hospitals were also considered for a subset of the more common indicators.-Method AHRQ/AusPSIs were adapted for use with New Zealand National Minimum Dataset administrative data for the period 2001-9. Crude positive event rates for each of the 16 indicators were assessed across New Zealand public hospitals. Variation over time for six more common indicators is presented using statistical control charts. Variation between hospitals was explored using rates adjusted for differences in patient variables including age, sex, ethnicity, rurality of residence, NZDep score and comorbidities.-Results The AHRQ/AusPSIs were applied to New Zealand administrative hospital data and some 99,366 admissions were associated with a positive indicator event. However rates for some indicators were low (

Aim

Method

Results

Conclusion

Author Information

Phil Hider1; Karl Parker2; Martin von Randow3; Barry Milne3; Roy Lay-Yee3; Peter Davis3 1. Department of Population Health, University of Otago, Christchurch2. SHORE and Whariki Research Centre, School of Public Health, Massey University 3. Compass, Centre of Methods and Policy Application in the Social Sciences, University of Auckland

Acknowledgements

Funding was received from the Health Research Council of New Zealand. Assistance from Drs Patrick Graham and Aidin Jalilzadeh with their statistical input is also gratefully acknowledged.

Correspondence

Dr Phil Hider, Department of Population Health, University of Otago, Christchurch, PO Box 4345, Christchurch 8140, New Zealand.

Correspondence Email

phil.hider@otago.ac.nz

Competing Interests

Nil.

1. Davis P, Lay-Yee R, Briant R, Schug S, Scott A. Adverse events in New Zealand public hospitals. Wellington: Ministry of Health; 2001.2. Wilson RM, Harrison BT, Gibberd RW, Hamilton JD. An analysis of the causes of adverse events from the Quality in Australian Health Care Study. Med J Aust. 1999 May 3;170(9):411-5. PubMed PMID: 10341771. English.3. Brennan TA, Leape LL, Laird NM, Hebert L, Localio AR, Lawthers AG, et al. Incidence of adverse events and negligence in hospitalized patients: results of the Harvard Medical Practice Study I. 1991. Qual Saf Health Care. 2004 Apr;13(2):145-51; discussion 51-2. PubMed PMID: 15069223. Pubmed Central PMCID: Source: NLM. PMC1743811. English.4. Shaw C. How can hospital performance be measured and monitored? Copenhagen: WHO, Regional Office for Europe 2003. Available from: http://www.euro.who.int/__data/assets/pdf_file/0009/74718/E82975.pdf.5. Mattke S, Epstein AM, Leatherman S. The OECD Health Care Quality Indicators Project: history and background. Int J Qual Health Care. 2006 Sep;18 Suppl 1:1-4. PubMed PMID: 16954509. English.6. Drosler SE, Klazinga NS, Romano PS, Tancredi DJ, Gogorcena Aoiz MA, Hewitt MC, et al. Application of patient safety indicators internationally: a pilot study among seven countries. Int J Qual Health Care. 2009 Aug;21(4):272-8. PubMed PMID: 19395469. English.7. Freeman T. using performance indicators to improve health care qualtiy in the public sector: a review of the literature. . Health Serv Manage Res. 2002 May;15(2):126-37.8. Romano PS, Geppert JJ, Davies S, Miller MR, Elixhauser A, McDonald KM. A national profile of patient safety in U.S. hospitals. Health Aff (Millwood). 2003 Mar-Apr;22(2):154-66. PubMed PMID: 12674418. English.9. McDonald K, Romano P, Geppert J, Davies S, Duncan B, Shojania K. Measures of Patient Safety Based on Hospital Administrative Data. The Patient Safety Indicators. . Rockville, MD: Agency for Healthcare Research and Quality; 2002.10. Rosen AK, Rivard P, Zhao S, Loveland S, Tsilimingras D, Christiansen CL, et al. Evaluating the patient safety indicators: how well do they perform on Veterans Health Administration data? Med Care. 2005 Sep;43(9):873-84. PubMed PMID: 16116352. English.11. McConchie S, Shepheard J, Waters S, McMillan AJ, Sundararajan V. The AusPSIs: the Australian version of the Agency of Healthcare Research and Quality patient safety indicators. Aust Health Rev. 2009 May;33(2):334-41. PubMed PMID: 19563325. English.12. Kaafarani HMA, Rosen AK. Using administrative data to identify surgical adverse events: an introduction to the Patient Safety Indicators. Am J Surg. 2009 Nov;198(5 Suppl):S63-8. PubMed PMID: 19874937. English.13. Kaafarani HMA, Borzecki AM, Itani KMF, Loveland S, Mull HJ, Hickson K, et al. Validity of selected Patient Safety Indicators: opportunities and concerns. J Am Coll Surg. 2011 Jun;212(6):924-34. PubMed PMID: 20869268. English.14. Classen D, Resar R, Griffin F, Frederico F, Frankel T, Himmel N. 'Global Trigger Tool' shows that adverse events in hospitals may be ten times greater than previously measured. . Health Aff (Millwood). 2011;30(4):581-9.15. Noble D, Pronovost P. Underreporting of patient safety incidents reduces health care's ability to quantify and accurately measure harm reduction. . J Patient Saf. 2010;6(4):247-50.16. Noble DJ, Pronovost PJ. Underreporting of patient safety incidents reduces health care's ability to quantify and accurately measure harm reduction. J Patient Saf. 2010 Dec;6(4):247-50. PubMed PMID: 21500613. English.17. Chaudry M, Gauld R, Horsburgh S. Hospital quality of care perofrmance measurement and reporting: what new Zealand can learn from the United States and United Kingdom. The New Zealand Medical Journal. 2012;125(1366).18. Intelligence DF. My hospital guide. 2014.19. Medicare. Hospital Compare 2014 [July 2014]. Available from: http://www.medicare.gov/hospitalcompare/search.html 20. SAS Institute Inc. Version 9.2 Cary, North Carolina, United States.2011.21. Anonymous. Patient safety indicators technical specifications version 4.3.: Agency for health Research and Quality 2011. Available from: http://www.qualityindicators.ahrq.gov/modules/PSI_TechSpec.aspx 22. Anonymous. Patient safety indicators technical specifications version 4.3 June 2011: Agency for Health Research and Quality; 2011 [cited 2011 16 November]. Available from: http://www.qualityindicators.ahrq.gov/modules/PSI_TechSpec.aspx 23. Anonymous. Patient safety indicators technical documents Melbourne: Department of Health, State Government of Victoria, Australia; 2009 [cited 2011 June]. Available from: http://www.health.vic.gov.au/psi/auspsi/technical-documents 24. Mohammed M, Worthington P, Woodhall W. Plotting basic control charts: tutorial notes for healthcare practitioners. Qual Saf Health Care. 2008;17(2):137-45.25. Salmond C, Crampton P, Atkinson J. NZDep Index of Deprivation users manual. Wellington: University of Otago, Wellington; 2007.26. Monk TG, Saini V, Weldon BC, Sigl JC. Anesthetic management and one-year mortality after noncardiac surgery. Anesthesia & Analgesia.100(1):4-10. PubMed PMID: 15616043.27. Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi J-C, et al. Coding algorithms for defining comorbdities in ICD-9-CM and ICD-10 administrative data. . Med Care. 2005;43(11):1130-39.28. Scott IA, Poole PJ, Jayathissa S. Improving quality and safety of hospital care: a reappraisal and an agenda for clinically relevant reform. Intern Med J. 2008 Jan;38(1):44-55. PubMed PMID: 18190414.29. Hussey PS, Burns RM, Weinick RM, Mayer L, Cerese J, Farley DO. Using a hospital quality improvement toolkit to improve performance on the AHRQ quality indicators. Jt Comm J Qual Patient Saf. 2013 Apr;39(4):177-84. PubMed PMID: 23641537. English.30. Rosen AK, Zhao S, Rivard P, Loveland S, Montez-Rath ME, Elixhauser A, et al. Tracking rates of Patient Safety Indicators over time: lessons from the Veterans Administration. Med Care. 2006 Sep;44(9):850-61. PubMed PMID: 16932137. English.31. Raleigh VS, Cooper J, Bremner SA, Scobie S. Patient safety indicators for England from hospital administrative data: case-control analysis and comparison with US data. Bmj. 2008;337:a1702. PubMed PMID: 18930971. Pubmed Central PMCID: Source: NLM. PMC2569150. English.32. Zhan C, Miller MR. Administrative data based patient safety research: a critical review. Quality & Safety in Health Care. 2003 Dec;12 Suppl 2:ii58-63. PubMed PMID: 14645897. Pubmed Central PMCID: Source: NLM. PMC1765777. English.33. Quan H. Adaptation of AHRQ patient safety indicators for use in ICD-10 administrative data by an international consortium. 2008 [August 2010]. Available from: http://www.ahrq.gov/downloads/pub/advances2/vol1/advances-quan_52.pdf 34. Parikh S, Christensen D, STuchberry P, Petersen J, Hutchinson A, Jackson T. Exploring in-hospital adverse drug events using ICD-10 codes. Aust Health Rev. 2014;38:454-60.35. Kim H, Capezuti E, Kovner C, Zhao Z, Boockvar K. Prevalence and predictors of adverse events in older surgical patients: impact of the present on admission indicator. Gerontologist. 2010 Dec;50(6):810-20. PubMed PMID: 20566833. English.36. Henderson KE, Recktenwald Aj, Reichley RM, Bailey TC, Waterman BM, Diekemper RL, et al. Clinical validation of the AHRQ postoperative venous thromboembolism patient safety indicator. Jt Comm J Qual Patient Saf. 2009 Jul;35(7):370-6. PubMed PMID: 19634805. English.37. Utter GH, Zrelak PA, Baron R, Tancredi DJ, Sadeghi B, Geppert JJ, et al. Positive predictive value of the AHRQ accidental puncture or laceration patient safety indicator. Ann Surg. 2009 Dec;250(6):1041-5. PubMed PMID: 19779328.38. Anonymous. Episodes of care - additional diagnoses (ICD-10-AM) Canberra: Australian Institute of Health and Welfare; 2011 [cited 2011 October ]. Available from: http://meteor.aihw.gov.au/content/index.phtml/itemId/391322 39. Cuthbertson BH, Boroujerdi M, Prescott G. The use of combined physiological parameters in the early recognition of the deteriorating acute medical patient.[Erratum appears in J R Coll Physicians Edinb. 2010 Jun;40(2):190]. Journal of the Royal College of Physicians of Edinburgh. 2010 Mar;40(1):19-25. PubMed PMID: 21125034.40. Naessens JM, Campbell CR, Huddleston JM, Berg BP, Lefante JJ, Williams AR, et al. A comparison of hospital adverse events identified by three widely used detection methods. Int J Qual Health Care. 2009;21(4):301-7. PubMed PMID: 2009396775. English.41. Duckett S, Coory M, Sketcher-Baker K. Identifying variations in quality of care in Queensland hospitals. Med J Aust. 2007;187(10):571-5.42. Health Mo. Ministry of Health; 2014 [July 2014]. Available from: http://www.health.govt.nz/our-work/primary-health-care/primary-health-care-subsidies-and-services/pho-performance-programme-and-transition-integrated-performance-and-incentive-framework 43. Anonymous. Hospital Compare: US Department of health and Human Services; 2011 [cited 2011 October]. Available from: http://www.hospitalcompare.hhs.gov/ 44. Scott IA, Ward M. Public reporting of hospital outcomes based on administrative data: risks and opportunities. Med J Aust. 2006 Jun 5;184(11):571-5. PubMed PMID: 16768665.45. Ketelaar N, Faber M, Flottorp S, Rygh L, Deane K, Eccles M, et al. Public release of performance data in changing the behaviour of healthcare consumers, professionals or organisations. Cochrane Database of Systematic Reviews 2011;11(CD004538).46. Davis P, Milne B, Parker K, Hider P, Lay-Yee R, Cumming J, et al. Efficiency, effectiveness, equity (E3). Evaluating hospital performance in three dimensions. Health Policy. 2013 Sep;112(1-2):19-27. PubMed PMID: 23537468. English.47. Jones M, Steiner S. Assessing the effect of estimation error on risk-adjusted CUSUM chart performance. Int J Qual Health Care. 2011;24:176-81.

Contact diana@nzma.org.nz
for the PDF of this article

View Article PDF

Since the publication of landmark studies detailing the prevalence of adverse events in hospitals,1–3 worldwide attention has focused on the need to monitor the safety and quality of inpatient care. Four main methods have emerged to monitor hospital performance across aspects of structure, processes and outcomes: regulatory inspection, third party evaluations, internal investigations and statistical indicators.4 Taking advantage of the availability of existing administrative data, indicators have become increasingly employed around the world5,6 as objective measures of hospital performance with two main purposes: summative assessments for external accountability and formative measures for internal quality improvement.7Leading this work, the United States-based Agency for Healthcare Research and Quality (AHRQ) has developed a series of quality and patient safety indicators.8 The AHRQ patient safety indicators (PSIs) were formulated to target occurrences likely to represent adverse events, such as foreign bodies left after procedures. They were developed from a rigorous process that involved extensive literature review, detailed analysis of the International Classification of Diseases (ICD-9) codes, and assessments by clinical panels and empirical analyses.9 The usefulness and validity of the indicators have now been well demonstrated in the United States.10 International interest in the indicators has seen them extended for use in other countries including an adaption by an Australian team at Victoria Healthcare (AusPSIs) based on the same iteration of the ICD-10 coding system that is shared with New Zealand.11The work undertaken for the AusPSIs involved electronic mapping of the AHRQ codes followed by detailed review by an expert coder with subsequent testing. The adaption of the AHRQ indicators to ICD10-AM coding provides an opportunity to explore their functionality in the New Zealand setting.The availability of large datasets that comprehensively include all admissions to public hospitals in a country provides an undisputed advantage in efforts to readily and reliably monitor the quality of inpatient care.12,13 Furthermore employing information coded to internationally agreed standards offers a potentially reliable basis for comparative assessment. The large resources needed to undertake internal investigations using medical record review such as were employed in the New Zealand Quality of Health Care Study1 nullify any chance they could function as an ongoing monitoring tool, although targeted options like the Global Trigger Tool offer considerable promise.14 Other alternatives such as using incident reporting data are plagued by uncertainties associated with the awareness and willingness of staff to volunteer information about their errors.15,16New Zealand is well prepared to capitalize on previous PSI work with the availability of a national inpatient dataset, the National Minimum Dataset (NMDS). The NMDS uses the ICD-10-AM coding system and data are obtained from all District Health Boards and hospitals using a common unique national patient identifier (National Health Index, NHI). However, despite the availability of a comprehensive dataset that includes all publicly funded hospitalisations, New Zealand lags behind other countries particularly the United States and the United Kingdom in its monitoring and reporting of quality measures related to hospital performance.17 Dr Foster in the United Kingdom18 and Hospital Compare in the United States,19 provide online up-to-date information about hospital performance using a range of indicators.The primary objective of this study was to explore the feasibility of applying the AHRQ/AusPSIs to New Zealand administrative data over the period 2001–9 and describe rates of positive indicator events related to medical and surgical care across New Zealand hospitals. Secondary objectives were to explore variation over time in positive indicator event rates and variation between hospitals after risk adjustment.MethodsAll data in the NMDS between and including January 2001 and December 2009 were obtained and linked by NHI to mortality records. Admissions were recorded to 91 public hospitals over the study period although not all hospitals recorded admissions in all years. The data from a small number of facilities were combined to allow for the opening and closing of institutions. Admissions to private facilities were excluded.Coding for the numerators and denominators for 16 indicators in ICD-10-AM (version 3) were developed in SAS (version 9.2) software,20 modified from information provided by AHRQ21,22 and AusPSI,23 and then checked with clinical and coding staff to verify their relevance for the New Zealand setting.PSIs were grouped into three categories: surgical (postoperative), medical, and general. Checks were undertaken of the PSI results to ensure there were no logical discrepancies, such as admissions flagged positive for a postoperative complication when a surgical procedure had not occurred. PSI rates were benchmarked against Australian and other international results and found to be generally consistent.Approval for the study was obtained from the Multi Region Ethics Committee (MEC/08/59).Statistical analyses and control charts—Descriptive statistics are presented for the population of all people discharged from New Zealand public hospitals from 2001 to 2009. Crude positive indicator rates are presented with 95% confidence intervals for the binomial estimates. In order to explore trends in the frequency of indicator positive events over time statistical process control charts are presented for the three most common indicators. Control charts present mean and upper and lower limits based on ±3 standard deviations and enable the identification of common or special causes of variation.24Event rates are presented for each 3-month period. To explore variation in the same three PSIs between hospitals event rates were first risk adjusted using methodology employed by AHRQ.21 The risk adjustment accounted for age, gender, ethnicity, rurality, deprivation and comorbidities and was undertaken with SAS20 software. Deprivation was measured according to the New Zealand Deprivation Score (NZDep01 or NZDep06) based on the domicile of the patient.25The NZDep scores were grouped into five quintiles with the most deprived areas being in quintile five. Rural or urban residence was based on the definition provided by Statistics New Zealand26 for the census area unit related to the patient's domicile. Urban areas were identified (as 0), and rural locations with high (as 1), moderate (as 2) or low (as 3) urban influence were identified. Up to 30 comorbidities were listed for each admission based on the modified Elixhauser set.27ResultsDescription of the patient safety indicators and population summary information—A description of the PSIs is included (Table 1). Over the study period there were 7,487,432 admissions to public hospitals. The people admitted to the facilities over the study period were predominantly female, European, and resided in urban locations and lower socio-economic areas (Table 2). Admission numbers rose steadily across the study period. Hypertension was the most frequent comorbidity (8.6%), while AIDS was rare (<0.1%). Table 1. Definitions of the patient safety indicators Indicator Category Cases At-risk population PSI 1 Complications of anaesthesia General / anaesthesia Episodes with anaesthesia complications All adult surgical discharges PSI 2 death in low mortality DRG Medical Patient death Adults in low-mortality DRGs, (DRGs with a total mortality rate less than 0.5% over the previous three years or less than 0.5% in any of the previous 3 years. PSI 3 Decubitus ulcer Medical Episodes with a decubitus ulcer All adult medical and surgical episodes PSI 4 Failure to rescue Medical Patient death Episodes, adults, potential complication of care (i.e. acute renal failure, DVT/PE, pneumonia, sepsis, shock or cardiac arrest, and GI haemorrhage/acute renal failure) PSI 5 Foreign body left during procedure General Episodes with foreign body left during procedure All adult episodes PSI 6 Iatrogenic pneumothorax General Episodes with pneumothorax. All adult episodes PSI 7 Selected infections due to medical care Medical Episodes with selected infection All adult episodes PSI 8 Postoperative hip fracture Post-op Episodes hip fracture All adult surgical episodes PSI 9 Postoperative haemorrhage or haematoma Post-op Episodes with postoperative haemorrhage or haematoma and postoperative control of haemorrhage or drainage of haematoma All adult surgical episodes PSI 10 Postoperative physiologic and metabolic derangement Post-op Episodes with physiologic and metabolic derangement All elective adult surgical episodes PSI 11 Postoperative respiratory failure Post-op Episodes with acute respiratory failure and a reintubation All adult elective surgical episodes PSI 12 Postoperative DVT or PE Post-op Episodes with pulmonary embolism or deep vein thrombosis All adult surgical episodes PSI 13 Postoperative sepsis Post-op Episodes of sepsis All adult elective surgical episodes PSI 14 Postoperative wound dehiscence Post-op Episodes with reclosure of postoperative disruption of abdominal wall All adult abdominopelvic surgical episodes PSI 15 Accidental puncture or laceration General Episodes with accidental puncture, laceration, cut or perforation All adult episodes PSI 16 Transfusion reaction General Episodes with a transfusion reaction All episodes with a transfusion PSI 17 Birth trauma – injury to neonate Obstetric Episodes with birth trauma All newborn PSI 18 Obstetric trauma – vaginal delivery with instrument Obstetric Episodes 3rd and 4th degree obstetric trauma. All vaginal delivery episodes PSI 19 Obstetric trauma – vaginal delivery without instrument Obstetric Episodes with 3rd and 4th degree obstetric All vaginal delivery episodes PSI 20 Obstetric trauma – caesarean delivery Obstetric Episodes with 3rd and 4th degree obstetric trauma All caesarean delivery episodes Table 2. Summary descriptive data for admissions to New Zealand hospitals 2001–9 Table 3. Number of positive cases, denominators and rates for patient safety indicators across New Zealand hospitals 2001–9 Variables Positive cases Admissions at risk Rate per 10,000 admissions (95% confidence interval) Range of rates per 10,000 admissions across NZ hospitals Medical 2 (death low mortality DRG) 2,157 2,024,136 11 (12–13) 0–20 3 (decubitus ulcer) 17,573 995,744 176 (174–179) 0–351 4 (failure to rescue) 10,240 90,920 1126.(1106–1147) 0–1622 7 (infections) 8,812 1,791,261 49 (48–50) 0–141 General 5 (foreign body) 351 4,990,050 0.7 (0.6–0.8) 0–2 6 (pneumothorax) 1,637 4,814,105 3 (3–4) 0–8 15 (puncture) 10,156 4,979,685 20 (20–21) 0–130 16 (transfusion reaction) 25 294,442 0.8 (0.6–1) 0–7 Anaesthesia 1 (complications) 25 1,663,890 0.15 (0.10–0.2) 0–1 Postoperative 8 (hip fracture) 476 1,036,097 4.5 (4–5) 0–12 9 (haemorrhage) 26,745 1,400,503 191 (189–193) 0–311 10 (physiologic derangement) 230 127,989 18 (16–20) 0–44 11 (respiratory failure) 133 92,602 14 (12–17) 0–64 12 (DVT/PE) 4,557 1,426,837 32 (31–33) 0–125 13 (sepsis) 274 21,172 129 (114–146) 0–870 14 (wound dehiscence) 584 123,814 47 (43–51) 0–101 Results for crude PSIs across study period—Some 99,366 potential safety-related events were identified from a combined total denominator of over 26.8 million admissions at risk of at least one PSI (Table 3). Some 12,397 events represented deaths, either in low mortality Diagnostic Related Groups (DRGs) (i.e. PSI2) or related to major complications such as pneumonia, thromboembolism, sepsis, renal failure, shock, cardiac arrest or gastrointestinal haemorrhage (i.e. PSI4). Some indicators were uncommon with half having less than 600 positive numerator events over the study period.By contrast, the denominators, numbers of admissions at risk of each PSI, were generally high (at least 21,172) such that event rates for 15 indicators were less than 0.2%. Positive event rates ranged between 0.15–1126 per 10,000 admissions and considerable variation was evident between hospitals. Inter-hospital variation was greatest for indicators with the highest event rates, particularly PSIs 4 and 13, such that rates ranged between 0–1622 and 0–870 per 10,000 admissions respectively.Among the categories there was least variation exhibited for the general /anaesthesia indicators (PSIs 1, 5, 6, 15, 16) which were generally infrequent and recorded a maximum average rate of 26 per 10,000 admissions over the study period. Three indicators (PSIs 4, 9, 12) were selected based on both their higher number of positive events and their higher rate of positive events per 10,000 admissions to further explore variation over time and between hospitals. Figure 1. Control charts for PSIs quarterly periods from Jan 2001 – Dec 2009 Figure 2: Risk adjusted PSI rates for New Zealand public hospitals 2001-2009Variation in positive event rates over time—Control charts describe variation in crude positive event rates over time. Special cause variation is evident for three indicators where rates occur outside of the control limits. In the case of one indicators (PSI 4) this variation is evident for more recent data. Other guidelines, such as runs of eight points on one side of the centre line, suggest that PSI12 may also be exhibiting special cause variation in later years. Such variation may prompt further investigation to consider possible explanations beginning with changes in the coding or management of the data.Variation in risk adjusted positive indicator event rates between hospitals—The three indicators also exhibit wide variation between hospitals for their risk-adjusted rates for the study period. Variation was highest for PSI4 where rates ranged between 0-907 per 1,000 discharges at 70 different hospitals. For all the PSIs most hospitals exhibited risk adjusted rates that were broadly similar however outliers were also evident. Many of these outliers related to the presence of small numerator and denominators. For example only 20 events were recorded at the hospital with PSI4 rate of 907 per 1,000 discharges, 4 events with the hospital with rate of 163 per 1,000 discharges for PSI 9 and 10 events for the hospital with a rate of 288 per 1,000 discharges for PSI12.DiscussionThe AHRQ/AusPSI indicators were designed to be used with computerised administrative data that have been already collected from hospitals. All publicly funded hospitals provide the data, and advantages for the indicators include comprehensiveness, unobtrusiveness and low cost (Table 6).28 The PSIs do not rely upon the awareness or efforts of health professionals to recognize adverse events under their care.The results from this study suggest that it is readily feasible to employ the AHRQ/AusPSIs in the New Zealand setting. However, it is noteworthy that event rates are relatively low for some of the indicators even over the combined nine years of the study (e.g., PSIs 1, 5, 6, 8 and 16) and their further utility appears limited. Among the three indicators presented in Figures 1 and 2, considerable variation is evident whether results are described across hospitals or over time. International use of the indicators has largely focused on their employment as part of a toolkit to improve patient care at a hospital,29 track progress over time30 and compare performance at the regional or national level.31Potential limitations associated with administrative data and the AHRQ PSIs have been well described (Table 4).32 The concerns need to be tempered by the realisation that indicators act as alerts or flags rather than definitive assessments of the quality of care. The usefulness of the indicators stems from their function as a screening tool and further efforts are usually required to interrogate the findings to ascertain whether patient safety problems actually exist when indicators suggest that patient care has been compromised.The principal concern with the use of PSIs centres on whether administrative data collected for another purpose can be reliably applied to patient safety measurement. A concern is that the range of patient safety issues identified by the PSIs is limited; for example no information is generated about the presence of adverse drug events by the indicators, yet these events make a large contribution to iatrogenic harm.1 Furthermore any identification of injury is further limited by the completeness and accuracy of the ICD-10 coding system. Although principal diagnoses are usually well recorded in most administrative databases doubts remain about the quality of recording for complications and comorbidities.33Some concerns related to these issues are being addressed. Firstly, new indicators have emerged to describe medication related events using Australian administrative data.34 In addition, the timing of any potential adverse event can now be better defined with New Zealand's adoption of the ‘present on admission' flag which ensures that clinical coders record whether a condition was present at admission or alternatively may have occurred as a complication during the inpatient stay.35 Finally, detailed work to validate the individual indicators by means of medical record review is now nearly complete and is producing some promising results.36, 37 Some of the indicators appear to be associated with high positive predictive values underscoring their role as useful screening tools. Work has been undertaken to validate the indicators in New Zealand using similar record review methodology. Table 4. Main advantages of using administrative data and indicators to measure patient safety Preliminary results suggest that the same PSIs are consistently associated with similarly high positive predictive values in the local setting. These findings are reassuring as, unlike some other countries, New Zealand administrative data only includes secondary diagnoses that were materially relevant to patient care during an inpatient stay.38 As the PSIs rely heavily on secondary diagnosis codes it was possible that some indicator events may either not have been identified or some exclusions could not be applied, so indicators may have been less accurate locally. The final results from the validation exercise will be important to further determine the applicability of the indicators in New Zealand.Limitations aside, the application of the PSIs to New Zealand data raises interesting questions about the local epidemiology of adverse events and the impact of quality improvement initiatives. For example, further work is needed to explore whether the decline in rates of PSI 4 (failure to rescue) can reliably be attributed to the increasing adoption of early warning scoring systems and outreach teams in hospitals.39 Further impetus for the use of PSIs to explore the epidemiology of adverse events comes from the suggestion that the indicators may detect different events from those identified by other methods.40The PSIs could be readily adopted by key organisations in New Zealand, such as the Health Quality and Safety Commission (HQSC) and the Health Roundtable, that are already involved with DHBs in the measurement, analysis and reporting of hospital performance information in New Zealand. The PSIs could augment their existing measures and contribute to their provision of customized dashboards that enable each DHB to assess their progress over time, and with suitably adjusted measures, help them compare themselves against others. Ideally these dashboards should monitor data with as close to real time information as possible. Employing the Variable Life Adjusted Display methodology that has been successfully deployed in Australia,41 they could be used to generate case-by case warnings about any significant differences between observed and expected outcomes in conjunction with a guided response that begins with the judicious scrutiny for any changes in coding and escalates when appropriate to the investigation of service delivery.Whilst their most important function would be to foster internal quality improvement by hospitals, on confirmation of their validity some of them could later be considered for an accountability function and incorporated into the Integrated Performance and Incentive Framework42 with the aim of supporting clinical governance and fostering continuous quality improvement for hospitals alongside primary care. Creating incentives for continuous improvement avoids some of the potential constraints associated with health targets that are limited to determining whether a finite level has been achieved rather than charting progress over time and encouraging, ongoing, iterative cycles of improvement.Considerable interest exists in the public reporting of performance information. In New Zealand, hospital performance data are mainly used internally and between DHBs. This contrasts with the United states where The Department of Health and Human Services in the United States now provides patient safety indicator results for named hospitals on the internet for public scrutiny.43 Proponents argue that such reporting generates opportunities for public accountability, scrutiny and competitive improvement between hospitals.Detractors have counseled about the pitfalls inherent in comparative public reporting, citing limitations of using inadequately validated indicators, insufficient adjustments for case mix, spurious findings occurring when hospitals are ranked even though considerable random variation may be present and the potential for gaming that can occur when attention is focused solely on specific measures.44 Limited evidence exists for the premise that public reporting of indicator data sponsors quality improvement45 and until confirmation is provided from rigorous evaluation in the local setting most attention should be given to promoting the use of PSIs by hospitals and DHBs for internal improvement processes possibly as part of a broader scorecard with metrics related to access, effectiveness and efficiency.46In conclusion, the PSIs can be readily applied to New Zealand's administrative data. Used judiciously, in conjunction with other established and emerging forms of quality assessment and monitoring, such as clinical audit, trigger tools and incident reporting, the PSIs offer another valuable tool for hospitals to assist them with their efforts to enhance patient safety.The advantages of New Zealand's rich administrative dataset combined with its small size suggest that promoting the use of PSIs to enhance patient safety should be readily achievable. Furthermore, New Zealand is well placed to take advantage of the work of existing organisations such as the HQSC and new developments in real time monitoring to sponsor their introduction.47With confirmation of validity, the PSIs could be extended to an accountability framework that encourages continuous improvement and perhaps comprises public reporting as long as it includes appropriate evaluation to ensure that undesired outcomes are not fostered.

Summary

Abstract

Introduction Increasing interest has focused on the safety of hospital care. The AusPSIs are a set of indicators developed from Australian administrative data to reliably identify inpatient adverse events in hospitals. The main aim of this study was to explore the application of the AHRQ/AusPSIs to New Zealand administrative hospital data related to medical and surgical care. Variation over time and across hospitals were also considered for a subset of the more common indicators.-Method AHRQ/AusPSIs were adapted for use with New Zealand National Minimum Dataset administrative data for the period 2001-9. Crude positive event rates for each of the 16 indicators were assessed across New Zealand public hospitals. Variation over time for six more common indicators is presented using statistical control charts. Variation between hospitals was explored using rates adjusted for differences in patient variables including age, sex, ethnicity, rurality of residence, NZDep score and comorbidities.-Results The AHRQ/AusPSIs were applied to New Zealand administrative hospital data and some 99,366 admissions were associated with a positive indicator event. However rates for some indicators were low (

Aim

Method

Results

Conclusion

Author Information

Phil Hider1; Karl Parker2; Martin von Randow3; Barry Milne3; Roy Lay-Yee3; Peter Davis3 1. Department of Population Health, University of Otago, Christchurch2. SHORE and Whariki Research Centre, School of Public Health, Massey University 3. Compass, Centre of Methods and Policy Application in the Social Sciences, University of Auckland

Acknowledgements

Funding was received from the Health Research Council of New Zealand. Assistance from Drs Patrick Graham and Aidin Jalilzadeh with their statistical input is also gratefully acknowledged.

Correspondence

Dr Phil Hider, Department of Population Health, University of Otago, Christchurch, PO Box 4345, Christchurch 8140, New Zealand.

Correspondence Email

phil.hider@otago.ac.nz

Competing Interests

Nil.

1. Davis P, Lay-Yee R, Briant R, Schug S, Scott A. Adverse events in New Zealand public hospitals. Wellington: Ministry of Health; 2001.2. Wilson RM, Harrison BT, Gibberd RW, Hamilton JD. An analysis of the causes of adverse events from the Quality in Australian Health Care Study. Med J Aust. 1999 May 3;170(9):411-5. PubMed PMID: 10341771. English.3. Brennan TA, Leape LL, Laird NM, Hebert L, Localio AR, Lawthers AG, et al. Incidence of adverse events and negligence in hospitalized patients: results of the Harvard Medical Practice Study I. 1991. Qual Saf Health Care. 2004 Apr;13(2):145-51; discussion 51-2. PubMed PMID: 15069223. Pubmed Central PMCID: Source: NLM. PMC1743811. English.4. Shaw C. How can hospital performance be measured and monitored? Copenhagen: WHO, Regional Office for Europe 2003. Available from: http://www.euro.who.int/__data/assets/pdf_file/0009/74718/E82975.pdf.5. Mattke S, Epstein AM, Leatherman S. The OECD Health Care Quality Indicators Project: history and background. Int J Qual Health Care. 2006 Sep;18 Suppl 1:1-4. PubMed PMID: 16954509. English.6. Drosler SE, Klazinga NS, Romano PS, Tancredi DJ, Gogorcena Aoiz MA, Hewitt MC, et al. Application of patient safety indicators internationally: a pilot study among seven countries. Int J Qual Health Care. 2009 Aug;21(4):272-8. PubMed PMID: 19395469. English.7. Freeman T. using performance indicators to improve health care qualtiy in the public sector: a review of the literature. . Health Serv Manage Res. 2002 May;15(2):126-37.8. Romano PS, Geppert JJ, Davies S, Miller MR, Elixhauser A, McDonald KM. A national profile of patient safety in U.S. hospitals. Health Aff (Millwood). 2003 Mar-Apr;22(2):154-66. PubMed PMID: 12674418. English.9. McDonald K, Romano P, Geppert J, Davies S, Duncan B, Shojania K. Measures of Patient Safety Based on Hospital Administrative Data. The Patient Safety Indicators. . Rockville, MD: Agency for Healthcare Research and Quality; 2002.10. Rosen AK, Rivard P, Zhao S, Loveland S, Tsilimingras D, Christiansen CL, et al. Evaluating the patient safety indicators: how well do they perform on Veterans Health Administration data? Med Care. 2005 Sep;43(9):873-84. PubMed PMID: 16116352. English.11. McConchie S, Shepheard J, Waters S, McMillan AJ, Sundararajan V. The AusPSIs: the Australian version of the Agency of Healthcare Research and Quality patient safety indicators. Aust Health Rev. 2009 May;33(2):334-41. PubMed PMID: 19563325. English.12. Kaafarani HMA, Rosen AK. Using administrative data to identify surgical adverse events: an introduction to the Patient Safety Indicators. Am J Surg. 2009 Nov;198(5 Suppl):S63-8. PubMed PMID: 19874937. English.13. Kaafarani HMA, Borzecki AM, Itani KMF, Loveland S, Mull HJ, Hickson K, et al. Validity of selected Patient Safety Indicators: opportunities and concerns. J Am Coll Surg. 2011 Jun;212(6):924-34. PubMed PMID: 20869268. English.14. Classen D, Resar R, Griffin F, Frederico F, Frankel T, Himmel N. 'Global Trigger Tool' shows that adverse events in hospitals may be ten times greater than previously measured. . Health Aff (Millwood). 2011;30(4):581-9.15. Noble D, Pronovost P. Underreporting of patient safety incidents reduces health care's ability to quantify and accurately measure harm reduction. . J Patient Saf. 2010;6(4):247-50.16. Noble DJ, Pronovost PJ. Underreporting of patient safety incidents reduces health care's ability to quantify and accurately measure harm reduction. J Patient Saf. 2010 Dec;6(4):247-50. PubMed PMID: 21500613. English.17. Chaudry M, Gauld R, Horsburgh S. Hospital quality of care perofrmance measurement and reporting: what new Zealand can learn from the United States and United Kingdom. The New Zealand Medical Journal. 2012;125(1366).18. Intelligence DF. My hospital guide. 2014.19. Medicare. Hospital Compare 2014 [July 2014]. Available from: http://www.medicare.gov/hospitalcompare/search.html 20. SAS Institute Inc. Version 9.2 Cary, North Carolina, United States.2011.21. Anonymous. Patient safety indicators technical specifications version 4.3.: Agency for health Research and Quality 2011. Available from: http://www.qualityindicators.ahrq.gov/modules/PSI_TechSpec.aspx 22. Anonymous. Patient safety indicators technical specifications version 4.3 June 2011: Agency for Health Research and Quality; 2011 [cited 2011 16 November]. Available from: http://www.qualityindicators.ahrq.gov/modules/PSI_TechSpec.aspx 23. Anonymous. Patient safety indicators technical documents Melbourne: Department of Health, State Government of Victoria, Australia; 2009 [cited 2011 June]. Available from: http://www.health.vic.gov.au/psi/auspsi/technical-documents 24. Mohammed M, Worthington P, Woodhall W. Plotting basic control charts: tutorial notes for healthcare practitioners. Qual Saf Health Care. 2008;17(2):137-45.25. Salmond C, Crampton P, Atkinson J. NZDep Index of Deprivation users manual. Wellington: University of Otago, Wellington; 2007.26. Monk TG, Saini V, Weldon BC, Sigl JC. Anesthetic management and one-year mortality after noncardiac surgery. Anesthesia & Analgesia.100(1):4-10. PubMed PMID: 15616043.27. Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi J-C, et al. Coding algorithms for defining comorbdities in ICD-9-CM and ICD-10 administrative data. . Med Care. 2005;43(11):1130-39.28. Scott IA, Poole PJ, Jayathissa S. Improving quality and safety of hospital care: a reappraisal and an agenda for clinically relevant reform. Intern Med J. 2008 Jan;38(1):44-55. PubMed PMID: 18190414.29. Hussey PS, Burns RM, Weinick RM, Mayer L, Cerese J, Farley DO. Using a hospital quality improvement toolkit to improve performance on the AHRQ quality indicators. Jt Comm J Qual Patient Saf. 2013 Apr;39(4):177-84. PubMed PMID: 23641537. English.30. Rosen AK, Zhao S, Rivard P, Loveland S, Montez-Rath ME, Elixhauser A, et al. Tracking rates of Patient Safety Indicators over time: lessons from the Veterans Administration. Med Care. 2006 Sep;44(9):850-61. PubMed PMID: 16932137. English.31. Raleigh VS, Cooper J, Bremner SA, Scobie S. Patient safety indicators for England from hospital administrative data: case-control analysis and comparison with US data. Bmj. 2008;337:a1702. PubMed PMID: 18930971. Pubmed Central PMCID: Source: NLM. PMC2569150. English.32. Zhan C, Miller MR. Administrative data based patient safety research: a critical review. Quality & Safety in Health Care. 2003 Dec;12 Suppl 2:ii58-63. PubMed PMID: 14645897. Pubmed Central PMCID: Source: NLM. PMC1765777. English.33. Quan H. Adaptation of AHRQ patient safety indicators for use in ICD-10 administrative data by an international consortium. 2008 [August 2010]. Available from: http://www.ahrq.gov/downloads/pub/advances2/vol1/advances-quan_52.pdf 34. Parikh S, Christensen D, STuchberry P, Petersen J, Hutchinson A, Jackson T. Exploring in-hospital adverse drug events using ICD-10 codes. Aust Health Rev. 2014;38:454-60.35. Kim H, Capezuti E, Kovner C, Zhao Z, Boockvar K. Prevalence and predictors of adverse events in older surgical patients: impact of the present on admission indicator. Gerontologist. 2010 Dec;50(6):810-20. PubMed PMID: 20566833. English.36. Henderson KE, Recktenwald Aj, Reichley RM, Bailey TC, Waterman BM, Diekemper RL, et al. Clinical validation of the AHRQ postoperative venous thromboembolism patient safety indicator. Jt Comm J Qual Patient Saf. 2009 Jul;35(7):370-6. PubMed PMID: 19634805. English.37. Utter GH, Zrelak PA, Baron R, Tancredi DJ, Sadeghi B, Geppert JJ, et al. Positive predictive value of the AHRQ accidental puncture or laceration patient safety indicator. Ann Surg. 2009 Dec;250(6):1041-5. PubMed PMID: 19779328.38. Anonymous. Episodes of care - additional diagnoses (ICD-10-AM) Canberra: Australian Institute of Health and Welfare; 2011 [cited 2011 October ]. Available from: http://meteor.aihw.gov.au/content/index.phtml/itemId/391322 39. Cuthbertson BH, Boroujerdi M, Prescott G. The use of combined physiological parameters in the early recognition of the deteriorating acute medical patient.[Erratum appears in J R Coll Physicians Edinb. 2010 Jun;40(2):190]. Journal of the Royal College of Physicians of Edinburgh. 2010 Mar;40(1):19-25. PubMed PMID: 21125034.40. Naessens JM, Campbell CR, Huddleston JM, Berg BP, Lefante JJ, Williams AR, et al. A comparison of hospital adverse events identified by three widely used detection methods. Int J Qual Health Care. 2009;21(4):301-7. PubMed PMID: 2009396775. English.41. Duckett S, Coory M, Sketcher-Baker K. Identifying variations in quality of care in Queensland hospitals. Med J Aust. 2007;187(10):571-5.42. Health Mo. Ministry of Health; 2014 [July 2014]. Available from: http://www.health.govt.nz/our-work/primary-health-care/primary-health-care-subsidies-and-services/pho-performance-programme-and-transition-integrated-performance-and-incentive-framework 43. Anonymous. Hospital Compare: US Department of health and Human Services; 2011 [cited 2011 October]. Available from: http://www.hospitalcompare.hhs.gov/ 44. Scott IA, Ward M. Public reporting of hospital outcomes based on administrative data: risks and opportunities. Med J Aust. 2006 Jun 5;184(11):571-5. PubMed PMID: 16768665.45. Ketelaar N, Faber M, Flottorp S, Rygh L, Deane K, Eccles M, et al. Public release of performance data in changing the behaviour of healthcare consumers, professionals or organisations. Cochrane Database of Systematic Reviews 2011;11(CD004538).46. Davis P, Milne B, Parker K, Hider P, Lay-Yee R, Cumming J, et al. Efficiency, effectiveness, equity (E3). Evaluating hospital performance in three dimensions. Health Policy. 2013 Sep;112(1-2):19-27. PubMed PMID: 23537468. English.47. Jones M, Steiner S. Assessing the effect of estimation error on risk-adjusted CUSUM chart performance. Int J Qual Health Care. 2011;24:176-81.

Contact diana@nzma.org.nz
for the PDF of this article

View Article PDF

Since the publication of landmark studies detailing the prevalence of adverse events in hospitals,1–3 worldwide attention has focused on the need to monitor the safety and quality of inpatient care. Four main methods have emerged to monitor hospital performance across aspects of structure, processes and outcomes: regulatory inspection, third party evaluations, internal investigations and statistical indicators.4 Taking advantage of the availability of existing administrative data, indicators have become increasingly employed around the world5,6 as objective measures of hospital performance with two main purposes: summative assessments for external accountability and formative measures for internal quality improvement.7Leading this work, the United States-based Agency for Healthcare Research and Quality (AHRQ) has developed a series of quality and patient safety indicators.8 The AHRQ patient safety indicators (PSIs) were formulated to target occurrences likely to represent adverse events, such as foreign bodies left after procedures. They were developed from a rigorous process that involved extensive literature review, detailed analysis of the International Classification of Diseases (ICD-9) codes, and assessments by clinical panels and empirical analyses.9 The usefulness and validity of the indicators have now been well demonstrated in the United States.10 International interest in the indicators has seen them extended for use in other countries including an adaption by an Australian team at Victoria Healthcare (AusPSIs) based on the same iteration of the ICD-10 coding system that is shared with New Zealand.11The work undertaken for the AusPSIs involved electronic mapping of the AHRQ codes followed by detailed review by an expert coder with subsequent testing. The adaption of the AHRQ indicators to ICD10-AM coding provides an opportunity to explore their functionality in the New Zealand setting.The availability of large datasets that comprehensively include all admissions to public hospitals in a country provides an undisputed advantage in efforts to readily and reliably monitor the quality of inpatient care.12,13 Furthermore employing information coded to internationally agreed standards offers a potentially reliable basis for comparative assessment. The large resources needed to undertake internal investigations using medical record review such as were employed in the New Zealand Quality of Health Care Study1 nullify any chance they could function as an ongoing monitoring tool, although targeted options like the Global Trigger Tool offer considerable promise.14 Other alternatives such as using incident reporting data are plagued by uncertainties associated with the awareness and willingness of staff to volunteer information about their errors.15,16New Zealand is well prepared to capitalize on previous PSI work with the availability of a national inpatient dataset, the National Minimum Dataset (NMDS). The NMDS uses the ICD-10-AM coding system and data are obtained from all District Health Boards and hospitals using a common unique national patient identifier (National Health Index, NHI). However, despite the availability of a comprehensive dataset that includes all publicly funded hospitalisations, New Zealand lags behind other countries particularly the United States and the United Kingdom in its monitoring and reporting of quality measures related to hospital performance.17 Dr Foster in the United Kingdom18 and Hospital Compare in the United States,19 provide online up-to-date information about hospital performance using a range of indicators.The primary objective of this study was to explore the feasibility of applying the AHRQ/AusPSIs to New Zealand administrative data over the period 2001–9 and describe rates of positive indicator events related to medical and surgical care across New Zealand hospitals. Secondary objectives were to explore variation over time in positive indicator event rates and variation between hospitals after risk adjustment.MethodsAll data in the NMDS between and including January 2001 and December 2009 were obtained and linked by NHI to mortality records. Admissions were recorded to 91 public hospitals over the study period although not all hospitals recorded admissions in all years. The data from a small number of facilities were combined to allow for the opening and closing of institutions. Admissions to private facilities were excluded.Coding for the numerators and denominators for 16 indicators in ICD-10-AM (version 3) were developed in SAS (version 9.2) software,20 modified from information provided by AHRQ21,22 and AusPSI,23 and then checked with clinical and coding staff to verify their relevance for the New Zealand setting.PSIs were grouped into three categories: surgical (postoperative), medical, and general. Checks were undertaken of the PSI results to ensure there were no logical discrepancies, such as admissions flagged positive for a postoperative complication when a surgical procedure had not occurred. PSI rates were benchmarked against Australian and other international results and found to be generally consistent.Approval for the study was obtained from the Multi Region Ethics Committee (MEC/08/59).Statistical analyses and control charts—Descriptive statistics are presented for the population of all people discharged from New Zealand public hospitals from 2001 to 2009. Crude positive indicator rates are presented with 95% confidence intervals for the binomial estimates. In order to explore trends in the frequency of indicator positive events over time statistical process control charts are presented for the three most common indicators. Control charts present mean and upper and lower limits based on ±3 standard deviations and enable the identification of common or special causes of variation.24Event rates are presented for each 3-month period. To explore variation in the same three PSIs between hospitals event rates were first risk adjusted using methodology employed by AHRQ.21 The risk adjustment accounted for age, gender, ethnicity, rurality, deprivation and comorbidities and was undertaken with SAS20 software. Deprivation was measured according to the New Zealand Deprivation Score (NZDep01 or NZDep06) based on the domicile of the patient.25The NZDep scores were grouped into five quintiles with the most deprived areas being in quintile five. Rural or urban residence was based on the definition provided by Statistics New Zealand26 for the census area unit related to the patient's domicile. Urban areas were identified (as 0), and rural locations with high (as 1), moderate (as 2) or low (as 3) urban influence were identified. Up to 30 comorbidities were listed for each admission based on the modified Elixhauser set.27ResultsDescription of the patient safety indicators and population summary information—A description of the PSIs is included (Table 1). Over the study period there were 7,487,432 admissions to public hospitals. The people admitted to the facilities over the study period were predominantly female, European, and resided in urban locations and lower socio-economic areas (Table 2). Admission numbers rose steadily across the study period. Hypertension was the most frequent comorbidity (8.6%), while AIDS was rare (<0.1%). Table 1. Definitions of the patient safety indicators Indicator Category Cases At-risk population PSI 1 Complications of anaesthesia General / anaesthesia Episodes with anaesthesia complications All adult surgical discharges PSI 2 death in low mortality DRG Medical Patient death Adults in low-mortality DRGs, (DRGs with a total mortality rate less than 0.5% over the previous three years or less than 0.5% in any of the previous 3 years. PSI 3 Decubitus ulcer Medical Episodes with a decubitus ulcer All adult medical and surgical episodes PSI 4 Failure to rescue Medical Patient death Episodes, adults, potential complication of care (i.e. acute renal failure, DVT/PE, pneumonia, sepsis, shock or cardiac arrest, and GI haemorrhage/acute renal failure) PSI 5 Foreign body left during procedure General Episodes with foreign body left during procedure All adult episodes PSI 6 Iatrogenic pneumothorax General Episodes with pneumothorax. All adult episodes PSI 7 Selected infections due to medical care Medical Episodes with selected infection All adult episodes PSI 8 Postoperative hip fracture Post-op Episodes hip fracture All adult surgical episodes PSI 9 Postoperative haemorrhage or haematoma Post-op Episodes with postoperative haemorrhage or haematoma and postoperative control of haemorrhage or drainage of haematoma All adult surgical episodes PSI 10 Postoperative physiologic and metabolic derangement Post-op Episodes with physiologic and metabolic derangement All elective adult surgical episodes PSI 11 Postoperative respiratory failure Post-op Episodes with acute respiratory failure and a reintubation All adult elective surgical episodes PSI 12 Postoperative DVT or PE Post-op Episodes with pulmonary embolism or deep vein thrombosis All adult surgical episodes PSI 13 Postoperative sepsis Post-op Episodes of sepsis All adult elective surgical episodes PSI 14 Postoperative wound dehiscence Post-op Episodes with reclosure of postoperative disruption of abdominal wall All adult abdominopelvic surgical episodes PSI 15 Accidental puncture or laceration General Episodes with accidental puncture, laceration, cut or perforation All adult episodes PSI 16 Transfusion reaction General Episodes with a transfusion reaction All episodes with a transfusion PSI 17 Birth trauma – injury to neonate Obstetric Episodes with birth trauma All newborn PSI 18 Obstetric trauma – vaginal delivery with instrument Obstetric Episodes 3rd and 4th degree obstetric trauma. All vaginal delivery episodes PSI 19 Obstetric trauma – vaginal delivery without instrument Obstetric Episodes with 3rd and 4th degree obstetric All vaginal delivery episodes PSI 20 Obstetric trauma – caesarean delivery Obstetric Episodes with 3rd and 4th degree obstetric trauma All caesarean delivery episodes Table 2. Summary descriptive data for admissions to New Zealand hospitals 2001–9 Table 3. Number of positive cases, denominators and rates for patient safety indicators across New Zealand hospitals 2001–9 Variables Positive cases Admissions at risk Rate per 10,000 admissions (95% confidence interval) Range of rates per 10,000 admissions across NZ hospitals Medical 2 (death low mortality DRG) 2,157 2,024,136 11 (12–13) 0–20 3 (decubitus ulcer) 17,573 995,744 176 (174–179) 0–351 4 (failure to rescue) 10,240 90,920 1126.(1106–1147) 0–1622 7 (infections) 8,812 1,791,261 49 (48–50) 0–141 General 5 (foreign body) 351 4,990,050 0.7 (0.6–0.8) 0–2 6 (pneumothorax) 1,637 4,814,105 3 (3–4) 0–8 15 (puncture) 10,156 4,979,685 20 (20–21) 0–130 16 (transfusion reaction) 25 294,442 0.8 (0.6–1) 0–7 Anaesthesia 1 (complications) 25 1,663,890 0.15 (0.10–0.2) 0–1 Postoperative 8 (hip fracture) 476 1,036,097 4.5 (4–5) 0–12 9 (haemorrhage) 26,745 1,400,503 191 (189–193) 0–311 10 (physiologic derangement) 230 127,989 18 (16–20) 0–44 11 (respiratory failure) 133 92,602 14 (12–17) 0–64 12 (DVT/PE) 4,557 1,426,837 32 (31–33) 0–125 13 (sepsis) 274 21,172 129 (114–146) 0–870 14 (wound dehiscence) 584 123,814 47 (43–51) 0–101 Results for crude PSIs across study period—Some 99,366 potential safety-related events were identified from a combined total denominator of over 26.8 million admissions at risk of at least one PSI (Table 3). Some 12,397 events represented deaths, either in low mortality Diagnostic Related Groups (DRGs) (i.e. PSI2) or related to major complications such as pneumonia, thromboembolism, sepsis, renal failure, shock, cardiac arrest or gastrointestinal haemorrhage (i.e. PSI4). Some indicators were uncommon with half having less than 600 positive numerator events over the study period.By contrast, the denominators, numbers of admissions at risk of each PSI, were generally high (at least 21,172) such that event rates for 15 indicators were less than 0.2%. Positive event rates ranged between 0.15–1126 per 10,000 admissions and considerable variation was evident between hospitals. Inter-hospital variation was greatest for indicators with the highest event rates, particularly PSIs 4 and 13, such that rates ranged between 0–1622 and 0–870 per 10,000 admissions respectively.Among the categories there was least variation exhibited for the general /anaesthesia indicators (PSIs 1, 5, 6, 15, 16) which were generally infrequent and recorded a maximum average rate of 26 per 10,000 admissions over the study period. Three indicators (PSIs 4, 9, 12) were selected based on both their higher number of positive events and their higher rate of positive events per 10,000 admissions to further explore variation over time and between hospitals. Figure 1. Control charts for PSIs quarterly periods from Jan 2001 – Dec 2009 Figure 2: Risk adjusted PSI rates for New Zealand public hospitals 2001-2009Variation in positive event rates over time—Control charts describe variation in crude positive event rates over time. Special cause variation is evident for three indicators where rates occur outside of the control limits. In the case of one indicators (PSI 4) this variation is evident for more recent data. Other guidelines, such as runs of eight points on one side of the centre line, suggest that PSI12 may also be exhibiting special cause variation in later years. Such variation may prompt further investigation to consider possible explanations beginning with changes in the coding or management of the data.Variation in risk adjusted positive indicator event rates between hospitals—The three indicators also exhibit wide variation between hospitals for their risk-adjusted rates for the study period. Variation was highest for PSI4 where rates ranged between 0-907 per 1,000 discharges at 70 different hospitals. For all the PSIs most hospitals exhibited risk adjusted rates that were broadly similar however outliers were also evident. Many of these outliers related to the presence of small numerator and denominators. For example only 20 events were recorded at the hospital with PSI4 rate of 907 per 1,000 discharges, 4 events with the hospital with rate of 163 per 1,000 discharges for PSI 9 and 10 events for the hospital with a rate of 288 per 1,000 discharges for PSI12.DiscussionThe AHRQ/AusPSI indicators were designed to be used with computerised administrative data that have been already collected from hospitals. All publicly funded hospitals provide the data, and advantages for the indicators include comprehensiveness, unobtrusiveness and low cost (Table 6).28 The PSIs do not rely upon the awareness or efforts of health professionals to recognize adverse events under their care.The results from this study suggest that it is readily feasible to employ the AHRQ/AusPSIs in the New Zealand setting. However, it is noteworthy that event rates are relatively low for some of the indicators even over the combined nine years of the study (e.g., PSIs 1, 5, 6, 8 and 16) and their further utility appears limited. Among the three indicators presented in Figures 1 and 2, considerable variation is evident whether results are described across hospitals or over time. International use of the indicators has largely focused on their employment as part of a toolkit to improve patient care at a hospital,29 track progress over time30 and compare performance at the regional or national level.31Potential limitations associated with administrative data and the AHRQ PSIs have been well described (Table 4).32 The concerns need to be tempered by the realisation that indicators act as alerts or flags rather than definitive assessments of the quality of care. The usefulness of the indicators stems from their function as a screening tool and further efforts are usually required to interrogate the findings to ascertain whether patient safety problems actually exist when indicators suggest that patient care has been compromised.The principal concern with the use of PSIs centres on whether administrative data collected for another purpose can be reliably applied to patient safety measurement. A concern is that the range of patient safety issues identified by the PSIs is limited; for example no information is generated about the presence of adverse drug events by the indicators, yet these events make a large contribution to iatrogenic harm.1 Furthermore any identification of injury is further limited by the completeness and accuracy of the ICD-10 coding system. Although principal diagnoses are usually well recorded in most administrative databases doubts remain about the quality of recording for complications and comorbidities.33Some concerns related to these issues are being addressed. Firstly, new indicators have emerged to describe medication related events using Australian administrative data.34 In addition, the timing of any potential adverse event can now be better defined with New Zealand's adoption of the ‘present on admission' flag which ensures that clinical coders record whether a condition was present at admission or alternatively may have occurred as a complication during the inpatient stay.35 Finally, detailed work to validate the individual indicators by means of medical record review is now nearly complete and is producing some promising results.36, 37 Some of the indicators appear to be associated with high positive predictive values underscoring their role as useful screening tools. Work has been undertaken to validate the indicators in New Zealand using similar record review methodology. Table 4. Main advantages of using administrative data and indicators to measure patient safety Preliminary results suggest that the same PSIs are consistently associated with similarly high positive predictive values in the local setting. These findings are reassuring as, unlike some other countries, New Zealand administrative data only includes secondary diagnoses that were materially relevant to patient care during an inpatient stay.38 As the PSIs rely heavily on secondary diagnosis codes it was possible that some indicator events may either not have been identified or some exclusions could not be applied, so indicators may have been less accurate locally. The final results from the validation exercise will be important to further determine the applicability of the indicators in New Zealand.Limitations aside, the application of the PSIs to New Zealand data raises interesting questions about the local epidemiology of adverse events and the impact of quality improvement initiatives. For example, further work is needed to explore whether the decline in rates of PSI 4 (failure to rescue) can reliably be attributed to the increasing adoption of early warning scoring systems and outreach teams in hospitals.39 Further impetus for the use of PSIs to explore the epidemiology of adverse events comes from the suggestion that the indicators may detect different events from those identified by other methods.40The PSIs could be readily adopted by key organisations in New Zealand, such as the Health Quality and Safety Commission (HQSC) and the Health Roundtable, that are already involved with DHBs in the measurement, analysis and reporting of hospital performance information in New Zealand. The PSIs could augment their existing measures and contribute to their provision of customized dashboards that enable each DHB to assess their progress over time, and with suitably adjusted measures, help them compare themselves against others. Ideally these dashboards should monitor data with as close to real time information as possible. Employing the Variable Life Adjusted Display methodology that has been successfully deployed in Australia,41 they could be used to generate case-by case warnings about any significant differences between observed and expected outcomes in conjunction with a guided response that begins with the judicious scrutiny for any changes in coding and escalates when appropriate to the investigation of service delivery.Whilst their most important function would be to foster internal quality improvement by hospitals, on confirmation of their validity some of them could later be considered for an accountability function and incorporated into the Integrated Performance and Incentive Framework42 with the aim of supporting clinical governance and fostering continuous quality improvement for hospitals alongside primary care. Creating incentives for continuous improvement avoids some of the potential constraints associated with health targets that are limited to determining whether a finite level has been achieved rather than charting progress over time and encouraging, ongoing, iterative cycles of improvement.Considerable interest exists in the public reporting of performance information. In New Zealand, hospital performance data are mainly used internally and between DHBs. This contrasts with the United states where The Department of Health and Human Services in the United States now provides patient safety indicator results for named hospitals on the internet for public scrutiny.43 Proponents argue that such reporting generates opportunities for public accountability, scrutiny and competitive improvement between hospitals.Detractors have counseled about the pitfalls inherent in comparative public reporting, citing limitations of using inadequately validated indicators, insufficient adjustments for case mix, spurious findings occurring when hospitals are ranked even though considerable random variation may be present and the potential for gaming that can occur when attention is focused solely on specific measures.44 Limited evidence exists for the premise that public reporting of indicator data sponsors quality improvement45 and until confirmation is provided from rigorous evaluation in the local setting most attention should be given to promoting the use of PSIs by hospitals and DHBs for internal improvement processes possibly as part of a broader scorecard with metrics related to access, effectiveness and efficiency.46In conclusion, the PSIs can be readily applied to New Zealand's administrative data. Used judiciously, in conjunction with other established and emerging forms of quality assessment and monitoring, such as clinical audit, trigger tools and incident reporting, the PSIs offer another valuable tool for hospitals to assist them with their efforts to enhance patient safety.The advantages of New Zealand's rich administrative dataset combined with its small size suggest that promoting the use of PSIs to enhance patient safety should be readily achievable. Furthermore, New Zealand is well placed to take advantage of the work of existing organisations such as the HQSC and new developments in real time monitoring to sponsor their introduction.47With confirmation of validity, the PSIs could be extended to an accountability framework that encourages continuous improvement and perhaps comprises public reporting as long as it includes appropriate evaluation to ensure that undesired outcomes are not fostered.

Summary

Abstract

Introduction Increasing interest has focused on the safety of hospital care. The AusPSIs are a set of indicators developed from Australian administrative data to reliably identify inpatient adverse events in hospitals. The main aim of this study was to explore the application of the AHRQ/AusPSIs to New Zealand administrative hospital data related to medical and surgical care. Variation over time and across hospitals were also considered for a subset of the more common indicators.-Method AHRQ/AusPSIs were adapted for use with New Zealand National Minimum Dataset administrative data for the period 2001-9. Crude positive event rates for each of the 16 indicators were assessed across New Zealand public hospitals. Variation over time for six more common indicators is presented using statistical control charts. Variation between hospitals was explored using rates adjusted for differences in patient variables including age, sex, ethnicity, rurality of residence, NZDep score and comorbidities.-Results The AHRQ/AusPSIs were applied to New Zealand administrative hospital data and some 99,366 admissions were associated with a positive indicator event. However rates for some indicators were low (

Aim

Method

Results

Conclusion

Author Information

Phil Hider1; Karl Parker2; Martin von Randow3; Barry Milne3; Roy Lay-Yee3; Peter Davis3 1. Department of Population Health, University of Otago, Christchurch2. SHORE and Whariki Research Centre, School of Public Health, Massey University 3. Compass, Centre of Methods and Policy Application in the Social Sciences, University of Auckland

Acknowledgements

Funding was received from the Health Research Council of New Zealand. Assistance from Drs Patrick Graham and Aidin Jalilzadeh with their statistical input is also gratefully acknowledged.

Correspondence

Dr Phil Hider, Department of Population Health, University of Otago, Christchurch, PO Box 4345, Christchurch 8140, New Zealand.

Correspondence Email

phil.hider@otago.ac.nz

Competing Interests

Nil.

1. Davis P, Lay-Yee R, Briant R, Schug S, Scott A. Adverse events in New Zealand public hospitals. Wellington: Ministry of Health; 2001.2. Wilson RM, Harrison BT, Gibberd RW, Hamilton JD. An analysis of the causes of adverse events from the Quality in Australian Health Care Study. Med J Aust. 1999 May 3;170(9):411-5. PubMed PMID: 10341771. English.3. Brennan TA, Leape LL, Laird NM, Hebert L, Localio AR, Lawthers AG, et al. Incidence of adverse events and negligence in hospitalized patients: results of the Harvard Medical Practice Study I. 1991. Qual Saf Health Care. 2004 Apr;13(2):145-51; discussion 51-2. PubMed PMID: 15069223. Pubmed Central PMCID: Source: NLM. PMC1743811. English.4. Shaw C. How can hospital performance be measured and monitored? Copenhagen: WHO, Regional Office for Europe 2003. Available from: http://www.euro.who.int/__data/assets/pdf_file/0009/74718/E82975.pdf.5. Mattke S, Epstein AM, Leatherman S. The OECD Health Care Quality Indicators Project: history and background. Int J Qual Health Care. 2006 Sep;18 Suppl 1:1-4. PubMed PMID: 16954509. English.6. Drosler SE, Klazinga NS, Romano PS, Tancredi DJ, Gogorcena Aoiz MA, Hewitt MC, et al. Application of patient safety indicators internationally: a pilot study among seven countries. Int J Qual Health Care. 2009 Aug;21(4):272-8. PubMed PMID: 19395469. English.7. Freeman T. using performance indicators to improve health care qualtiy in the public sector: a review of the literature. . Health Serv Manage Res. 2002 May;15(2):126-37.8. Romano PS, Geppert JJ, Davies S, Miller MR, Elixhauser A, McDonald KM. A national profile of patient safety in U.S. hospitals. Health Aff (Millwood). 2003 Mar-Apr;22(2):154-66. PubMed PMID: 12674418. English.9. McDonald K, Romano P, Geppert J, Davies S, Duncan B, Shojania K. Measures of Patient Safety Based on Hospital Administrative Data. The Patient Safety Indicators. . Rockville, MD: Agency for Healthcare Research and Quality; 2002.10. Rosen AK, Rivard P, Zhao S, Loveland S, Tsilimingras D, Christiansen CL, et al. Evaluating the patient safety indicators: how well do they perform on Veterans Health Administration data? Med Care. 2005 Sep;43(9):873-84. PubMed PMID: 16116352. English.11. McConchie S, Shepheard J, Waters S, McMillan AJ, Sundararajan V. The AusPSIs: the Australian version of the Agency of Healthcare Research and Quality patient safety indicators. Aust Health Rev. 2009 May;33(2):334-41. PubMed PMID: 19563325. English.12. Kaafarani HMA, Rosen AK. Using administrative data to identify surgical adverse events: an introduction to the Patient Safety Indicators. Am J Surg. 2009 Nov;198(5 Suppl):S63-8. PubMed PMID: 19874937. English.13. Kaafarani HMA, Borzecki AM, Itani KMF, Loveland S, Mull HJ, Hickson K, et al. Validity of selected Patient Safety Indicators: opportunities and concerns. J Am Coll Surg. 2011 Jun;212(6):924-34. PubMed PMID: 20869268. English.14. Classen D, Resar R, Griffin F, Frederico F, Frankel T, Himmel N. 'Global Trigger Tool' shows that adverse events in hospitals may be ten times greater than previously measured. . Health Aff (Millwood). 2011;30(4):581-9.15. Noble D, Pronovost P. Underreporting of patient safety incidents reduces health care's ability to quantify and accurately measure harm reduction. . J Patient Saf. 2010;6(4):247-50.16. Noble DJ, Pronovost PJ. Underreporting of patient safety incidents reduces health care's ability to quantify and accurately measure harm reduction. J Patient Saf. 2010 Dec;6(4):247-50. PubMed PMID: 21500613. English.17. Chaudry M, Gauld R, Horsburgh S. Hospital quality of care perofrmance measurement and reporting: what new Zealand can learn from the United States and United Kingdom. The New Zealand Medical Journal. 2012;125(1366).18. Intelligence DF. My hospital guide. 2014.19. Medicare. Hospital Compare 2014 [July 2014]. Available from: http://www.medicare.gov/hospitalcompare/search.html 20. SAS Institute Inc. Version 9.2 Cary, North Carolina, United States.2011.21. Anonymous. Patient safety indicators technical specifications version 4.3.: Agency for health Research and Quality 2011. Available from: http://www.qualityindicators.ahrq.gov/modules/PSI_TechSpec.aspx 22. Anonymous. Patient safety indicators technical specifications version 4.3 June 2011: Agency for Health Research and Quality; 2011 [cited 2011 16 November]. Available from: http://www.qualityindicators.ahrq.gov/modules/PSI_TechSpec.aspx 23. Anonymous. Patient safety indicators technical documents Melbourne: Department of Health, State Government of Victoria, Australia; 2009 [cited 2011 June]. Available from: http://www.health.vic.gov.au/psi/auspsi/technical-documents 24. Mohammed M, Worthington P, Woodhall W. Plotting basic control charts: tutorial notes for healthcare practitioners. Qual Saf Health Care. 2008;17(2):137-45.25. Salmond C, Crampton P, Atkinson J. NZDep Index of Deprivation users manual. Wellington: University of Otago, Wellington; 2007.26. Monk TG, Saini V, Weldon BC, Sigl JC. Anesthetic management and one-year mortality after noncardiac surgery. Anesthesia & Analgesia.100(1):4-10. PubMed PMID: 15616043.27. Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi J-C, et al. Coding algorithms for defining comorbdities in ICD-9-CM and ICD-10 administrative data. . Med Care. 2005;43(11):1130-39.28. Scott IA, Poole PJ, Jayathissa S. Improving quality and safety of hospital care: a reappraisal and an agenda for clinically relevant reform. Intern Med J. 2008 Jan;38(1):44-55. PubMed PMID: 18190414.29. Hussey PS, Burns RM, Weinick RM, Mayer L, Cerese J, Farley DO. Using a hospital quality improvement toolkit to improve performance on the AHRQ quality indicators. Jt Comm J Qual Patient Saf. 2013 Apr;39(4):177-84. PubMed PMID: 23641537. English.30. Rosen AK, Zhao S, Rivard P, Loveland S, Montez-Rath ME, Elixhauser A, et al. Tracking rates of Patient Safety Indicators over time: lessons from the Veterans Administration. Med Care. 2006 Sep;44(9):850-61. PubMed PMID: 16932137. English.31. Raleigh VS, Cooper J, Bremner SA, Scobie S. Patient safety indicators for England from hospital administrative data: case-control analysis and comparison with US data. Bmj. 2008;337:a1702. PubMed PMID: 18930971. Pubmed Central PMCID: Source: NLM. PMC2569150. English.32. Zhan C, Miller MR. Administrative data based patient safety research: a critical review. Quality & Safety in Health Care. 2003 Dec;12 Suppl 2:ii58-63. PubMed PMID: 14645897. Pubmed Central PMCID: Source: NLM. PMC1765777. English.33. Quan H. Adaptation of AHRQ patient safety indicators for use in ICD-10 administrative data by an international consortium. 2008 [August 2010]. Available from: http://www.ahrq.gov/downloads/pub/advances2/vol1/advances-quan_52.pdf 34. Parikh S, Christensen D, STuchberry P, Petersen J, Hutchinson A, Jackson T. Exploring in-hospital adverse drug events using ICD-10 codes. Aust Health Rev. 2014;38:454-60.35. Kim H, Capezuti E, Kovner C, Zhao Z, Boockvar K. Prevalence and predictors of adverse events in older surgical patients: impact of the present on admission indicator. Gerontologist. 2010 Dec;50(6):810-20. PubMed PMID: 20566833. English.36. Henderson KE, Recktenwald Aj, Reichley RM, Bailey TC, Waterman BM, Diekemper RL, et al. Clinical validation of the AHRQ postoperative venous thromboembolism patient safety indicator. Jt Comm J Qual Patient Saf. 2009 Jul;35(7):370-6. PubMed PMID: 19634805. English.37. Utter GH, Zrelak PA, Baron R, Tancredi DJ, Sadeghi B, Geppert JJ, et al. Positive predictive value of the AHRQ accidental puncture or laceration patient safety indicator. Ann Surg. 2009 Dec;250(6):1041-5. PubMed PMID: 19779328.38. Anonymous. Episodes of care - additional diagnoses (ICD-10-AM) Canberra: Australian Institute of Health and Welfare; 2011 [cited 2011 October ]. Available from: http://meteor.aihw.gov.au/content/index.phtml/itemId/391322 39. Cuthbertson BH, Boroujerdi M, Prescott G. The use of combined physiological parameters in the early recognition of the deteriorating acute medical patient.[Erratum appears in J R Coll Physicians Edinb. 2010 Jun;40(2):190]. Journal of the Royal College of Physicians of Edinburgh. 2010 Mar;40(1):19-25. PubMed PMID: 21125034.40. Naessens JM, Campbell CR, Huddleston JM, Berg BP, Lefante JJ, Williams AR, et al. A comparison of hospital adverse events identified by three widely used detection methods. Int J Qual Health Care. 2009;21(4):301-7. PubMed PMID: 2009396775. English.41. Duckett S, Coory M, Sketcher-Baker K. Identifying variations in quality of care in Queensland hospitals. Med J Aust. 2007;187(10):571-5.42. Health Mo. Ministry of Health; 2014 [July 2014]. Available from: http://www.health.govt.nz/our-work/primary-health-care/primary-health-care-subsidies-and-services/pho-performance-programme-and-transition-integrated-performance-and-incentive-framework 43. Anonymous. Hospital Compare: US Department of health and Human Services; 2011 [cited 2011 October]. Available from: http://www.hospitalcompare.hhs.gov/ 44. Scott IA, Ward M. Public reporting of hospital outcomes based on administrative data: risks and opportunities. Med J Aust. 2006 Jun 5;184(11):571-5. PubMed PMID: 16768665.45. Ketelaar N, Faber M, Flottorp S, Rygh L, Deane K, Eccles M, et al. Public release of performance data in changing the behaviour of healthcare consumers, professionals or organisations. Cochrane Database of Systematic Reviews 2011;11(CD004538).46. Davis P, Milne B, Parker K, Hider P, Lay-Yee R, Cumming J, et al. Efficiency, effectiveness, equity (E3). Evaluating hospital performance in three dimensions. Health Policy. 2013 Sep;112(1-2):19-27. PubMed PMID: 23537468. English.47. Jones M, Steiner S. Assessing the effect of estimation error on risk-adjusted CUSUM chart performance. Int J Qual Health Care. 2011;24:176-81.

Contact diana@nzma.org.nz
for the PDF of this article

Subscriber Content

The full contents of this pages only available to subscribers.

LOGINSUBSCRIBE