View Article PDF

The Systematised Nomenclature of Medicine—Clinical Terms (SNOMED-CT) coding system has been adopted by the Ministry of Health in Aotearoa New Zealand to replace other medical coding systems.1 The MOH has mandated that Emergency Departments (ED) in New Zealand must report all visits using SNOMED-CT codes for Chief Presenting Complaint (CPC), Procedures and Diagnoses to the National Non-admitted Patient Collection (NNPAC) by July 2021.2 It is hoped that this will correct a large gap in the data about acute presentations which is not captured currently.3 This is especially important as there are implications for disease surveillance, new trends in recreational drug toxicity and especially relevant in the current context of SARS COV-2: early identification of possible pandemics.4 Auckland City Hospital (ACH) emergency department (ED) introduced coding CPC with SNOMED-CT in March 2019 as an ‘early adopter’ pilot site for the MOH national rollout and are currently in the process of introducing procedure and diagnosis codes. In the MOH reference set for emergency care there were initially 144 possible presenting complaints, although this has now been expanded to 152 after feedback from stakeholders.2 We currently do not know how accurate coding of CPC using SNOMED-CT is in a New Zealand ED context. The aim of the current study is to determine how accurate coding CPC with SNOMED-CT was by triage staff compared to clinician’s notes.

Method

This was a retrospective chart review.

Setting

ACH is an urban tertiary academic centre that has an annual census of approximately 76,000 patients 15 years or older.

Case selection

One thousand consecutive presentations to ED were selected, between the dates 19 July 2019 and 23 July 2019. The start date was arbitrarily chosen to give staff sufficient time (four months) to get used to using SNOMED-CT.

Recording of presenting complaints

Triage CPC

CPC was recorded in real-time as soon as possible on patient arrival at ED. This is done by trained triage nurses for all ED presentations, except those ambulance patients requiring immediate resuscitation, or primary care referrals to inpatient teams. Clerical staff record the CPC for patients referred by their primary care provider or patients requiring immediate resuscitation (while clinical staff provide immediate care). A single CPC is recorded from the MOH reference set into the electronic patient management system (Triage CPC). There is provision within the system to add free text comments about the CPC or mention other presenting complaints if there are multiple (Triage Comments).

Auditor CPC

Treating clinicians typically begin their clinical notes stating the presenting complaint and history of presenting complaints. These notes were audited, and the auditor recorded what the clinician stated as the presenting complaints (Auditor CPC). The first presenting complaint mentioned in the clinical notes was recorded as the first Auditor CPC. Up to five Auditor CPC were recorded if there were multiple presenting complaints. The auditors were a senior emergency nurse and a trained final year medical student. The lead author reviewed the first few cases with the auditors, until satisfied that data collection was reliable. No formal check of data accuracy was performed. Where there was uncertainty about the CPC in a particular case the auditors discussed with the lead author and resolved by consensus. Only the date/time of presentation, visit number and NHI were provided to the auditors. Patient records were then accessed by the auditors using the clinical records systems Concerto (Orion Health, Auckland, New Zealand) and 3M™ ChartView™ (3M Company, Salt Lake City, Utah, US) using a password-protected computer based within the ED. Electronic clinical notes for all patients were reviewed independently by an auditor blinded to the real-time CPC. The auditor recorded CPC based on the first treating clinician’s clinical notes onto a purpose-built electronic data collection form in Microsoft Excel® (Microsoft Corporation, Redmond, Washington, US). The first treating clinician was either an independent nurse practitioner or doctor. When there was more than one presenting complaint recorded by the first treating clinician the auditor could record up to five presenting complaints.

Outcomes

The primary outcome was the agreement between triage CPC and the first recorded auditor CPC. Secondary analyses were differences between patient groups, and overall agreement between triage CPC plus triage comments and all recorded auditor CPCs.

Data collection

The following data was retrieved from the hospital data warehouse: event number, NHI, date/time of presentation; age, sex, ethnicity, CPC and free text triage comments fields. This was received by one author in the form of a .csv file and stored in a password-protected computer in a locked office in the ED.

Data handling

Once all data was collected the two data sources were merged into one file for analysis by one author.

Data analysis

Auditor’s CPC was compared directly to the triage CPC and triage comments. The proportion exact matches between triage CPC or triage comments and auditor CPC was recorded as n, % 95% CI. The 95% CI was calculated using Graphpad QuickCalcs (http://www.graphpad.com/quickcalcs/confInterval1/), San Diego, CA, US. Tests for differences between groups were done using Chi2, using Open Epi.5 A two-tailed p<0.05 was considered statistically significant.

Ethics and institutional approval

As an audit of the data quality undertaken by staff in the department where this data was collected and no identifying data would be reported, this study did not meet the threshold to require ethical approval. Institutional approval was obtained from the ADHB research office (A+8678).

Results

The case selection process is shown in Figure 1. Of 1,000 consecutive presentations, 852 were seen in ED and had notes available for the analysis (85%). The demographic details for the cases are shown in Table 1. There were 89 different presenting complaints recorded in real-time of 144 possible (62%) from the original MOH reference set. The most common CPC were related to trauma, with all ‘injury’ related CPC 175/852 (20.5%), ‘abdominal pain’ 94 (11.0%), ‘chest pain’ 87 (10.2%), ‘shortness of breath’ 66 (7.7%), ‘referral for investigation’ 43 (5.0)%, and ‘fever symptoms’ 30 (3.5%).

Figure 1: Case selection.

Table 1: Demographics of sample population.

SD = standard deviation, CI = confidence interval.

Figure 2 shows the auditor’s first CPC agreement with the triage CPC, which was 514/852, 60.3% (95% CI 57–64%). When triage free text comments were also considered, the agreement between the auditor’s CPCs derived from the first clinician’s notes and the triage presenting complaints was 732/852, 85.9% (95% CI 83–88%) of cases (Figure 3).

Figure 2: Agreement between auditor and real-time SNOMED-CT CPC

SNOMED-CT= Systematised Nomenclature of Medicine – Clinical Terms, CPC = chief presenting complaint.

Figure 3: Agreement between auditor and real-Time SNOMED-CT CPC + triage comments

SNOMED-CT= Systematised Nomenclature of Medicine – Clinical Terms, CPC = chief presenting complaint.

There was disagreement between either real-time CPC and triage comment and any auditor selected for 186 patients across 51 CPC.

For the common clinical complaints related to trauma, ‘abdominal pain’, ‘chest pain’ and ‘shortness of breath’, there was substantial agreement between the auditor’s primary CPC and the real-time CPC, 661/852=77.5%, k=0.66 (0.61, 0.70), as shown in Table 2.

Table 2: Agreement for common CPC.

CPC = chief presenting complaint.

The worst agreement was for CPC that were non-specific. Agreement for ‘referral for investigation’ was (6/43, 14%). For this CPC, the auditors recorded an alternative more specific CPC in 37/43 (86%) cases (Figure 4). When triage comments were included in the comparisons, the level of agreement increased to 24/43 (56%).

Figure 4: Clinician CPC compared to triage CPC ‘referral for investigation’.

Green indicates agreement between auditor and real-time CPC, red indicates auditor CPC when real-time CPC was ‘referral for investigation’. CPC = chief presenting complaint.

Similarly, for ‘fever symptoms’ there was only 17/29, 57% agreement on CPC which improved to 24/29 (80%) with triage comments included. For ‘general weakness/fatigue/unwell’ there was no agreement on CPC, 0/16 (0%). This improved to 8/16 (50%) when triage comments were included.

There were occasional disagreements for many CPC: For example, eight patients with injuries who were coded as having ‘pain in body part (no recent injury)’ and three patients with ‘fall(s) no significant injury’ when injuries were present. Similarly, 11 patients had injury-related CPC who were coded as pain without injury by the auditors. Also, many CPC overlap, such as ‘collapse/syncope’ and ‘fall(s) no significant injury’ (four cases coded either way in real-time or by auditor); for acute toxicological presentations CPC may also overlap, for example ‘alcohol intoxication’, ‘drug intoxication’, ‘overdose of drug’ or ‘ingestion of harmful substance’ (seven cases with variable coding either in real-time or by auditors).

Table 3 shows the agreement based on demographic characteristics. There was no difference in the rate of agreement on CPC by age category or sex. There was a difference for different ethnicities which bordered statistical significance. The most agreement was for Pacific peoples, 66% (95% CI 57–74) and least for Māori, 51% (95%CI 41-61). This difference was not evident when triage comments were also considered int the agreement calculation (Table 3).

Table 3: Agreement on CPC by demographic variable.

Discussion

This is the first audit of the accuracy of CPC coding using SNOMED-CT compared to first treating clinician documentation of presenting complaints in the New Zealand ED setting. The overall accuracy was reasonable in our view, with good agreement for the majority of CPC, especially when triage comments were also considered. Differences observed between different age groups, sex or ethnicity were trivial.

Accuracy for the CPC ‘general weakness/fatigue/unwell’ and ‘referral for investigation’ CPCs were poor, with three quarters or more of these having more specific CPC recorded by the treating clinician. As a result of this audit we have fed back to triage staff to avoid using more general CPCs where possible. The ‘referral for investigation’ CPC was used for many primary care referrals and for those patients brought by ambulance direct to the resuscitation room—for these patients it was expedient for clerks who entered the patients into the hospital information system to use this code. The other feedback provided was to avoid trying to label patients with a diagnosis using the CPC presumptive diagnoses can be placed in the free text comments if required.

Given that patients often present to ED with more than one presenting complaint, histories evolve over time and that the emphasis given by patients or clinicians to a particular presenting complaint when more than one symptom is present may differ, the accuracy observed in this study was considered reasonable, and compares well to that previously reported in Australia by research staff determining CPC from free text. In the study by Wagholikar and colleagues, agreement between two research assistants extrapolating CPC from free text is reported for the five most common CPC (abdominal pain, chest pain, shortness of breath, trauma and other, n=364). Using the raw data provided, it was possible to estimate the overall interobserver agreement in their study: k=0.56 (0.50, 0.617), n=364,6 which is moderate agreement. In our audit using the same five categories there was substantial agreement.7

It is clear that when using the CPC for audit of particular conditions to determine the denominator of interest, or for disease surveillance more than one CPC needs to be flagged to maximise the sensitivity of searches. This was evident for the trauma-related CPC where patients may be coded by the location of their injury or the symptoms caused; education is required such that trauma presentations should not be coded as ‘pain in body part (no injury)’. In the case of SARS-Cov-2 (or future viral respiratory pandemics) the relevant CPC may be ‘fever symptoms’; ‘shortness of breath’; ‘cough’; ‘sore throat’ or ‘exposure to communicable disease’. Simply screening for one of these will miss relevant cases.

Having a short but comprehensive list of searchable presenting complaints is a vast improvement for our department. Our previous system of free text data entry for presenting complaints made any form of clinical audit based on presenting complaints very difficult, due to the myriad of variations of spellings, misspelling, acronym use and other abbreviations that were possible for each and every presenting complaint. Ongoing audit and education will be important to improve/maintain real-time coding accuracy for CPC.

The next steps for our department are to introduce the reference sets for ED procedures and diagnosis. Once this is embedded at a national level, we will for the first time have a real picture of the demand for and the opportunity to determine the value added by ED care in New Zealand.8

Conclusion

The accuracy of SNOMED-CT CPC was reasonable in our department, although several areas for improvement were identified, most importantly the need to avoid using more general CPC when a specific CPC is available. Use of CPC for research or disease surveillance will require careful consideration of local coding practice and use of multiple CPC in searches to maximise sensitivity.

Summary

Abstract

Aim

The Systematised Nomenclature of Medicine – Clinical Terms (SNOMED-CT) coding system has been introduced to emergency departments in New Zealand, starting with the patient’s chief presenting complaint (CPC). However, it is not known how accurate CPC coding at triage is, or whether coding accuracy varies by patient characteristics. The aim of this study was to determine the accuracy of CPC coding.

Method

We compared the real-time triage recorded CPC with the presenting complaints recorded in medical notes by the treating clinician. Agreement was determined by exact CPC matches and the kappa statistic.

Results

From 1,000 consecutive presentations 852 were eligible (89 CPCs). Real-time CPC agreed with clinical notes in 514/852 (60.3%) cases. When real-time free text comments were included, agreement was 732/852 (85.9%). There were no differences by age, sex or ethnicity. Agreement for the common CPCs: ‘trauma’ (21%); ‘abdominal pain’ (11%), ‘chest pain’ 87 (10%) and ‘shortness of breath’ (8%) was substantial, k=0.66 (95% CI: 0.61, 0.70). Use of non-specific CPC such as ‘referral for investigation’ (5%) and ‘general weakness/fatigue/unwell’ (2%) was uncommon but associated with poor agreement.

Conclusion

The accuracy of real-time coding for CPC using SNOMED-CT was reasonable, except for non-specific CPCs, which should be avoided where possible.

Author Information

Peter G Jones, Associate Professor, Department of Surgery, School of Medicine, University of Auckland, Auckland; Emergency Physician, Adult Emergency Department, Auckland City Hospital, Auckland; Annette Dance, Charge Nurse, Adult Emergency Department, Auckland City Hospital, Auckland; Nikhar Shah, House Officer, Auckland City Hospital, Auckland.

Acknowledgements

The authors would like to thank Dr Mark Gardener for his assistance with data collection for this study.

Correspondence

Prof Peter Jones, Adult Emergency Department, Auckland City Hospital, ADHB, Private Bag 92024 Victoria St West, Auckland 1142.

Correspondence Email

peterj@adhb.govt.nz

Competing Interests

Prof Jones is the clinical lead at the Ministry of Health for the Shorter Stays in Emergency Departments health target and on the sector advisory group for implementation of SNOMED-CT.

1. MOH: SNOMED-CT [20/4/20]. Available from: http://www.health.govt.nz/nz-health-statistics/classification-and-terminology/new-zealand-snomed-ct-national-release-centre/snomed-ct

2. MOH: HISO 10048 Emergency Care Data Standard [20/4/20]. Available from: http://www.health.govt.nz/publication/hiso-10048-emergency-care-data-standard

3. Jones P, Hughes T, Morton T. Information technology coding systems in the emergency department: It is not the tools, it is how we use them. Emergency Medicine Australasia 2019; 31(5):700–01. doi: 10.1111/1742-6723.13378

4. Elkin PL, Brown SH, Balas A, et al. Biosurveillance evaluation of SNOMED CT's terminology (BEST Trial): coverage of chief complaints. Studies in Health Technology & Informatics 2008; 136:797–802.

5. Dean AG, Sullivan KM, Soe MM. OpenEpi: Open Source Epidemiologic Statistics for Public Health, Version. www.OpenEpi.com, updated 2013/04/06, accessed 2020/04/28.

6. Wagholikar AS, Lawley MJ, Hansen DP, et al. Identifying symptom groups from Emergency Department presenting complaint free text using SNOMED CT. AMIA Annu Symp Proc 2011; 2011:1446–53 [published Online First: 2011/10/22].

7. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33(1):159–74 [published Online First: 1977/03/01].

8. Hughes T, Higginson I, Mann C. Tariffs in emergency care. British journal of hospital medicine (London, England : 2005) 2014; 75(11):631–6. doi: 10.12968/hmed.2014.75.11.631 [published Online First: 2014/11/11].

Contact diana@nzma.org.nz
for the PDF of this article

View Article PDF

The Systematised Nomenclature of Medicine—Clinical Terms (SNOMED-CT) coding system has been adopted by the Ministry of Health in Aotearoa New Zealand to replace other medical coding systems.1 The MOH has mandated that Emergency Departments (ED) in New Zealand must report all visits using SNOMED-CT codes for Chief Presenting Complaint (CPC), Procedures and Diagnoses to the National Non-admitted Patient Collection (NNPAC) by July 2021.2 It is hoped that this will correct a large gap in the data about acute presentations which is not captured currently.3 This is especially important as there are implications for disease surveillance, new trends in recreational drug toxicity and especially relevant in the current context of SARS COV-2: early identification of possible pandemics.4 Auckland City Hospital (ACH) emergency department (ED) introduced coding CPC with SNOMED-CT in March 2019 as an ‘early adopter’ pilot site for the MOH national rollout and are currently in the process of introducing procedure and diagnosis codes. In the MOH reference set for emergency care there were initially 144 possible presenting complaints, although this has now been expanded to 152 after feedback from stakeholders.2 We currently do not know how accurate coding of CPC using SNOMED-CT is in a New Zealand ED context. The aim of the current study is to determine how accurate coding CPC with SNOMED-CT was by triage staff compared to clinician’s notes.

Method

This was a retrospective chart review.

Setting

ACH is an urban tertiary academic centre that has an annual census of approximately 76,000 patients 15 years or older.

Case selection

One thousand consecutive presentations to ED were selected, between the dates 19 July 2019 and 23 July 2019. The start date was arbitrarily chosen to give staff sufficient time (four months) to get used to using SNOMED-CT.

Recording of presenting complaints

Triage CPC

CPC was recorded in real-time as soon as possible on patient arrival at ED. This is done by trained triage nurses for all ED presentations, except those ambulance patients requiring immediate resuscitation, or primary care referrals to inpatient teams. Clerical staff record the CPC for patients referred by their primary care provider or patients requiring immediate resuscitation (while clinical staff provide immediate care). A single CPC is recorded from the MOH reference set into the electronic patient management system (Triage CPC). There is provision within the system to add free text comments about the CPC or mention other presenting complaints if there are multiple (Triage Comments).

Auditor CPC

Treating clinicians typically begin their clinical notes stating the presenting complaint and history of presenting complaints. These notes were audited, and the auditor recorded what the clinician stated as the presenting complaints (Auditor CPC). The first presenting complaint mentioned in the clinical notes was recorded as the first Auditor CPC. Up to five Auditor CPC were recorded if there were multiple presenting complaints. The auditors were a senior emergency nurse and a trained final year medical student. The lead author reviewed the first few cases with the auditors, until satisfied that data collection was reliable. No formal check of data accuracy was performed. Where there was uncertainty about the CPC in a particular case the auditors discussed with the lead author and resolved by consensus. Only the date/time of presentation, visit number and NHI were provided to the auditors. Patient records were then accessed by the auditors using the clinical records systems Concerto (Orion Health, Auckland, New Zealand) and 3M™ ChartView™ (3M Company, Salt Lake City, Utah, US) using a password-protected computer based within the ED. Electronic clinical notes for all patients were reviewed independently by an auditor blinded to the real-time CPC. The auditor recorded CPC based on the first treating clinician’s clinical notes onto a purpose-built electronic data collection form in Microsoft Excel® (Microsoft Corporation, Redmond, Washington, US). The first treating clinician was either an independent nurse practitioner or doctor. When there was more than one presenting complaint recorded by the first treating clinician the auditor could record up to five presenting complaints.

Outcomes

The primary outcome was the agreement between triage CPC and the first recorded auditor CPC. Secondary analyses were differences between patient groups, and overall agreement between triage CPC plus triage comments and all recorded auditor CPCs.

Data collection

The following data was retrieved from the hospital data warehouse: event number, NHI, date/time of presentation; age, sex, ethnicity, CPC and free text triage comments fields. This was received by one author in the form of a .csv file and stored in a password-protected computer in a locked office in the ED.

Data handling

Once all data was collected the two data sources were merged into one file for analysis by one author.

Data analysis

Auditor’s CPC was compared directly to the triage CPC and triage comments. The proportion exact matches between triage CPC or triage comments and auditor CPC was recorded as n, % 95% CI. The 95% CI was calculated using Graphpad QuickCalcs (http://www.graphpad.com/quickcalcs/confInterval1/), San Diego, CA, US. Tests for differences between groups were done using Chi2, using Open Epi.5 A two-tailed p<0.05 was considered statistically significant.

Ethics and institutional approval

As an audit of the data quality undertaken by staff in the department where this data was collected and no identifying data would be reported, this study did not meet the threshold to require ethical approval. Institutional approval was obtained from the ADHB research office (A+8678).

Results

The case selection process is shown in Figure 1. Of 1,000 consecutive presentations, 852 were seen in ED and had notes available for the analysis (85%). The demographic details for the cases are shown in Table 1. There were 89 different presenting complaints recorded in real-time of 144 possible (62%) from the original MOH reference set. The most common CPC were related to trauma, with all ‘injury’ related CPC 175/852 (20.5%), ‘abdominal pain’ 94 (11.0%), ‘chest pain’ 87 (10.2%), ‘shortness of breath’ 66 (7.7%), ‘referral for investigation’ 43 (5.0)%, and ‘fever symptoms’ 30 (3.5%).

Figure 1: Case selection.

Table 1: Demographics of sample population.

SD = standard deviation, CI = confidence interval.

Figure 2 shows the auditor’s first CPC agreement with the triage CPC, which was 514/852, 60.3% (95% CI 57–64%). When triage free text comments were also considered, the agreement between the auditor’s CPCs derived from the first clinician’s notes and the triage presenting complaints was 732/852, 85.9% (95% CI 83–88%) of cases (Figure 3).

Figure 2: Agreement between auditor and real-time SNOMED-CT CPC

SNOMED-CT= Systematised Nomenclature of Medicine – Clinical Terms, CPC = chief presenting complaint.

Figure 3: Agreement between auditor and real-Time SNOMED-CT CPC + triage comments

SNOMED-CT= Systematised Nomenclature of Medicine – Clinical Terms, CPC = chief presenting complaint.

There was disagreement between either real-time CPC and triage comment and any auditor selected for 186 patients across 51 CPC.

For the common clinical complaints related to trauma, ‘abdominal pain’, ‘chest pain’ and ‘shortness of breath’, there was substantial agreement between the auditor’s primary CPC and the real-time CPC, 661/852=77.5%, k=0.66 (0.61, 0.70), as shown in Table 2.

Table 2: Agreement for common CPC.

CPC = chief presenting complaint.

The worst agreement was for CPC that were non-specific. Agreement for ‘referral for investigation’ was (6/43, 14%). For this CPC, the auditors recorded an alternative more specific CPC in 37/43 (86%) cases (Figure 4). When triage comments were included in the comparisons, the level of agreement increased to 24/43 (56%).

Figure 4: Clinician CPC compared to triage CPC ‘referral for investigation’.

Green indicates agreement between auditor and real-time CPC, red indicates auditor CPC when real-time CPC was ‘referral for investigation’. CPC = chief presenting complaint.

Similarly, for ‘fever symptoms’ there was only 17/29, 57% agreement on CPC which improved to 24/29 (80%) with triage comments included. For ‘general weakness/fatigue/unwell’ there was no agreement on CPC, 0/16 (0%). This improved to 8/16 (50%) when triage comments were included.

There were occasional disagreements for many CPC: For example, eight patients with injuries who were coded as having ‘pain in body part (no recent injury)’ and three patients with ‘fall(s) no significant injury’ when injuries were present. Similarly, 11 patients had injury-related CPC who were coded as pain without injury by the auditors. Also, many CPC overlap, such as ‘collapse/syncope’ and ‘fall(s) no significant injury’ (four cases coded either way in real-time or by auditor); for acute toxicological presentations CPC may also overlap, for example ‘alcohol intoxication’, ‘drug intoxication’, ‘overdose of drug’ or ‘ingestion of harmful substance’ (seven cases with variable coding either in real-time or by auditors).

Table 3 shows the agreement based on demographic characteristics. There was no difference in the rate of agreement on CPC by age category or sex. There was a difference for different ethnicities which bordered statistical significance. The most agreement was for Pacific peoples, 66% (95% CI 57–74) and least for Māori, 51% (95%CI 41-61). This difference was not evident when triage comments were also considered int the agreement calculation (Table 3).

Table 3: Agreement on CPC by demographic variable.

Discussion

This is the first audit of the accuracy of CPC coding using SNOMED-CT compared to first treating clinician documentation of presenting complaints in the New Zealand ED setting. The overall accuracy was reasonable in our view, with good agreement for the majority of CPC, especially when triage comments were also considered. Differences observed between different age groups, sex or ethnicity were trivial.

Accuracy for the CPC ‘general weakness/fatigue/unwell’ and ‘referral for investigation’ CPCs were poor, with three quarters or more of these having more specific CPC recorded by the treating clinician. As a result of this audit we have fed back to triage staff to avoid using more general CPCs where possible. The ‘referral for investigation’ CPC was used for many primary care referrals and for those patients brought by ambulance direct to the resuscitation room—for these patients it was expedient for clerks who entered the patients into the hospital information system to use this code. The other feedback provided was to avoid trying to label patients with a diagnosis using the CPC presumptive diagnoses can be placed in the free text comments if required.

Given that patients often present to ED with more than one presenting complaint, histories evolve over time and that the emphasis given by patients or clinicians to a particular presenting complaint when more than one symptom is present may differ, the accuracy observed in this study was considered reasonable, and compares well to that previously reported in Australia by research staff determining CPC from free text. In the study by Wagholikar and colleagues, agreement between two research assistants extrapolating CPC from free text is reported for the five most common CPC (abdominal pain, chest pain, shortness of breath, trauma and other, n=364). Using the raw data provided, it was possible to estimate the overall interobserver agreement in their study: k=0.56 (0.50, 0.617), n=364,6 which is moderate agreement. In our audit using the same five categories there was substantial agreement.7

It is clear that when using the CPC for audit of particular conditions to determine the denominator of interest, or for disease surveillance more than one CPC needs to be flagged to maximise the sensitivity of searches. This was evident for the trauma-related CPC where patients may be coded by the location of their injury or the symptoms caused; education is required such that trauma presentations should not be coded as ‘pain in body part (no injury)’. In the case of SARS-Cov-2 (or future viral respiratory pandemics) the relevant CPC may be ‘fever symptoms’; ‘shortness of breath’; ‘cough’; ‘sore throat’ or ‘exposure to communicable disease’. Simply screening for one of these will miss relevant cases.

Having a short but comprehensive list of searchable presenting complaints is a vast improvement for our department. Our previous system of free text data entry for presenting complaints made any form of clinical audit based on presenting complaints very difficult, due to the myriad of variations of spellings, misspelling, acronym use and other abbreviations that were possible for each and every presenting complaint. Ongoing audit and education will be important to improve/maintain real-time coding accuracy for CPC.

The next steps for our department are to introduce the reference sets for ED procedures and diagnosis. Once this is embedded at a national level, we will for the first time have a real picture of the demand for and the opportunity to determine the value added by ED care in New Zealand.8

Conclusion

The accuracy of SNOMED-CT CPC was reasonable in our department, although several areas for improvement were identified, most importantly the need to avoid using more general CPC when a specific CPC is available. Use of CPC for research or disease surveillance will require careful consideration of local coding practice and use of multiple CPC in searches to maximise sensitivity.

Summary

Abstract

Aim

The Systematised Nomenclature of Medicine – Clinical Terms (SNOMED-CT) coding system has been introduced to emergency departments in New Zealand, starting with the patient’s chief presenting complaint (CPC). However, it is not known how accurate CPC coding at triage is, or whether coding accuracy varies by patient characteristics. The aim of this study was to determine the accuracy of CPC coding.

Method

We compared the real-time triage recorded CPC with the presenting complaints recorded in medical notes by the treating clinician. Agreement was determined by exact CPC matches and the kappa statistic.

Results

From 1,000 consecutive presentations 852 were eligible (89 CPCs). Real-time CPC agreed with clinical notes in 514/852 (60.3%) cases. When real-time free text comments were included, agreement was 732/852 (85.9%). There were no differences by age, sex or ethnicity. Agreement for the common CPCs: ‘trauma’ (21%); ‘abdominal pain’ (11%), ‘chest pain’ 87 (10%) and ‘shortness of breath’ (8%) was substantial, k=0.66 (95% CI: 0.61, 0.70). Use of non-specific CPC such as ‘referral for investigation’ (5%) and ‘general weakness/fatigue/unwell’ (2%) was uncommon but associated with poor agreement.

Conclusion

The accuracy of real-time coding for CPC using SNOMED-CT was reasonable, except for non-specific CPCs, which should be avoided where possible.

Author Information

Peter G Jones, Associate Professor, Department of Surgery, School of Medicine, University of Auckland, Auckland; Emergency Physician, Adult Emergency Department, Auckland City Hospital, Auckland; Annette Dance, Charge Nurse, Adult Emergency Department, Auckland City Hospital, Auckland; Nikhar Shah, House Officer, Auckland City Hospital, Auckland.

Acknowledgements

The authors would like to thank Dr Mark Gardener for his assistance with data collection for this study.

Correspondence

Prof Peter Jones, Adult Emergency Department, Auckland City Hospital, ADHB, Private Bag 92024 Victoria St West, Auckland 1142.

Correspondence Email

peterj@adhb.govt.nz

Competing Interests

Prof Jones is the clinical lead at the Ministry of Health for the Shorter Stays in Emergency Departments health target and on the sector advisory group for implementation of SNOMED-CT.

1. MOH: SNOMED-CT [20/4/20]. Available from: http://www.health.govt.nz/nz-health-statistics/classification-and-terminology/new-zealand-snomed-ct-national-release-centre/snomed-ct

2. MOH: HISO 10048 Emergency Care Data Standard [20/4/20]. Available from: http://www.health.govt.nz/publication/hiso-10048-emergency-care-data-standard

3. Jones P, Hughes T, Morton T. Information technology coding systems in the emergency department: It is not the tools, it is how we use them. Emergency Medicine Australasia 2019; 31(5):700–01. doi: 10.1111/1742-6723.13378

4. Elkin PL, Brown SH, Balas A, et al. Biosurveillance evaluation of SNOMED CT's terminology (BEST Trial): coverage of chief complaints. Studies in Health Technology & Informatics 2008; 136:797–802.

5. Dean AG, Sullivan KM, Soe MM. OpenEpi: Open Source Epidemiologic Statistics for Public Health, Version. www.OpenEpi.com, updated 2013/04/06, accessed 2020/04/28.

6. Wagholikar AS, Lawley MJ, Hansen DP, et al. Identifying symptom groups from Emergency Department presenting complaint free text using SNOMED CT. AMIA Annu Symp Proc 2011; 2011:1446–53 [published Online First: 2011/10/22].

7. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33(1):159–74 [published Online First: 1977/03/01].

8. Hughes T, Higginson I, Mann C. Tariffs in emergency care. British journal of hospital medicine (London, England : 2005) 2014; 75(11):631–6. doi: 10.12968/hmed.2014.75.11.631 [published Online First: 2014/11/11].

Contact diana@nzma.org.nz
for the PDF of this article

View Article PDF

The Systematised Nomenclature of Medicine—Clinical Terms (SNOMED-CT) coding system has been adopted by the Ministry of Health in Aotearoa New Zealand to replace other medical coding systems.1 The MOH has mandated that Emergency Departments (ED) in New Zealand must report all visits using SNOMED-CT codes for Chief Presenting Complaint (CPC), Procedures and Diagnoses to the National Non-admitted Patient Collection (NNPAC) by July 2021.2 It is hoped that this will correct a large gap in the data about acute presentations which is not captured currently.3 This is especially important as there are implications for disease surveillance, new trends in recreational drug toxicity and especially relevant in the current context of SARS COV-2: early identification of possible pandemics.4 Auckland City Hospital (ACH) emergency department (ED) introduced coding CPC with SNOMED-CT in March 2019 as an ‘early adopter’ pilot site for the MOH national rollout and are currently in the process of introducing procedure and diagnosis codes. In the MOH reference set for emergency care there were initially 144 possible presenting complaints, although this has now been expanded to 152 after feedback from stakeholders.2 We currently do not know how accurate coding of CPC using SNOMED-CT is in a New Zealand ED context. The aim of the current study is to determine how accurate coding CPC with SNOMED-CT was by triage staff compared to clinician’s notes.

Method

This was a retrospective chart review.

Setting

ACH is an urban tertiary academic centre that has an annual census of approximately 76,000 patients 15 years or older.

Case selection

One thousand consecutive presentations to ED were selected, between the dates 19 July 2019 and 23 July 2019. The start date was arbitrarily chosen to give staff sufficient time (four months) to get used to using SNOMED-CT.

Recording of presenting complaints

Triage CPC

CPC was recorded in real-time as soon as possible on patient arrival at ED. This is done by trained triage nurses for all ED presentations, except those ambulance patients requiring immediate resuscitation, or primary care referrals to inpatient teams. Clerical staff record the CPC for patients referred by their primary care provider or patients requiring immediate resuscitation (while clinical staff provide immediate care). A single CPC is recorded from the MOH reference set into the electronic patient management system (Triage CPC). There is provision within the system to add free text comments about the CPC or mention other presenting complaints if there are multiple (Triage Comments).

Auditor CPC

Treating clinicians typically begin their clinical notes stating the presenting complaint and history of presenting complaints. These notes were audited, and the auditor recorded what the clinician stated as the presenting complaints (Auditor CPC). The first presenting complaint mentioned in the clinical notes was recorded as the first Auditor CPC. Up to five Auditor CPC were recorded if there were multiple presenting complaints. The auditors were a senior emergency nurse and a trained final year medical student. The lead author reviewed the first few cases with the auditors, until satisfied that data collection was reliable. No formal check of data accuracy was performed. Where there was uncertainty about the CPC in a particular case the auditors discussed with the lead author and resolved by consensus. Only the date/time of presentation, visit number and NHI were provided to the auditors. Patient records were then accessed by the auditors using the clinical records systems Concerto (Orion Health, Auckland, New Zealand) and 3M™ ChartView™ (3M Company, Salt Lake City, Utah, US) using a password-protected computer based within the ED. Electronic clinical notes for all patients were reviewed independently by an auditor blinded to the real-time CPC. The auditor recorded CPC based on the first treating clinician’s clinical notes onto a purpose-built electronic data collection form in Microsoft Excel® (Microsoft Corporation, Redmond, Washington, US). The first treating clinician was either an independent nurse practitioner or doctor. When there was more than one presenting complaint recorded by the first treating clinician the auditor could record up to five presenting complaints.

Outcomes

The primary outcome was the agreement between triage CPC and the first recorded auditor CPC. Secondary analyses were differences between patient groups, and overall agreement between triage CPC plus triage comments and all recorded auditor CPCs.

Data collection

The following data was retrieved from the hospital data warehouse: event number, NHI, date/time of presentation; age, sex, ethnicity, CPC and free text triage comments fields. This was received by one author in the form of a .csv file and stored in a password-protected computer in a locked office in the ED.

Data handling

Once all data was collected the two data sources were merged into one file for analysis by one author.

Data analysis

Auditor’s CPC was compared directly to the triage CPC and triage comments. The proportion exact matches between triage CPC or triage comments and auditor CPC was recorded as n, % 95% CI. The 95% CI was calculated using Graphpad QuickCalcs (http://www.graphpad.com/quickcalcs/confInterval1/), San Diego, CA, US. Tests for differences between groups were done using Chi2, using Open Epi.5 A two-tailed p<0.05 was considered statistically significant.

Ethics and institutional approval

As an audit of the data quality undertaken by staff in the department where this data was collected and no identifying data would be reported, this study did not meet the threshold to require ethical approval. Institutional approval was obtained from the ADHB research office (A+8678).

Results

The case selection process is shown in Figure 1. Of 1,000 consecutive presentations, 852 were seen in ED and had notes available for the analysis (85%). The demographic details for the cases are shown in Table 1. There were 89 different presenting complaints recorded in real-time of 144 possible (62%) from the original MOH reference set. The most common CPC were related to trauma, with all ‘injury’ related CPC 175/852 (20.5%), ‘abdominal pain’ 94 (11.0%), ‘chest pain’ 87 (10.2%), ‘shortness of breath’ 66 (7.7%), ‘referral for investigation’ 43 (5.0)%, and ‘fever symptoms’ 30 (3.5%).

Figure 1: Case selection.

Table 1: Demographics of sample population.

SD = standard deviation, CI = confidence interval.

Figure 2 shows the auditor’s first CPC agreement with the triage CPC, which was 514/852, 60.3% (95% CI 57–64%). When triage free text comments were also considered, the agreement between the auditor’s CPCs derived from the first clinician’s notes and the triage presenting complaints was 732/852, 85.9% (95% CI 83–88%) of cases (Figure 3).

Figure 2: Agreement between auditor and real-time SNOMED-CT CPC

SNOMED-CT= Systematised Nomenclature of Medicine – Clinical Terms, CPC = chief presenting complaint.

Figure 3: Agreement between auditor and real-Time SNOMED-CT CPC + triage comments

SNOMED-CT= Systematised Nomenclature of Medicine – Clinical Terms, CPC = chief presenting complaint.

There was disagreement between either real-time CPC and triage comment and any auditor selected for 186 patients across 51 CPC.

For the common clinical complaints related to trauma, ‘abdominal pain’, ‘chest pain’ and ‘shortness of breath’, there was substantial agreement between the auditor’s primary CPC and the real-time CPC, 661/852=77.5%, k=0.66 (0.61, 0.70), as shown in Table 2.

Table 2: Agreement for common CPC.

CPC = chief presenting complaint.

The worst agreement was for CPC that were non-specific. Agreement for ‘referral for investigation’ was (6/43, 14%). For this CPC, the auditors recorded an alternative more specific CPC in 37/43 (86%) cases (Figure 4). When triage comments were included in the comparisons, the level of agreement increased to 24/43 (56%).

Figure 4: Clinician CPC compared to triage CPC ‘referral for investigation’.

Green indicates agreement between auditor and real-time CPC, red indicates auditor CPC when real-time CPC was ‘referral for investigation’. CPC = chief presenting complaint.

Similarly, for ‘fever symptoms’ there was only 17/29, 57% agreement on CPC which improved to 24/29 (80%) with triage comments included. For ‘general weakness/fatigue/unwell’ there was no agreement on CPC, 0/16 (0%). This improved to 8/16 (50%) when triage comments were included.

There were occasional disagreements for many CPC: For example, eight patients with injuries who were coded as having ‘pain in body part (no recent injury)’ and three patients with ‘fall(s) no significant injury’ when injuries were present. Similarly, 11 patients had injury-related CPC who were coded as pain without injury by the auditors. Also, many CPC overlap, such as ‘collapse/syncope’ and ‘fall(s) no significant injury’ (four cases coded either way in real-time or by auditor); for acute toxicological presentations CPC may also overlap, for example ‘alcohol intoxication’, ‘drug intoxication’, ‘overdose of drug’ or ‘ingestion of harmful substance’ (seven cases with variable coding either in real-time or by auditors).

Table 3 shows the agreement based on demographic characteristics. There was no difference in the rate of agreement on CPC by age category or sex. There was a difference for different ethnicities which bordered statistical significance. The most agreement was for Pacific peoples, 66% (95% CI 57–74) and least for Māori, 51% (95%CI 41-61). This difference was not evident when triage comments were also considered int the agreement calculation (Table 3).

Table 3: Agreement on CPC by demographic variable.

Discussion

This is the first audit of the accuracy of CPC coding using SNOMED-CT compared to first treating clinician documentation of presenting complaints in the New Zealand ED setting. The overall accuracy was reasonable in our view, with good agreement for the majority of CPC, especially when triage comments were also considered. Differences observed between different age groups, sex or ethnicity were trivial.

Accuracy for the CPC ‘general weakness/fatigue/unwell’ and ‘referral for investigation’ CPCs were poor, with three quarters or more of these having more specific CPC recorded by the treating clinician. As a result of this audit we have fed back to triage staff to avoid using more general CPCs where possible. The ‘referral for investigation’ CPC was used for many primary care referrals and for those patients brought by ambulance direct to the resuscitation room—for these patients it was expedient for clerks who entered the patients into the hospital information system to use this code. The other feedback provided was to avoid trying to label patients with a diagnosis using the CPC presumptive diagnoses can be placed in the free text comments if required.

Given that patients often present to ED with more than one presenting complaint, histories evolve over time and that the emphasis given by patients or clinicians to a particular presenting complaint when more than one symptom is present may differ, the accuracy observed in this study was considered reasonable, and compares well to that previously reported in Australia by research staff determining CPC from free text. In the study by Wagholikar and colleagues, agreement between two research assistants extrapolating CPC from free text is reported for the five most common CPC (abdominal pain, chest pain, shortness of breath, trauma and other, n=364). Using the raw data provided, it was possible to estimate the overall interobserver agreement in their study: k=0.56 (0.50, 0.617), n=364,6 which is moderate agreement. In our audit using the same five categories there was substantial agreement.7

It is clear that when using the CPC for audit of particular conditions to determine the denominator of interest, or for disease surveillance more than one CPC needs to be flagged to maximise the sensitivity of searches. This was evident for the trauma-related CPC where patients may be coded by the location of their injury or the symptoms caused; education is required such that trauma presentations should not be coded as ‘pain in body part (no injury)’. In the case of SARS-Cov-2 (or future viral respiratory pandemics) the relevant CPC may be ‘fever symptoms’; ‘shortness of breath’; ‘cough’; ‘sore throat’ or ‘exposure to communicable disease’. Simply screening for one of these will miss relevant cases.

Having a short but comprehensive list of searchable presenting complaints is a vast improvement for our department. Our previous system of free text data entry for presenting complaints made any form of clinical audit based on presenting complaints very difficult, due to the myriad of variations of spellings, misspelling, acronym use and other abbreviations that were possible for each and every presenting complaint. Ongoing audit and education will be important to improve/maintain real-time coding accuracy for CPC.

The next steps for our department are to introduce the reference sets for ED procedures and diagnosis. Once this is embedded at a national level, we will for the first time have a real picture of the demand for and the opportunity to determine the value added by ED care in New Zealand.8

Conclusion

The accuracy of SNOMED-CT CPC was reasonable in our department, although several areas for improvement were identified, most importantly the need to avoid using more general CPC when a specific CPC is available. Use of CPC for research or disease surveillance will require careful consideration of local coding practice and use of multiple CPC in searches to maximise sensitivity.

Summary

Abstract

Aim

The Systematised Nomenclature of Medicine – Clinical Terms (SNOMED-CT) coding system has been introduced to emergency departments in New Zealand, starting with the patient’s chief presenting complaint (CPC). However, it is not known how accurate CPC coding at triage is, or whether coding accuracy varies by patient characteristics. The aim of this study was to determine the accuracy of CPC coding.

Method

We compared the real-time triage recorded CPC with the presenting complaints recorded in medical notes by the treating clinician. Agreement was determined by exact CPC matches and the kappa statistic.

Results

From 1,000 consecutive presentations 852 were eligible (89 CPCs). Real-time CPC agreed with clinical notes in 514/852 (60.3%) cases. When real-time free text comments were included, agreement was 732/852 (85.9%). There were no differences by age, sex or ethnicity. Agreement for the common CPCs: ‘trauma’ (21%); ‘abdominal pain’ (11%), ‘chest pain’ 87 (10%) and ‘shortness of breath’ (8%) was substantial, k=0.66 (95% CI: 0.61, 0.70). Use of non-specific CPC such as ‘referral for investigation’ (5%) and ‘general weakness/fatigue/unwell’ (2%) was uncommon but associated with poor agreement.

Conclusion

The accuracy of real-time coding for CPC using SNOMED-CT was reasonable, except for non-specific CPCs, which should be avoided where possible.

Author Information

Peter G Jones, Associate Professor, Department of Surgery, School of Medicine, University of Auckland, Auckland; Emergency Physician, Adult Emergency Department, Auckland City Hospital, Auckland; Annette Dance, Charge Nurse, Adult Emergency Department, Auckland City Hospital, Auckland; Nikhar Shah, House Officer, Auckland City Hospital, Auckland.

Acknowledgements

The authors would like to thank Dr Mark Gardener for his assistance with data collection for this study.

Correspondence

Prof Peter Jones, Adult Emergency Department, Auckland City Hospital, ADHB, Private Bag 92024 Victoria St West, Auckland 1142.

Correspondence Email

peterj@adhb.govt.nz

Competing Interests

Prof Jones is the clinical lead at the Ministry of Health for the Shorter Stays in Emergency Departments health target and on the sector advisory group for implementation of SNOMED-CT.

1. MOH: SNOMED-CT [20/4/20]. Available from: http://www.health.govt.nz/nz-health-statistics/classification-and-terminology/new-zealand-snomed-ct-national-release-centre/snomed-ct

2. MOH: HISO 10048 Emergency Care Data Standard [20/4/20]. Available from: http://www.health.govt.nz/publication/hiso-10048-emergency-care-data-standard

3. Jones P, Hughes T, Morton T. Information technology coding systems in the emergency department: It is not the tools, it is how we use them. Emergency Medicine Australasia 2019; 31(5):700–01. doi: 10.1111/1742-6723.13378

4. Elkin PL, Brown SH, Balas A, et al. Biosurveillance evaluation of SNOMED CT's terminology (BEST Trial): coverage of chief complaints. Studies in Health Technology & Informatics 2008; 136:797–802.

5. Dean AG, Sullivan KM, Soe MM. OpenEpi: Open Source Epidemiologic Statistics for Public Health, Version. www.OpenEpi.com, updated 2013/04/06, accessed 2020/04/28.

6. Wagholikar AS, Lawley MJ, Hansen DP, et al. Identifying symptom groups from Emergency Department presenting complaint free text using SNOMED CT. AMIA Annu Symp Proc 2011; 2011:1446–53 [published Online First: 2011/10/22].

7. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33(1):159–74 [published Online First: 1977/03/01].

8. Hughes T, Higginson I, Mann C. Tariffs in emergency care. British journal of hospital medicine (London, England : 2005) 2014; 75(11):631–6. doi: 10.12968/hmed.2014.75.11.631 [published Online First: 2014/11/11].

Contact diana@nzma.org.nz
for the PDF of this article

Subscriber Content

The full contents of this pages only available to subscribers.

LOGINSUBSCRIBE