  |
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Journal of the New Zealand Medical Association,
02-April-2004, Vol 117 No 1191
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
What makes a good performance indicator? Devising primary
care performance indicators for New Zealand
Peter Crampton, Roshan Perera, Sue Crengle, Antony Dowell,
Philippa Howden-Chapman, Robin Kearns, Tom Love, Bev Sibthorpe, and Margaret
Southwick
As is true of other disciplines, primary care in New Zealand
is not exempt from the search for the ‘holy grail of quality’.
Indeed, there is currently significant activity and debate in New Zealand
related to clinical governance and quality improvement in primary care (for
example, Royal New Zealand College of General
Practitioners,1 First
Health,2 and Healthcare
Aotearoa).3
The Primary Health Care
Strategy4 charts a course for primary
care where, increasingly, primary care and public health strategies are
coordinated and inter-meshed, with the overall objective of improving population
health and reducing health inequalities (Figure 1).
Figure 1. The interacting roles of primary care and
public health
 New approaches to measuring performance may be required to
serve this new strategy, which focus on the effectiveness of multidisciplinary
teams working under the aegis of Primary Health Organisations (PHOs). These new
approaches, in turn, raise basic questions that must be answered. For example,
the increasing complexity of primary care suggests that performance indicators
are now required to reflect a number of different perspectives (Table 1), but
clearly not all perspectives can be given equal weight.
Decisions need to be made regarding the balance of
perspectives. Similarly, decisions need to be made regarding the number of
indicators that can reasonably be expected to be used. The new contract for
general practitioners in the UK specifies around 140 quality indicators—76
for clinical care, 56 in organisational areas, 4 assessing patients’
experiences, plus indicators for additional
services.5,6 Are 140 indicators too many or too
few? How much resource and effort should be invested in performance monitoring
by providers?
Given that the New Zealand Ministry of Health proposes that
performance indicators are required for PHO contracting, then these, and other
key decisions regarding performance indicators, require wide debate within the
primary care sector—with input from consumers and a wide variety of
providers and stakeholders; including, general practitioners, Maori and Pacific
providers, and primary care professionals working in rural and urban settings.
This paper discusses a range of preliminary questions about performance
indicators, and introduces a collaborative research effort that aims to underpin
and inform a New Zealand approach to performance indicators for primary
care.
What are primary care performance indicators?Terminology is varied and
inconsistently used. Nevertheless, standard frameworks for assessing the
performance of healthcare systems, such as those of
Maxwell,7 Murray and
Frenk,8 the
OECD,9 and the World Health
Organisation,10 all include the notion of
effectiveness—do health services result in improved health?
This is the central question underlying performance
measurement. Performance indicators can be thought of as fitting within an
overarching quality framework that seeks to improve patient care and population
health outcomes, and reduce health inequalities (Figure 2).
Figure 2. Quality, outcomes and performance
 Performance indicators have been defined as measurable
elements of practice performance for which there is evidence or consensus that
they can be used to assess the quality, and hence change in quality, of care
provided.11 Performance indicators are an
important component of (and adjunct) to most approaches to quality
management,12 and can relate to clinical,
population health, financial and organisational performance. Performance
indicators may contribute to quality improvement programmes in two
ways.
First, they can promote wider use of evidence-based
interventions; for example, in the secondary prevention of coronary heart
disease.13 Second, they may assist in the
evaluation of quality improvement programmes run at a practice or PHO level. It
seems reasonable, therefore, that primary care performance indicators should be
limited to those factors that can be directly influenced by primary care
organisations, rather than attempt to assess the performance of the whole health
system plus broader social
programmes.11,13–15 If the latter were
the case, primary care could be unfairly ‘judged’.
The overlapping concept of ‘outcomes’ is related
to performance indicators, but tends to be a broader generic term used to refer
to the results of care of individuals or
populations,16,17 which are frequently long
delayed.11 Two principal domains of outcome are
health status (for example, diabetes related mortality) and user evaluation (for
example, patient satisfaction surveys). Health status outcomes may be considered
a simple count of health events that may be contingent upon a wide variety of
factors.14,16
In practice, most outcome indicators are also measures of
need for healthcare services, and so serve a dual purpose. Outcomes research in
primary care is generally either concerned with the impact of healthcare on
disease processes, or the impact of healthcare on population health. With the
former, outcomes research requires an understanding of the natural history of
the condition, defined objectives of care against which the outcomes are
measured; understanding of the full range of inputs that may influence the
outcome, the hypothesised relationship between the inputs and outcomes, and a
suitable research design to study outcomes.17
p.5
In the case of population health, while it is often not
possible to make such clear connections between inputs and health outcomes, the
same principles apply.
Current use of performance indicators in primary care—too much or too little?Measuring performance of healthcare
providers is a universal contemporary theme in health
systems.18 Despite this, New Zealand lacks a
nationally agreed approach to primary care performance indicators. Currently, it
is likely there is a mixture of under- and over-use of performance indicators,
little national consistency in their application, lack of clarity about any
underlying theoretical development, and lack of local validation.
Furthermore, it is not clear how existing international
performance indicator frameworks might be suitable for New Zealand. The presence
of an indigenous population, both fee-for-service and capitation payment
mechanisms in general practice, and particular health-related organisations
(such as PHARMAC and ACC) provide a social and institutional setting different
from many other countries.
Internationally, the importance accorded the development of
performance and outcome indicators is underscored by the large amount of effort
in the UK and the US in developing respectively the national performance
framework,19,20 and
HEDIS21 indicators for healthcare plans. In
light of this situation, our purpose is to raise (for discussion) the questions
of ‘what’, ‘when’, and ‘how’, with respect
to performance monitoring of primary care in New Zealand.
Who are the main users of primary care performance indicators?The validity, reliability, and
acceptability of performance indicators depend to some extent on their intended
uses.15 That is, the design of performance
indicators is intimately related to their purpose. Four key uses for performance
indicators for primary care in New Zealand are listed below. It is probably
neither possible nor desirable that these four purposes be achieved by a single
set of performance indicators. Indeed, the aims of ongoing quality enhancement
and incentive funding may be in conflict, if funding incentives result in less
emphasis on desirable activities, in favour of those which are financially
rewarded.
First, the government, communities and Maori iwi require
tools for assessing the effectiveness and quality of care offered by different
types of primary care organisations. Specifically, performance indicators may be
helpful for assisting the public understand the healthcare system, and of
conveying (to communities) the differences in the ways that key elements of care
are provided in different populations and different localities.
Second, development of performance indicators will assist
primary care organisations and healthcare providers in assessing the
effectiveness of their own activities—in improving the health of their
population and reducing inequalities. Already much time and effort is expended
in collecting data in primary care, yet there is little consensus on which data
should be collected. Focusing data collection on a limited set of meaningful
performance indicators could result in a reduction in workload in some cases,
and increased data collection in others.
Third, performance indicators are used in a policy and
funding context, where there is increasing emphasis on accountability of primary
care providers to their communities and to funders, and on information
collection and sharing.4 p.24
The Primary Health Care Strategy
states: ‘More research and evaluation is required to resolve
issues—such as the degree of variation in service provision, the most
appropriate ways to target limited resources, the most efficient ways to provide
care, and what services are best in different
circumstances’.4 p.26 Clearly, therefore,
performance indicators are a key tool for assessing the impact of organisational
changes, and the performance of provider organisations.
Fourth, performance indicators are useful for research.
Given the increasing organisational heterogeneity in New Zealand's primary care
sector,22,23 there is an urgent need for
studies comparing the performance of care in different demographic, cultural,
epidemiological, and organisational contexts. If primary care
‘models’ are to be compared meaningfully, then validated performance
measures are required.
What should primary care performance indicators measure?We contend that a theoretical
foundation is absolutely necessary in the formulation of performance indicators.
A theoretical framework provides answers to questions such as: why do we need
indicators? What should they measure? How should they be constructed? In the
absence of clear answers to these questions, performance indicators can be used
variously as a means of unjustifiably punishing primary care providers, or as a
tool for shifting funding in response to political or lobby group
pressures.
Several theoretical approaches to primary care performance
indicators have been advanced. (Also, it is evident from the literature that
indicators research is sometimes carried out without any explicit theoretical
elaboration.) Four theoretical approaches are given here as examples.
First, Campbell et al16
proposed a framework for assessing quality of primary care that builds upon
Donabedian’s classic healthcare quality triad related to structure,
process, and outcome.24 They assess
quality-of-care on two key dimensions: access and effectiveness—do users
get the care they need, and is the care effective when they get it?
Effectiveness, in turn, has two dimensions: clinical effectiveness and
interpersonal relationships. Structure refers to the organisational factors that
determine how care is provided, and is divided into two domains—physical
characteristics (eg, opening hours, user charges, booking system, waiting times)
and staff characteristics (eg, teamwork, language skills). Process-of-care
includes technical interventions (eg, prescribing and screening tests), and
interpersonal interactions between staff and users. Process-of-care may be
further classified into preventive care, care for chronic disorders, and care
for acute illness.
At the population or societal level, Campbell et al identify
four outcomes of importance: health status, user evaluation (such as
satisfaction and enablement),25 cost of
providing the service (which, when combined with outcomes, brings in the notion
of efficiency), and equity (fairness). Furthermore, Campbell et al consider
equity to be a sub-component of access relevant to structure and process; they
define it as the extent to which all individuals in a population access the care
they need in a timely way (as opposed to equality-of-access, which implies equal
access to services, irrespective of need).
Second, Van Norren et al26
proposed an action-oriented framework that focuses on intermediate variables
that directly affect health status, and
can also be influenced by primary care interventions. The key feature of their
framework is a set of intermediate variables (such as breastfeeding) that link
the social and biological systems.
The intermediate variables are grouped into five categories,
and have the following characteristics—they are both behavioural and
biological in kind (ensuring a direct biological effect on health while also
ensuring that they can be influenced by interventions or policy); they have a
particular focus of effect on a biological risk factor; they exert a direct
effect on the factor in question; and they are susceptible to primary care
interventions (preferably an item of priority in primary care programmes). In
this framework, as with the Campbell et al framework, it is not necessary to
measure the precise magnitude of the effects of intermediate variables on
morbidity or mortality.
Third, McColl et al13
described an evidence-based approach for developing performance indicators for
primary care groups. As with Campbell et al, they explicitly identified two key
domains of performance broadly related to access and effectiveness. The focus of
their research was effectiveness indicators, and involved three key
steps—they identified interventions of proven efficacy for which primary
care has a key responsibility; they estimated the number of preventable deaths
or events in a primary care group or locality of 100,000 people if all those
eligible were receiving the intervention; and they compared the potential
indicators they derived with the indicators proposed by the UK government.
Furthermore, they demonstrated considerable differences between their
evidence-based indicators and those in the national framework. While the paper
discusses numerous methodological problems with their approach, their proposed
evidence-based indicators provide a useful set to consider in New Zealand.
Fourth, the Australian National Health Performance Committee
developed a national performance measurement framework for the health system
(adapted from the Canadian Institute of Health Information framework as part of
the Canadian Roadmap Initiative
[www.cihi.ca]).27 The Australian framework
consists of three tiers: health-status and outcomes, determinants of health, and
health system performance. The Committee grouped health system performance into
nine dimensions, with any single indicator providing information across one or
more of the nine dimensions. The nine dimensions that are defined and discussed
in the report relate to the degree to which health services are: effective,
appropriate, efficient, responsive, accessible, safe, continuous, capable, and
sustainable. Two over-arching dimensions are applied to the three tiers: quality
and equity. Advantages of the Australian framework include its practical
applicability, its comparability with the Canadian framework, and its capacity
to provide data for comparative studies with New Zealand.
Despite the diversity of theoretical approaches to primary
care performance indicators exemplified above, there is very little research
that evaluates different approaches thereby enabling links to be made between
the different approaches to performance-monitoring, and improved quality of
services and health outcomes.
How should performance indicators be constructed?Examination of the literature
suggests a number of important criteria for primary care performance indicators
(Table 2).11,13,14,16,26–28 Clearly, not
all these characteristics can possibly apply to every indicator. Arguably the
only absolutely essential criterion is that the indicator be attributable to
healthcare; the weight given to the various criteria is at the discretion of
those devising indicator sets and there is very little evidence to guide
weighting decisions. Nevertheless, having the criteria for an
‘ideal’ indicator identified and listed, allows ongoing critique of
indicators, and increases awareness of the limitations of indicators. It is also
important to identify characteristics for sets of performance
indicators—characteristics that may not apply to any given indicator, but
that must apply to a set (Table
3).11,13–15,26-29
How should we monitor the performance of performance indicators?Little research has been carried
out into the effectiveness of quality programmes, let alone the use of
performance indicators.30 Nevertheless,
accountability and quality concerns remain at the forefront of the policy
agenda, and so approaches to performance monitoring must be devised that are as
robust as possible (given the lack of supporting evidence). More importantly
perhaps, the implementation of indicators should be monitored and researched so
that the value of performance indicators themselves can be judged. Does the use
of performance indicators contribute to health gains and reductions in
inequalities? Is the use of performance indicators leading to positive change in
the organisation of services? In other words, to state the obvious, findings
from using indicators should be the beginning not the end of a process of
enquiry.15
What are some barriers to implementing performance indicators?There is a range of potential
barriers to implementing performance indicators. Perhaps the most important
barrier arises when primary care providers perceive that indicators are not
relevant in their context or lack face validity. This barrier is best addressed
by collaborative development of indicators that involves provider and community
perspectives, and clear communication of the rationale for indicators. The
perception amongst primary care organisations that indicators may be used in a
punitive way may slow their acceptance and use. Similarly, high cost of data
collection (for the Ministry of Health, Primary Health Organisations, or primary
care practices) is likely to inhibit uptake and use of indicators.
Where are we heading with primary care performance indicators?Efforts are underway by the
Ministry of Health to establish an initial set of performance indicators that
can be used in the context of PHO development. For example, the Ministry of
Health commissioned an independent group to carry out an iterative, Delphi-like
process to achieve consensus on a small set of clinical performance indicators
that have face-validity, and where data about performance may be readily
available.31,32 A small number of
population-oriented indicators are also being explored. The Ministry has formed
a technical advisory group, consisting of a range of primary care stakeholders,
to assist in the implementation of this interim set of indicators (personal
communication, Jon Foley, Ministry of Health, November 2003).
We support the introduction of a more systematic approach to
performance indicators in primary care and are heartened that existing
literature, and The Primary Health Care
Strategy, support a positive continuous improvement approach rather than
a punitive or sanctioning model of performance monitoring. We believe that
performance monitoring should be carried out in a way that actively contributes
to the development of population-based primary care in New Zealand while
remaining in the scope of the primary care budget.
ConclusionsAlthough a variety of theories
contribute to our understanding of what constitute good primary care performance
indicators, there is (currently) little consensus on which data should be
collected by primary care organisations. The Ministry of Health is leading a
project, in collaboration with primary care stakeholders and researchers, aimed
at developing a standardised approach to selecting and evaluating primary care
performance indicators that draws on international experience and also takes
into account New Zealand's unique primary care context. However, more research,
which evaluates the contribution of performance indicators to improved quality
of primary care and health outcomes, is needed.
Acknowledgements:
We thank Jon Foley (Senior Policy Analyst,
Ministry of Health) and two anonymous reviewers for comments on an earlier draft
of this paper.
Correspondence:
Peter Crampton, Department of Public Health, Wellington School of Medicine and
Health Sciences, PO Box 7343, Wellington. Fax: (04) 389 5319; email: cramptonp@wnmeds.ac.nz
References:
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| | Current
issue | Search journal |
Archived issues | Classifieds
| Hotline (free ads) Subscribe | Contribute | Advertise | Contact Us | Copyright | Other Journals |
|