	Table of contents

	Current issue

	Search journal

	Archived issues

	NZMJ Obituaries 1887-2008

	Classifieds

	Hotline (free ads)

	How to subscribe

	How to contribute

	How to advertise

	Contact Us

	Copyright

	Other journals

Journal of the New Zealand Medical Association, 07-August-2009, Vol 122 No 1300

Patient satisfaction in New Zealand

Gerard Zwier

Abstract

Aims To determine whether the Patient Survey Guidelines developed for district health board (DHBs) in 2000 are being adhered to and what the results of the survey can tell us about patient satisfaction in New Zealand.

Method Nationwide patient survey data obtained from the Ministry of Health under the Official Information Act was examined to determine the validity and reliability of the survey instrument. Data was then analysed to ascertain whether patients are more or less satisfied than in previous years and what factors influence patient satisfaction.

Results To determine the reliability of the instrument, the “Cronbach alpha” statistic was calculated. The patient satisfaction instrument demonstrated high levels of reliability for patient satisfaction measures of “communication”, a “personal approach” and “organising patient care”. Assessment of convergent validity showed that the highest correlations were found between items that measured closely related aspects of patient care and, conversely, discriminant validity was demonstrated by the very low correlations between items that measured unrelated aspects of patient care.

Conclusion The current dataset of some 229,000 inpatient and 254,000 outpatient records from 21 New Zealand DHBs has the potential to generate extremely valuable information that can be employed to increase patient satisfaction throughout New Zealand. However, while high levels of reliability and validity were demonstrated, the majority of DHBs do not implement the survey as required by Patient Survey Guidelines which it developed in 2000.

It is exactly 8 years ago since the Sector Accountability & Funding Directorate, then known as Crown Company Monitoring & Advisory Unit or CCMAU), published the “Patient Satisfaction Survey Guidelines 2000”.1 This report, which embodied the collaborative effort of several Ministry of Health staff and a team of public hospital Quality Managers and Customer Services personnel, described the newly proposed Inpatient and Outpatient questionnaires and explained in great detail the “Best Practice” methodology that should be used by all New Zealand public hospitals so that they would be able to monitor patient satisfaction accurately and reliably.

In an accompanying letter, the then Hon Minister of Health, Ms Annette King, said that (these guidelines) would...

Improve the statistical robustness of survey results and the consistency with which district health boards (DHBs) can apply them.
Expand the base of the patient populations being surveyed.
Focus the questions asked on the key determinants of patient satisfaction, from the patients’ perspective.

The question that needs to be asked now is, has the implementation of the new survey gone to plan? Has the statistical robustness of the statistics and usefulness of the results been demonstrated? More to the point, and keeping in mind the issues raised in previous publications (Zwier & Clarke2,3), are DHBs now in a position to use the data to better understand and/or increase patient satisfaction?

Method

The dataset covering the last 8 years was analysed on the basis of the survey results submitted by each DHB to the Directorate. This database, which presently contains 229,000 inpatient and 254,000 outpatient records from 21 New Zealand DHBs, incorporates patient satisfaction ratings on 17 inpatient and 15 outpatient items respectively. It presents New Zealand with a treasure trove of information, both from the perspective of statistical analyses and from the potential use that can be made of it to further improve our patient satisfaction ratings.

To permit analyses of specific aspects of care, the inpatient questionnaire asks questions about patient perceptions of the Emergency Department, the availability of staff, the manner in which they were treated by staff (did they receive enough information, did the staff treat them with dignity and respect?), their opinion of the hospital’s facilities (safety & security, cleanliness, food), discharge procedures and the adequacy of communication between different departments involved in their care.

The outpatient questionnaire covers the usual topics such as the patients’ perceptions of the appointment system, the manner in which they were treated by staff (did they receive enough information, did staff ask permission to treat the patient?), their opinion of the clinic’s facilities (e.g. cleanliness), the adequacy of communication between different departments involved in their care, and their satisfaction with the organisation of their care with other service providers.

The present overview is divided into two separate sections:

An assessment of the reliability and validity of the questionnaire, and
An analysis of the results of the survey data using ESPRI software.

This overview is concluded with a recommendation regarding future requirements.

Results

How reliable and valid is the data?

A preliminary investigation into the reliability and validity of the present survey was carried out. Because if it were found to be severely lacking, a lot of effort would have been made to no avail. The public could rightly accuse the government of wasting good public hospital money.

Is the prescribed method implemented?

When the question was posed whether the DHBs are surveying their patient population using the method prescribed in the Patient Survey Guidelines, it became clear that some do but most don’t:

Table 1 shows that only Auckland, Bay of Plenty, Canterbury, and Taranaki consistently achieve the minimum number of required questionnaires returned by patients. Five other DHBs (i.e. Capital & Coast, Counties Manukau, Hutt Valley, Nelson Marlborough, and Waikato) achieve this some of the time. A number of DHBs regularly miss out on achieving the required sample size while West Coast submits less than a dozen questionnaires each quarter and might as well not participate.

Even when DHBs are sending out a sufficiently large enough number of questionnaires, the response rate is in most cases quite low. Excluding such obvious errors as made by Hutt DHB which in the first quarter this year recorded sending out 600 questionnaires and receiving 609 responses, the average response rate among these DHBs is around 35%.

Table 1 Over and under target numbers and response rate by district health board (DHB)

DHB	2008 quarter 1		2008 quarter 2		2008 quarter 3		2008 quarter 4
DHB	Over or under	Response rate %	Over or under	Response rate %	Over or under	Response rate %	Over or under	Response rate %
Auckland	90	34%	156	37%	153	37%	118	35%
Bay of Plenty	511	37%	390	29%	496	29%	411	26%
Canterbury	24	43%	23	43%	4	41%	22	44%
Capital & Coast	86	33%	565	37%	232	39%	-171	25%
Counties Manukau	9	26%	-73	20%	66	30%	-29	23%
Hawke's Bay	-132	35%	-158	33%	-146	34%	-205	31%
Hutt Valley	278	107%	150	85%	-137	25%	-113	37%
Lakes	-34	33%	-32	33%	-57	31%	-82	32%
Mid Central	-116	44%	-74	51%	-105	46%	-74	51%
Nelson Marlborough	37	38%	27	46%	-43	37%	-4	42%
Northland	-44	36%	-2	39%	-59	33%	-26	36%
Otago	-70	49%			-86	46%	-109	42%
South Canterbury	-97	45%	-97	42%	-123	40%	-110	39%
Southland	-156	35%	-134	39%	-20	38%	-125	40%
Tairawhiti	-154	24%	-194	25%	-163	26%	-193	24%
Taranaki	18	35%	23	36%	41	39%	38	37%
Waikato	9	35%	36	40%	223	38%	-19	38%
Wairarapa	-101	43%	-117	39%	-144	33%	-107	35%
Waitemata	-142	32%	-156	34%	-117	35%	-170	32%
West Coast
Whanganui	-50	41%	-51	39%	-71	34%	-60	37%

In addition, various DHBs have amended the stipulated questionnaire by changing the sequence or adding in new questions such that comparability of results is lost. Hutt Valley DHB, for example, rearranges the entire sequence of items and intersperses some 60 additional questions to the prescribed 17 inpatient questions.

Furthermore, the bias in the sample caused by self-selection (older and European patients are more likely to respond than are younger and Māori/Pacific patients) has lead in virtually all cases to a lack of representativeness of the resulting sample of patients: older and European patients are over-represented and younger and Māori/Pacific patients are under-represented.

Yet disappointingly, the agency charged with monitoring the implementation, i.e. The Sector Accountability & Funding Directorate of the Ministry of Health which is responsible for funding, monitoring and ensuring the sector is compliant with accountability expectations, has taken no action to rectify these shortcomings.

Consequently, for most DHBs the number of questionnaires used to calculate the patient satisfaction scores on a quarterly basis is insufficient and the detailed reporting that is done by the Ministry of Health (e.g. in the quarterly produced DHB Hospital Benchmark Information Report) is shaky at best.

Instead of encouraging the DHBs to improve their performance and increase their sample size, the Directorate issued a directive to all DHBs at the start of the new financial year (July 2008) that data on the patient population make up (age, sex, ethnicity) was no longer required—the reason given was that the information wasn’t used anyway. That this makes it impossible to do checks on the extent to which samples accurately represent patient populations appears to have been regarded as unimportant.

But does this mean that the results of the nationwide patient survey are totally unreliable and worthless? What happens when the reliability and validity of the data is examined?

Reliability

Across the board, and on a scale where 1=very poor and 5=very good, average patient satisfaction ratings for inpatient services range from 3.74 (quality of hospital food) to 4.56 (treating the patient with dignity and respect). For outpatient services, the scores range from 4.33 (informing the outpatient about how long they would have to wait) to 4.52 (treating the patient with dignity and respect).

The scores are well distributed and have relatively large standard deviations ranging from 14% to 32%. The relatively smaller standard deviations on items measuring patients’ rating on being treated with “dignity and respect” suggest the high scores are unanimously endorsed whereas, conversely, large standard deviations on items measuring satisfaction with hospital food (inpatients) and waiting times (outpatients) demonstrate that there is considerable variability across the 21 DHBs on these measures of quality.

To determine the reliability of the inpatient and outpatient questionnaires, the most commonly used measure of internal consistency was calculated: a statistic called “Cronbach alpha”. The value of alpha can range between 0 and 1 and it is generally accepted that if a set of items has an alpha above 0.60, it is usually considered to be internally consistent. If it goes above 0.80, it signifies a very high reliability.

Following Nelson et al (1989),4 who assessed the reliability and validity of the 68-item “Patient Judgement System” (PJS), the alpha statistic of the New Zealand inpatient and outpatient survey was also measured. Although the New Zealand questionnaires were not constructed to assess patient satisfaction on a set of dimensions (as does the 68-item PJS), results show that on measures that gauge satisfaction among inpatients with specific aspects of treatment such as communication (i.e. providing explanation and information), adopting a personal approach and facets of organising patient care, high alpha levels of 0.88, 0.86, and 0.85 were achieved. Similar Cronbach alpha levels were achieved when constructs such as “explanation” and a “personal approach” were analysed among outpatient ratings.

Another method by which one can assess the reliability of a survey instrument is to perform a test-retest reliability analysis. Test-retest reliability estimates are obtained by repeating the measurement using the same questionnaire under as nearly equivalent conditions as possible. However, as it is not possible to re-administer the questionnaire to the same patient 3 months later, the average absolute value of the difference between the two means of two consecutive periods was compared.

The results show extremely small changes in the average scores from one period to the next. When the entire sample is compared in this manner, the difference among inpatients and outpatients over comparable calendar quarters is less than half a percent. Without even taking into account the possibility that some of these differences are caused by actual changes in the delivery process, this stability of measurement provides further support for the reliability of the measures.

Validity

Further analyses focussing on the annual period ending December 2008 show that there is substantial variability across the DHBs on all items in both questionnaires. These statistically significant differences between the DHBs (many at p<0.01, others at p<0.05) provide some support for the validity of the items used.

In the absence of a set of different scales all measuring the same construct, the best example of convergent validity must be the way in which all items are in some way or another associated with the one general validity indicator variable, namely an item which relates directly to the patient’s overall satisfaction with his or her treatment.

The results indicate that, among inpatients, the “overall satisfaction” item is highly correlated with items such as staff availability (r=0.68), being treated with dignity and respect (r=0.68) and being listened to (r=0.67). Among outpatients, overall satisfaction is most strongly correlated among items asking patients to rate staff on how well they explained their condition and informed them about their care (r=0.76).

It is reassuring to note that the highest correlations were found between items that measured closely related aspects of patient care. For instance, among inpatients, information given by ED staff on: (a) the patient’s condition and (b) length of waiting time (items one and two) were very strongly correlated (r=0.76). Among outpatients, the high correlation (r=0.70) between (a) approval of the effort exerted by staff to make an appointment time that suited the patient and (b) satisfaction with the appointment time itself (items 1 and 2) was most revealing.

Conversely, discriminant validity of the nation-wide patient survey is shown by the very low correlations between items such as satisfaction with the quality of hospital food and informed consent (r=0.27). Similarly, among outpatients, a low correlation was evident between the item measuring satisfaction with waiting time and cleanliness (r=0.29).

As the survey clearly distinguishes between items that ought to correlate with one another and items between which one would not expect to find a strong association, these findings provide additional empirical support for the validity of these items.

The relationship between satisfaction and demographic variables

Keeping in mind that the sample size is not sufficiently large to analyse the data on a quarterly basis, and acknowledging the lack of representativeness caused by self-selection of respondents, the characteristics of the sample can nevertheless be scrutinized on the basis of a 12-month period.

Age and sex

The inpatient sample during the 12-month period ending December 2008 consists of 24,533 patients: 12,917 female patients and 11,616 male patients.

Figure 1 shows that the distribution of age between the two sexes is disproportionate due to greater percentage of childbearing women in the 24-44 year age bracket.

Figure 1 Distribution of age and sex in the sample

Ethnicity

Across the board, 80% of these inpatients are European, 10% are Māori, 2% are Pacific Islanders, and 2% are of Asian origin.

Figure 2 shows that Māori and Pacific Island patients are disproportionately represented in the lower age bands while European patients make up 94% of the over 85-year-old age group.

Figure 2. Distribution of age and ethnic group in the sample

Comparing the distribution of non-European inpatients across all DHBs (West Coast is excluded because of its very small sample size), it is evident that Otago has the smallest percentage and Counties Manukau the largest percentage of non-European inpatients (see Figure 3).

Satisfaction as a function of demographic variables

Before the question “How satisfied are New Zealand patients?” can be answered, it is crucial that the relationship between patient satisfaction and demographic variables is understood.

As expected, results show that patient satisfaction rates are a function of age, sex and ethnic group. For instance, Figure 4 shows that age is strongly correlated with satisfaction: older patients are more likely to express greater satisfaction than are younger patients (p<0.01).

Figure 3 Distribution of non-European patients across district health boards

Figure 4 Distribution of overall satisfaction as a function of age

Similarly, patient satisfaction correlates with the patient’s sex (males are more likely to express satisfaction; p<.01) and ethnicity (European patients are more likely than Māori patients to reply with “very good” or “good” when asked to say how satisfied they are; p<0.01; Asian ethnicities are much less likely to answer with “very good” 5).

Thus it is no surprise that hospitals with proportionally more female patients, more non-European patients and a younger population will tend to have lower patient satisfaction rates than hospitals with more and older European male patients. Comparisons between DHBs will have to take this into account to be of any use.

The best way therefore to make appropriate and valid comparisons is either to apply a post-stratification weighting method (i.e. weighting each response using inverted selection probabilities multiplied by the ratios of expected to observed counts) or by confining one’s analysis to a subset of the database, e.g. a specific age or ethnic group or sex.

Another issue is the difference in size between New Zealand hospitals and DHBs. There is sufficient evidence to indicate that, compared to smaller country hospitals, the larger city hospitals with more complicated booking systems, more complex case management, more departments, more facilities, and being physically larger in terms of the ground they occupy, are less likely to have greater patient satisfaction.

In order to provide a level playing field when comparing patient satisfaction rates, the New Zealand Patient Satisfaction Index, which is a quarterly report produced by the author, uses weighting factors to take into account differences in patient profile between the various DHBs and compares satisfaction ratings between DHBs of approximately similar size.

Patient satisfaction in New Zealand

Now the question can be answered: “How satisfied are New Zealand patients?”

Contrary to what is often reported in the popular press about discontented hospital patients, the analysis of the 24,814 inpatients who answered the general “overall satisfaction” question during the most recent 12-month period shows that 65% are very satisfied and an additional 24% are satisfied. This suggests that, across the board, 89% of all inpatients say they had a good hospital experience. Only 8% of inpatients say that their satisfaction is only “average” while 3% of inpatients express overall dissatisfaction.

Similarly, of the 28,432 outpatients who answered this same question about their overall satisfaction with outpatient services and facilities, 67% indicate that their satisfaction is “very good” and an additional 24% reply with “good”. This means that more than 9 out of 10 outpatients are positive about their treatment by the outpatient services. Yet 6% rate their satisfaction as “average” and now only 2% are dissatisfied (one percent respond with “poor” and another 1% respond with “very poor”).

Investigating whether these percentages have increased or decreased over time, it is found that, while overall inpatient satisfaction has not changed much over time, there has been large and significant improvement over the last eight years in terms of outpatient satisfaction. (The Ministry of Health combines the two measures claiming that overall patient satisfaction has increased 6). The increase in outpatient satisfaction is illustrated in the control chart shown in Figure 5.

The control chart shows the “Upper Control Limit” and the “Lower Control Limit” of the series over the last 34 quarterly periods.

Figure 5. Control chart showing overall outpatient satisfaction in New Zealand

The Upper and Lower control limits will vary depending on the variation from quarter to quarter: the greater the variation, the wider the space between the limits. These control limits represent three standard deviations on either side of the distribution.

For any increase in satisfaction to be significant, the combined percentage of “very good” and “good” responses must be greater than the Upper Control Limit. Conversely, any real decrease in satisfaction can only occur when the series dips below the Lower Control Limit.

This increase in outpatient satisfaction has been particularly evident in the smaller DHBs such as Hawke’s Bay, Lakes, South Canterbury, Tairawhiti, Taranaki, and Wairarapa.

But if the patient survey was only able to show general satisfaction rates, any analysis would be rather limited and would not be able to show progress on specific aspects of care or identify which issues should be addressed.

Having data available that stretches back to Sept 2000 allows us to ask questions such as “What was the impact on patient satisfaction when new facilities were built for inpatients?” For example, what happened to satisfaction with cleanliness of facilities at Auckland Hospital when the new city hospital was opened in October 2003?

Figure 6 shows that after a short period of adjustment, there was a substantial and statistically significant (p<.01) increase in satisfaction with cleanliness in the years following the use of the new facilities.

Figure 6. Inpatient satisfaction with cleaning at ADHB

Conclusion

It was demonstrated that patient satisfaction survey data is both reliable and valid. There are shortcomings in the collection of the data, but it has potential to be used to answer questions such as:

Which DHBs have experienced an increase and which DHBs have experienced a decrease in overall patient satisfaction?
In what area(s) of patient care have the increases/decreases been most salient?
What strategies to improve patient satisfaction have proved to be effective and which have proved to be relatively ineffective?

Both the Sector Accountability & Funding Directorate and the DHBs have a responsibility to ensure that the Patient Survey Guidelines developed specifically for this purpose are implemented properly.

To achieve the stated objectives underlying the initiation of the patient survey, the following changes need to be implemented.

The Directorate is required to:

Inform DHBs that deviation from the guidelines will not be tolerated.
Insist that DHBs make greater effort in ensuring scientific validity of the survey.
Encourage the use of the results of the survey by presenting the results in a timely fashion and in a user-friendly format.

And the DHBs are required to:

Use the prescribed questionnaires, not change the sequence of the questions or insert additional questions
Send out the correct number of questionnaires to achieve the required sample size
Carry out data entry check procedures
Provide the required patient population statistics to the Directorate on time
Make use of the results of the survey to improve satisfaction rates in their DHB.

Only when the Sector Accountability & Funding Directorate and the DHBs work together on this project will patients benefit from the huge investment in resources that has been made over the last 10 years.

Competing interests: Gerard Zwier is Managing Director of Health Services Consumer Research Limited (HSCR), a company which carries out patient surveys for several DHBs. HSCR also produces the New Zealand Patient Satisfaction Index which is a report based on data obtained from the government under the Official Information Act.

Author information: Gerard Zwier, Managing Director, Health Services Consumer Research Limited, Auckland

Correspondence: Dr Gerard Zwier, Health Services Consumer Research Limited, PO Box 440, Shortland Street, Auckland 1140, New Zealand. Email: gzwier@xtra.co.nz

References:

CCMAU. Patient Satisfaction Survey Guidelines. Unpublished Crown Company Monitoring Advisory Unit Report June 2000.
Zwier G, Clarke D. How well do we monitor patient satisfaction? Problems with the nation-wide patient survey. N Z Med J. 1999;112 (1097);371–5.
Zwier G, Clarke D. Benchmarking Patient Satisfaction: Do we have a level playing field? NZ Health Hospital, Jan-Feb 2001, p21–4.
Nelson, EC, Hays RD, Larson C, Batalden PB. The Patient Judgment System: Reliability and Validity. QRB. June 1989.
Zwier G. Patient Satisfaction in New Zealand Asian communities. Web log www.hscr.co.nz July 2008.
Patient satisfaction up at near record levels. Media Release. Wellington: Ministry of Health; April 2008.