12th July 2013, Volume 126 Number 1378

Rachel Grocott, Scott Metcalfe, Paul Alexander, Rachel Werner

PHARMAC, the Pharmaceutical Management Agency, is a Government agency that decides which pharmaceuticals will be subsidised in New Zealand. These decisions are made using nine decision criteria;1 one of these criteria is cost-effectiveness.

The relative cost-effectiveness of a pharmaceutical is assessed using cost-utility analysis (CUA). PHARMAC has been undertaking CUA to inform decision-making for nearly 20 years.2–9 CUA is also used by many health funding and assessment agencies internationally.10–12

Relevance of cost-utility analysis to clinical practice

The basic principle of economic analysis, including cost-utility analysis, is that choices need to be made between alternative uses of resources, as resources (e.g. money, personnel, time, etc.) will always be insufficient to support all possible activities. Therefore, by choosing to use resources one way, we forgo other opportunities to use the same resources. Such decisions are made by consumers and health professionals daily, be it as simple as deciding how much time to allocate to each patient, or whether to go out for dinner versus cooking at home. All options are associated with various costs (financial and non-financial) and benefits that need to be considered.

With pharmaceuticals, demand will always exceed our ability to pay, therefore choices are inevitable. CUA provides decision makers with information on the health gains and costs associated with various funding options, so informed decisions can be made. These analyses are based on technical clinical and economic information (as detailed in this article), and PHARMAC staff often seek further advice from clinicians when undertaking economic analysis. This includes advice on the relevance of empirical evidence to the New Zealand population; expected long-term outcomes (in cases where trials end too early); and quality of life impacts. The understanding of the components of CUA helps clinicians provide the specialist advice that PHARMAC requires when modelling the benefits and costs of a pharmaceutical. Further, through the understanding of the inputs of a CUA, clinicians can better understand how the results of a CUA are generated, and why certain treatments may be considered less cost-effective than others.

So what exactly is CUA and why is it so important?

Cost-utility analysis

CUA is a form of economic analysis that quantitatively assesses the health outcomes and costs of a proposed treatment compared with an alternative treatment (often current clinical practice).13,14

Another form of economic analysis that is well-known is cost-effectiveness analysis. CUA differs from cost-effectiveness analysis in the way that health outcomes are measured. With cost-effectiveness analysis, outcomes are measured in common units, such as life years saved or myocardial infarctions prevented. However, as the outcome measures are diverse, it is difficult to compare the cost-effectiveness of treatments for different health conditions.

With CUA, health outcomes are measured using a common currency, usually quality-adjusted life years (QALYs). QALYs take into account changes in patients’ health-related quality of life as well as duration of survival. This enables comparison between the cost-effectiveness of interventions that treat different conditions.

This article describes how CUA is undertaken at PHARMAC (as outlined in the recently updated Prescription for Pharmacoeconomic Analysis8,9). Note that different approaches to CUA may be used by other organisations. PHARMAC has undertaken extensive consultation on the CUA methodology used (Endnote 1), and will continue to review these methods.

Researching and critiquing clinical evidence

Before a CUA is undertaken, there needs to be evidence of net clinical benefit (i.e. that benefits exceed harms), and evidence of relative clinical benefit (i.e. that the treatment is more effective compared with current clinical practice).

Funding applications to PHARMAC generally include some evidence of clinical effectiveness. PHARMAC staff review the applications and ensure that all key evidence is included. In most cases this information is provided to the Pharmacology and Therapeutics Advisory Committee (PTAC) for advice on the pharmaceuticals and their benefits (Endnote 2).

When reviewing the clinical evidence, PHARMAC recommends that well-conducted randomised controlled trials (RCTs) and meta-analyses are the preferred data sources. In the absence of valid RCTs, it is recommended that evidence from the highest available level of study design should be considered with reference to the limitations of the study design.

Evidence is critically appraised, using frameworks such as the Graphic Appraisal Tool for Epidemiology (GATE),15,16 and assessment undertaken on the applicability of the trial to the New Zealand health sector. The following table (Table 1) outlines key factors to consider when critically appraising a clinical trial.

Table 1. Key factors to consider in critical appraisal of clinical trials
Internal validity—How reliable are the trial results?
Availability of data
Were all available trial data used?
Were there quality controls (e.g. was the trial published in a peer-reviewed journal)?
Number of patients
Was the sample size large enough to rule out effects due to chance (i.e. false negatives and false positives)?
Was the effect large enough to be statistically significant even in a small sample size?
Method of randomisation, including adequate concealment
Was there likely to be any selection bias or confounding?
Was there adequate reporting of appropriate randomisation and how this was kept concealed?
Were patients, clinicians and assessors blinded?
Length and completeness of follow-up
Were patients followed for an adequate time period?
How often were patients assessed?
Was analysis by intention-to-treat (including drop-outs and deaths)?
Selection of endpoints
Were the endpoint/outcome measures relevant?
External validity—How relevant are the trial results?
Patient population
Was the patient population in the trial similar to those considered for funding?
Comparator
Was the comparator consistent with current clinical practice in New Zealand?
Dose, formulation and administration regimen
Were these consistent with recommended treatment regimes in New Zealand?

Throughout the article a theoretical worked example is provided to illustrate how a CUA can be constructed and how the results are generated. Each section of this article that details how to undertake CUA will refer to this worked example, building on information in the previous section, with the results of the CUA generated towards the end of the article.

Worked example: A new pharmaceutical is available for advanced bowel cancer (Treatment A). There is already a treatment funded and used widely for advanced bowel cancer (Treatment B). Patients require treatment for 6 months.
On reviewing the clinical evidence, it is established that there has been one randomised controlled trial that has assessed the effectiveness of Treatment A compared with Treatment B for treating advanced bowel cancer. Critical appraisal of the trial indicates that it is of high quality.

Transforming the clinical evidence to estimate QALYs

Economic analysis has two distinct phases:

  • Gathering evidence; and
  • Processing evidence to estimate the effectiveness and relative cost-effectiveness of the pharmaceutical for the proposed indication(s) in the New Zealand clinical setting.

The second phase involves developing economic models that combine information on:

  • Natural disease progression;
  • Clinical effectiveness of the pharmaceuticals (usually obtained from clinical trial data);
  • Health-related quality of life; and
  • Health sector costs and savings (including those incurred in hospitals).

This is illustrated in the diagram below (Figure 1).

Figure 1. Economic model inputs and output

grocott1

Economic models usually consist of a series of branches, representing the expected health outcomes of treatments. When constructing a model the first task is to define the disease in terms of different states that represent clinically important events in the disease process. Transition probabilities are then assigned for movement between these states.17,18 These probabilities are often based on data from clinical trials, for example the probability of response to treatment, probability of an adverse event, etc.

PHARMAC models often take a lifetime horizon (based on the age and gender-specific life-expectancy of the target population) in order to capture the longer term effects of the disease and proposed treatment. However, in some cases the duration of a trial may be too short to show the full impact of treatment. It may therefore be necessary to extrapolate the trial data, based on assumptions regarding the expected long-term outcomes.19–21 For example, many of the trials for smoking cessation treatments evaluated quit rates at 1 year, even though longer-term quit rate data is more relevant. Assumptions therefore need to be made regarding the proportion of patients who would relapse after 1 year.

Assumptions are unavoidable in CUA, however these are made as robust as possible. PHARMAC staff always test the analysis to determine how sensitive the results are to key inputs.

A model is constructed, as illustrated below:

grocott2

Worked example: The clinical trial assessing the benefit of the proposed treatment for advanced bowel cancer (Treatment A), compared with current treatment (Treatment B), reports the following:
  • Treatment A extends survival by approximately 2 months (average survival of 12 months) compared with Treatment B (average survival of 10 months);
  • Patients administered Treatment A had lower rates of treatment-related nausea and vomiting compared with patients administered Treatment B.

Estimating benefits—quality-adjusted life years

Health outcomes in CUA are measured in a common unit—quality-adjusted life years (QALYs). QALYs measure the effect of changes in life expectancy and health-related quality of life that result from treatment. QALYs are the most widely used measure for integrating the effect of treatment on survival and health-related quality of life; however they have been subject to some debate (Endnote 3).22–34

Health-related quality of life—The health-related quality of life component of the QALY takes into account a range of factors that affect people’s quality of life, such as ability to undertake and enjoy leisure activities, freedom from pain, and ability to work and be independent. Health-related quality of life is included in a CUA in the form of a utility value. A utility represents the strength of preference or desirability for a specific level of health status or outcome.

Substantial empirical data are available on the preferences people place on various combinations of factors affecting quality of life. These data have been obtained through various methods, including large surveys that ask participants to rate their health according to the dimensions of the instrument used. Various instruments are available.35

In the New Zealand setting, PHARMAC estimates the health-related quality of life (or utility) associated with a health state using the EuroQol 5D (EQ-5D) instrument. This particular tool consists of five dimensions (mobility, self-care, usual activities, pain/discomfort, and anxiety/depression) and three levels (no problems, some problems and extreme problems). In order to derive utility values that represent New Zealand population preferences, a survey was undertaken of 3000 randomly selected New Zealanders, and was completed by 1360 people.36

The authors of the survey undertook regression analysis, and produced 245 unique utility values, each representing the net aggregate impact of physical, emotional, and social functioning on quality of life. These utility values range from 1 (perfect health) to 0 (death), and also include negative values (states considered to be worse than death). Such values can be applied to a range of diseases and conditions.

Using the EQ-5D, PHARMAC obtains utility values through a process of mapping descriptions of the symptoms patients experience in the health state to the relevant generic health states in the EQ-5D. The utility value generated is then validated through published literature and/or expert clinical advice.

Quality-adjusted life years—Once information is obtained on the impact of the disease on health-related quality of life, and this information is used to obtain utility values, the QALYs can then be estimated. QALYs are calculated by multiplying the duration of time spent in a health state by the health-related quality of life weight (i.e. utility score) associated with that health state. Under the QALY framework, one QALY is equivalent to living one year in perfect health, or two years at half of perfect health, and so on.

In the example below (Figure 2), life expectancy (the number of years left before death) is 6.00. Quality-adjusted life expectancy (the number of QALYs left before death) is 4.75. This is calculated by multiplying each life year by the average quality of life experienced in that year [(4×1)+(1×0.5)+(1×0.25)]. This is equivalent to the area under the curve.

Figure 2. Schematic of quality-adjusted life expectancy

grocott3

The diagram below (Figure 3) illustrates how a theoretical intervention may gain QALYs through both improving patient quality of life and life extension. Without treatment the QALYs are estimated to be 6.00 (life expectancy of 10 years with a quality of life of 0.6). However with the intervention life-expectancy is extended by 2 years, and quality of life improved to 0.9 for 8 years, and then reduces to 0.6 for remaining 4 years. The estimated QALYs are 9.60 [(8×0.9)+(4×0.6)]. Therefore the intervention results in a gain of 3.60 QALYs (9.60−6.00).

Figure 3. Schematic of QALY gains

grocott4

Worked example: Clinical evidence indicates that patients administered Treatment B (current treatment) have higher rates of treatment-related nausea and vomiting, impacting their quality of life. Using the EQ-5D, it is assumed that patients administered Treatment B have a health-related quality of life of 0.4, and patients administered Treatment A (proposed treatment) have a health-related quality of life of 0.6 (on a scale of 0–1).
As outlined previously, patients administered Treatment A have an average survival of 12 months, and patients administered Treatment B have an average survival of 10 months. The QALY for patients administered Treatment A is therefore estimated to be 0.60 (1×0.6), compared with 0.33 (10/12×0.4) for patients administered Treatment B.
The additional QALY gain of Treatment A compared with Treatment B is therefore estimated to be 0.27 (0.60–0.33).

Estimating costs

PHARMAC CUAs are undertaken from the perspective of the funder (with regards to PHARMAC’s decision criteria1). PHARMAC CUAs therefore aim to include all relevant costs (and savings) to the heath sector resulting from funding a treatment.5 This includes:

  • Pharmaceutical costs;
  • Hospital costs;
  • Outpatient costs (for example, laboratory tests, specialist visits, etc.); and
  • Direct patient healthcare costs that Government partially subsidises (for example, General Practitioner visits and residential care). The cost included in the CUA is the cost to the Government plus the additional cost to the patient.

CUAs undertaken by PHARMAC therefore include more than just pharmaceutical costs, and take into account any financial impact a treatment may have on the healthcare sector.5 For example, a CUA on a new oral treatment to replace an infusion will take into account all potential cost-offsets to the health sector associated with patients no longer requiring an infusion.

PHARMAC does not however include the cost-offsets from inability to work, as this is taken into account when estimating QALYs (from reduced health-related quality of life). In addition, valuing lost income would likely bias against treatment that benefit those who do not earn an income (for example, the elderly and children).

Worked example: The total cost per patient of 6 months’ treatment with Treatment A is $5500, compared with a cost of $900 per patient for Treatment B. In addition, Treatment B is an infusion administered at a hospital outpatient unit, at a cost of approximately $1500 per patient. Treatment A is an oral treatment.
The evidence indicates that 10% of patients in the clinical trial needed to be hospitalised due to severe nausea and vomiting with Treatment B, compared with none (0%) of the patients taking Treatment A. The cost per hospitalisation is estimated to be $2500. The total cost of Treatment A is therefore $5500, compared with a total cost of Treatment B of $2650 [$900+$1500+$250 ($2500×0.1)].
The additional cost of Treatment A compared with Treatment B is therefore estimated to be $2850 ($5500–$2650).
Putting it all together—the output of CUA

At PHARMAC the results of CUA are expressed as the difference in QALYs gained divided by the difference in costs between treatments. This provides us with information on the amount of additional health benefit that would be gained as a result of the additional expenditure, and is expressed as the QALYs gained per dollar spent.9 This is shown in the following formula:

content05.jpg

This result can then be multiplied by $1 million to provide information on QALYs gained per million dollars spent, which is a more useful way of expressing the results due to the scale of the investments and values involved.

Another way of expressing the CUA result is the more commonly used ‘cost per QALY’ (i.e. the additional cost of treatment per QALY gained). For example, a treatment may have a cost per QALY of $10,000, indicating that the treatment costs an additional $10,000 to gain 1 QALY (which is the same as 100 QALYs gained per $1 million).

Note that cost per QALY and QALYs gained per $1 million are the same information, but presented in a different way. PHARMAC uses the measure QALYs gained per $1 million to emphasise the additional health gain for the investment.37–41

All future costs and benefits are discounted to the present value42 using the current annual discount rate of 3.5% (Endnote 4).43

The inputs and assumptions in CUAs are always tested to determine how sensitive the results are to various assumptions, such as any uncertainty in long-term benefits of treatment (beyond the period of the trials).

Worked example: As outlined in the QALY section, the additional QALY gain of Treatment A compared with Treatment B is 0.27. The additional cost of funding Treatment A is $2850 (refer to the cost section).
The QALYs gained per $1 million are therefore the additional QALYs divided by the additional cost, multiplied by 1 million (0.27/$2850×$1M). This gives a result of about 95 QALYs gained per $1 million spent.
Therefore, for every million dollars of the total health budget invested in the new medicine (Treatment A), an additional 95 units of benefit (QALYs) would be gained. This result can also be presented as a ‘cost per QALY’ (additional net cost divided by QALY gain), giving a result of approximately $10,600 (i.e. costing the overall health sector $10,600 for each QALY gained).

Use of CUA in decision-making

PHARMAC uses CUA to compare the cost-effectiveness of a pharmaceutical with other pharmaceuticals that could be funded instead. As a proposal to invest in a pharmaceutical can only be considered ‘cost-effective’ in comparison with another proposal, PHARMAC does not have a cost-effectiveness threshold (or pre-determined QALYs per million invested amount) that indicates whether or not a pharmaceutical is ‘cost-effective’. Also, cost-effectiveness is only one of nine decision criteria used by PHARMAC.5

A proposal may be more cost-effective than another but rate poorly on other decision criteria and, therefore, may not be funded. In addition, what is considered to be cost-effective varies with the amount of funding available. This is not just in terms of the total budget each year, but also the available budget anticipated in the future.37,41

Once sufficient information on a proposal is available (such as PTAC priority and cost-effectiveness), this information is compiled and considered by PHARMAC according to its nine decision criteria1 (outlined in Table 2 below).

Table 2. PHARMAC’s decision criteria
PHARMAC’s decision criteria are:
  • The health needs of all eligible people within New Zealand.
  • The particular needs of Māori and Pacific peoples.
  • The availability and suitability of existing medicines, therapeutic medical devices and related products and related things.
  • The clinical benefits and risks of pharmaceuticals.
  • The cost-effectiveness of meeting health needs by funding pharmaceuticals, rather than by using other publicly-funded health and disability support services.
  • The budgetary impact (in terms of the pharmaceutical budget and the Government's overall health budget) of any changes to the Pharmaceutical Schedule.
  • The direct cost to health service users.
  • The Government's priorities for health funding, as set out in any objectives notified by the Crown to PHARMAC, in PHARMAC's Funding Agreement, or elsewhere.
  • Any other criteria that PHARMAC thinks are relevant. PHARMAC will carry out the necessary consultation whenever we intend to take any 'other criteria' into account.

Pharmaceuticals awaiting funding consideration are prioritised against other expenditure options (either listing of new pharmaceuticals or expanding access to existing pharmaceuticals), with the overall aim of identifying those proposals that would provide the best health outcomes if funded. PHARMAC conducts regular prioritisation reviews of all active proposals.

In summary, cost-utility analysis provides valuable information on the amount of additional QALYs (or health benefits) that can be gained from the budget available. This information can be used to inform prioritisation decisions. It is a particularly useful tool for organisations working within a constrained budget who are seeking to identify proposals that offer the most health gains relative to their cost.

Summary

Cost-utility analysis (CUA) is a form of economic analysis that has been used by PHARMAC for nearly 20 years. It is also used by many health funding and assessment agencies internationally. So what is CUA and why is it so important? This article describes the process involved in undertaking CUA, including critical appraisal of clinical evidence; transforming the evidence to estimate quality-adjusted life years (QALYs); estimating costs; and how this information is combined to obtain an output that can be used to inform decision-making. The article also describes how PHARMAC uses CUA to prioritise pharmaceuticals for funding in New Zealand.

Author Information

 Rachel Grocott, Senior Health Economist; Scott Metcalfe, Chief Advisor Population Medicine / Deputy Medical Director (Epidemiology); Paul Alexander, Health Economist; Rachel Werner, Health Economist; PHARMAC, Wellington

Competing interests: The authors are PHARMAC staff. PHARMAC is currently reviewing its Operating Policies and Procedures, including its decision criteria. More information about the review can be found at http://www.pharmac.health.nz/about/operating-policies-and-procedures/decision-criteria-consultation 

References

Endnotes:
  1. For further information on PHARMAC’s consideration of consultation responses to the PFPA version 2, please refer to the following webpages: http://www.pharmac.health.nz/ckeditor_assets/attachments/12/consultation-responses.pdf
  2. http://www.pharmac.health.nz/ckeditor_assets/attachments/14/details-key-amendments.pdf
  3. PTAC is PHARMAC’s primary clinical advisory committee, and there are also a number of PTAC subcommittees, made up of experts in specialist clinical fields such as cardiology and oncology. PTAC makes recommendations to PHARMAC for the assignment of high, medium, or low priorities for proposals; or that a proposal be declined or referred to a subcommittee. PTAC uses the same decision criteria as PHARMAC1 when evaluating pharmaceuticals.
  4. Despite the advantages of using a single indicator to measure effectiveness, QALYs have been debated on ethical and operational grounds.
  5. A key criticism is that QALYs assume uniform preferences (i.e. each QALY has equal value regardless to whom it accrues). This criticism is based on the results of CUAs often being applied within a utilitarian framework. However, CUA is capable of being applied to achieve the desired distribution of QALYs through attaching weights to the estimated QALY gains.
  6. Some argue that QALY calculations bias against elderly due to their shortened life expectancy resulting in fewer QALY gains. However empirical evidence concerning the public’s view on this issue indicates that most would support the favouring of younger people, entailed by the QALY approach. Also note that it is the marginal differences in QALYs which count as benefits, not the average length of survival.22-24
  7. Discounting is used to compare treatments that have costs and benefits that occur at different times. The extent to which future benefits and costs are discounted in comparison with the present is reflected in the discount rate. As the discount rate increases, future benefits and costs become less important when compared with benefits and costs occurring in the present.42 PHARMAC’s current discount rate of 3.5% is based on the five-year average real risk-free long-term government bond rate for New Zealand.
References:
  1. PHARMAC. Operating policies and procedures of the Pharmaceutical Management Agency (“PHARMAC”), Third Edition, January 2006. http://www.pharmac.govt.nz/2005/12/22/231205.pdf
  2. Bennett W, McNee W, Metcalfe S, Wright JM. Use of statins In New Zealand, subsidy of statins is limited to particular groups of patients. BMJ 1997;315:161. http://www.bmj.com/content/315/7122/1615
  3. Braae R, McNee W, Moore D. Managing pharmaceutical expenditure while increasing access. The pharmaceutical management agency (PHARMAC) experience. Pharmacoeconomics. 1999 Dec;16(6):649-60.
  4. Moodie P, Metcalfe S, McNee W. Response from PHARMAC: difficult choices. N Z Med J. 2003;116:U361. http://journal.nzma.org.nz/journal/116-1170/361/
  5. Metcalfe S, Dougherty S, Brougham M, Moodie P. PHARMAC measures savings elsewhere to the health sector. N Z Med J. 2003;116:U362. http://journal.nzma.org.nz/journal/116-1170/362/
  6. A prescription for Pharmacoeconomic Analysis: methods for cost-utility analysis, Version 1.1 PHARMAC: Wellington, New Zealand, 2004. http://www.pharmac.health.nz/ckeditor_assets/attachments/6/pfpa-v-1-1.pdf
  7. Prescription for Pharmacoeconomic Analysis: methods for cost-utility analysis, Version 2 PHARMAC: Wellington, New Zealand, 2007. http://www.pharmac.health.nz/ckeditor_assets/attachments/7/pfpa-2_0.pdf
  8. Prescription for Pharmacoeconomic Analysis: methods for cost-utility analysis, Version 2.1 PHARMAC: Wellington, New Zealand, 2012. http://www.pharmac.govt.nz/2012/06/26/PFPAFinal.pdf
  9. Grocott R, Metcalfe S. PHARMAC's updated guidelines for cost-utility analyses, with new QALYs per $1M metric. N Z Med J. 2012;125:U5274. http://journal.nzma.org.nz/journal/125-1358/5274/
  10. National Institute of Health and Clinical Excellence. Guide to the Methods of Technology Appraisal. London: NICE, 2008. http://www.nice.org.uk/media/B52/A7/TAMethodsGuideUpdatedJune2008.pdf
  11. Canadian Agency for Drugs and Technologies in Health. Guidelines for the Economic Evaluation of Health Technologies. 3rd edition. Ottawa: CADTH, 2006. http://www.cadth.ca/media/pdf/186_EconomicGuidelines_e.pdf
  12. Pharmaceutical Benefits Advisory Committee. Guidelines for Preparing Submissions to the Pharmaceutical Benefits Advisory Committee, version 4.3. Canberra: PBAC, 2008. http://www.health.gov.au/internet/main/publishing.nsf/Content/pbacguidelines-index
  13. Drummond MF, Sculpher MJ, Torrance GW, O'Brien B, Stoddart GL. Methods for the Economic Evaluation of Health Care Programmes. 3rd ed. Oxford: Oxford University Press, 2005.
  14. Siegel JE, Torrance GW, Russell LB, Luce BR, Weinstein MC, Gold MR. Guidelines for pharmacoeconomic studies. Recommendations from the panel on cost effectiveness in health and medicine. Panel on Cost Effectiveness in Health and Medicine. Pharmacoeconomics. 1997;11:159-68.
  15. Jackson R, Ameratunga S, Broad J, Connor J, Lethaby A, et al. The GATE frame: critical appraisal with pictures. Evid Based Med. 2006;11:35-8.
  16. Evidence-Based Practice and Critical Appraisal (updated 4 February 2011). Effective Practice, Informatics & Quality Improvement (EPIQ), at Epidemiology and Biostatistics, School of Population Health, Faculty of Medical and Health Sciences, University of Auckland. http://www.fmhs.auckland.ac.nz/soph/depts/epi/epiq/ebp.aspx
  17. Sonnenberg FA, Beck JR. Markov models in medical decision making: a practical guide. Med Decis Making. 1993 Oct-Dec;13(4):322-38.
  18. Briggs A, Sculpher M. An introduction to Markov modelling for economic evaluation. Pharmacoeconomics. 1998 Apr;13(4):397-409.
  19. Brennan A, Akehurst R. Modelling in health economic evaluation. What is its place? What is its value? Pharmacoeconomics. 2000 May;17(5):445-59.
  20. Weinstein MC. Recent developments in decision-analytic modelling for economic evaluation. Pharmacoeconomics. 2006;24(11):1043-53.
  21. Soto J. Health economic evaluations using decision analytic modeling. Principles and practices--utilization of a checklist to their development and appraisal. Int J Technol Assess Health Care. 2002 Winter;18(1):94-111.
  22. Kawachi I, Bethwaite P, Bethwaite J. The use of quality-adjusted life years (QALYs) in the economic appraisal of health care. N Z Med J. 1990 Feb 14;103(883):46-8.
  23. Schwartz S, Richardson J, Glasziou PP. Quality-adjusted life years: origins, measurements, applications, objections. Aust J Public Health. 1993 Sep;17(3):272-8.
  24. Prieto L, Sacristán JA. Problems and solutions in calculating quality-adjusted life years (QALYs). Health Qual Life Outcomes. 2003 Dec 19;1:80.
  25. Schwappach DL. Resource allocation, social values and the QALY: a review of the debate and empirical evidence. Health Expect 2002;5:210-22.
  26. Wagstaff A. QALYs and the equity-efficiency trade-off. J Health Econ 1991;10(1):21-41. Erratum in: J Health Econ 1993 Jul;12(2):237.
  27. Dolan P, Shaw R, Tsuchiya A, Williams A. QALY maximisation and people's preferences: a methodological review of the literature. Health Econ. 2005 Feb;14(2):197-208. http://onlinelibrary.wiley.com/doi/10.1002/hec.924/pdf
  28. Singer P, McKie J, Kuhse H, Richardson J. Double jeopardy and the use of QALYs in health care allocation. J Med Ethics. 1995;21:144-50. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1376689/pdf/jmedeth00296-0016.pdf
  29. Harris J. Double jeopardy and the veil of ignorance--a reply. J Med Ethics. 1995;21:151-7. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1376690/pdf/jmedeth00296-0023.pdf
  30. McKie J, Kuhse H, Richardson J, Singer P. Double jeopardy, the equal value of lives and the veil of ignorance: a rejoinder to Harris. J Med Ethics. 1996 Aug;22(4):204-8. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1376998/pdf/jmedeth00303-0014.pdf
  31. Smith MD, Drummond M, Brixner D. Moving the QALY forward: rationale for change. Value Health 2009;12 Suppl 1:S1-4. http://www.ispor.org/meetings/invitational/QALY/Paper1.pdf
  32. Weinstein MC, Torrance G, McGuire A. QALYs: the basics. Value Health 2009;12 Suppl 1:S5-9. Erratum in: Value Health. 2010 Dec;13(8):1065. http://www.ispor.org/meetings/invitational/QALY/Paper2revised.PDF
  33. Nord E, Daniels N, Kamlet M. QALYs: some challenges. Value Health 2009;12 Suppl 1:S10-5. http://www.ispor.org/meetings/invitational/QALY/Paper3.pdf
  34. Lipscomb J, Drummond M, Fryback D, Gold M, Revicki D. Retaining, and enhancing, the QALY. Value Health 2009;12 Suppl 1:S18-26.
  35. Nord E. A review of synthetic health indicators. Background paper prepared for the OECD Directorate for Education, Employment, Labour, and Social Affairs, June 1997.
  36. Devlin NJ, Hansen P, Kind P, Williams AH. Logical inconsistencies in survey respondents’ health state valuations – a methodological challenge for estimating social tariffs. Health Economics. 2003; 12:529-44.
  37. Metcalfe S, Rodgers A, Werner R, Schousboe C. PHARMAC has no cost-effectiveness threshold. N Z Med J. 2012;125:99-101. http://journal.nzma.org.nz/journal/125-1350/5083/
  38. Craig BA, Black MA. Incremental cost-effectiveness ratio and incremental net-health benefit: two sides of the same coin. Expert Rev Pharmacoecon Outcomes Res. 2001;1:37-46. http://www.expert-reviews.com/doi/pdf/10.1586/14737167.1.1.37
  39. Zethraeus N, Johannesson M, Jönsson B, Löthgren M, Tambour M. Advantages of using the net-benefit approach for analysing uncertainty in economic evaluation studies. Pharmacoeconomics. 2003;21:39-48. http://adisonline.com/pharmacoeconomics/Abstract/2003/21010/Advantages_of_Using_the_Net_Benefit_Approach_for.3.aspx
  40. Metcalfe S, Grocott R. Comments on "Simoens, S. Health economic assessment: a methodological primer. Int. J. Environ. Res. Public Health 2009, 6,2950-2966"—New Zealand in fact has no cost-effectiveness threshold. Int J Environ Res Public Health. 2010;7:1831-4. http://www.mdpi.com/1660-4601/7/4/1831/
  41. Grocott R. Applying Programme Budgeting Marginal Analysis in the health sector: 12 years of experience. Exp Rev Pharmacoecon Outcomes Res 2009;9:181-7 http://www.expert-reviews.com/doi/abs/10.1586/erp.09.2
  42. West RR, McNabb R, Thompson AG, Sheldon TA, Grimley Evans J. Estimating implied rates of discount in healthcare decision-making. Health Technol Assess. 2003;7(38):1-60.
  43. Grocott R, Metcalfe S. PHARMAC's updated guidelines for cost-effectiveness analyses, with new discount rate. N Z Med J. 2007;120:U2641. http://journal.nzma.org.nz/journal/120-1258/2641/