Frequency of low-value care in Alberta, Canada: a retrospective cohort study
============================================================================

* Finlay A McAlister
* Meng Lin
* Jeff Bakal
* Stafford Dean

## Abstract

**Objective** To determine how frequently 10 low-value services highlighted by Choosing Wisely are done and what factors influence their provision.

**Methods** This is a retrospective cohort study using routinely collected health data from five linked data sets from 2012 to 2015 in the Canadian province of Alberta to determine the frequency with which 10 low-value services were provided.

**Results** Between 2012 and 2015, 162 143 people (4% of all 3 814 536 adult Albertans and 5% of the 3  423 135 who saw a physician at least once in that time frame) received at least one of the 10 low-value services, including 29.8% of Albertans older than 75 years (57 811 of 194 068). The proportion of adults receiving low-value services ranged from carotid artery imaging in 0.1% of asymptomatic adults without cerebrovascular disease, to prostate-specific antigen (PSA) testing in 55.5% of men 75 years or older without a history of prostate cancer. Although age, Charlson scores and frequency of primary care visits were associated with low-value service provision, the directions of the association differed across services; however, higher socioeconomic status, increased frequency of specialist contact and higher ratio of specialists to primary care physicians in the patient’s region were associated with an increased risk of receiving all of the low-value services we examined. The low-value services which resulted in the greatest costs to the healthcare system were cervical cancer screening in women older than 65 without history of cervical dysplasia or genital cancer, PSA testing in men older than 75 without history of prostate cancer and preoperative stress testing/cardiac imaging before non-cardiac surgery.

**Conclusions** Even within a universal coverage healthcare system, the proportion of patients receiving low-value services varied widely (from <0.1% to 56%). Increased use was associated with higher socioeconomic status, increased frequency of specialist contact and higher ratio of specialists to primary care physicians.

*   ambulatory care
*   healthcare quality improvement
*   health services research

## Introduction

Choosing Wisely ([www.choosingwisely.org](http://www.choosingwisely.org)) and the ‘Do not do’ recommendations from the National Institute for Health and Care Excellence ([www.nice.org.uk](http://www.nice.org.uk)) have focused attention on low-value care, defined as healthcare practices providing minimal or no benefit to recipients. Studies in the USA1–6 have demonstrated that between 2% and 42% of all patients may experience at least some low-value care each year and that this accounts for at least 3% of total healthcare spending (although some estimates place it much higher)7—this is likely an underestimate as these studies did not include downstream costs arising from false-positive tests or detection of incidentalomas. Wide variations between regions were demonstrated in these American studies, and factors such as socioeconomic status, variable healthcare coverage and variable access to physician services were postulated as potential reasons. However, little empirical evidence has been published from other countries. Thus, we designed this study to examine the provision of low-value services in an entire Canadian province with an integrated universal-access, free at the point of service, healthcare system.

## Methods

### Design

This is a retrospective cohort study using routinely collected health data from five linked data sets from 2012 to 2015. Of the more than 400 examples of low-value services described on current Choosing Wisely lists, we examined 10 that were included in the initial list generated by the American Board of Internal Medicine in 2012, have also been identified on Canadian Choosing Wisely lists, were evaluated in the six US studies cited above, and can be identified in administrative data using published, validated claims-based algorithms. We excluded two additional items investigated in the US studies (back imaging for non-specific low back pain and vitamin D testing in patients not at risk for metabolic bone disease) as targeted interventions were undertaken in Alberta to reduce the frequency of these tests during the years we studied.8 9 The 10 services we studied represent a mix of tests that would be ordered by primary care physicians or by specialists, and our case definitions and eligibility criteria are outlined in the online supplementary eTable.

### Supplementary file 1

[[SP1.docx]](pending:yes) 

### Setting

Alberta (like all Canadian provinces) has a single-payer, government-funded healthcare system that provides universal access to over 4.3 million people for hospital, emergency department (ED) and physician services that are free at the point of care. Alberta also has a province-wide electronic health record that provides access to all laboratory and diagnostic imaging tests performed on Albertans within the province. This study received ethics approval from the Health Ethics Research Board at xxxxxxx (details blinded) with waiver of informed consent.

### Patient involvement

Patients were not involved in this research project but are involved in various Choosing Wisely groups.

### Data sources

This study used five administrative databases: (1) the Discharge Abstract Database, which records the admission date, discharge date, most responsible diagnosis and up to 25 other diagnoses and procedures for all acute care hospitalisations; (2) the Ambulatory Care Database, which records all patient visits to hospital-based physicians’ offices or EDs with coding for up to 10 conditions; (3) the Physician Claims Database, which tracks all physician claims for services and includes up to three diagnoses per encounter; (4) the Laboratory and Diagnostic Imaging Databases, which track all outpatient laboratory and imaging tests done in Alberta; and (5) the Alberta Health Care Insurance Registry, which includes patient postal code (permitting adjustment for neighbourhood level proxies of socioeconomic status). Every individual in Alberta has a unique personal health identifier, and these numbers were used to link between data sets; only de-identified data after linkage were available to the investigators.

### Study cohort

We identified all Albertans aged 18 years or older presenting to a healthcare provider at least once between 2012 and 2015.

### Covariates

We identified comorbidities for each patient using the ICD-9 and ICD-10 codes from the Discharge Abstract Database for any hospitalisations, any ED visits and any outpatient visits in the 2 years prior to and including the index visit. The validity of diagnoses captured in these data sets using two hits in the outpatient/ED records and/or one hit in the Discharge Abstract Database has been established in Alberta.10 

### Outcomes

Our primary outcome was the proportion of patients receiving at least one of any of the 10 low-value services during the 3 years we studied. We used the operational case definitions employed in the US studies, which were based on the Current Procedural Terminology codes and patient diagnoses based on the ICD-9 and ICD-10 codes present in all healthcare databases (the Discharge Abstract Database, the Ambulatory Care Database and billing claims) in all three study years (online supplementary eTable). Of note, one of the US criteria was colorectal cancer screening for patients older than 85 years; while we examined that, we also evaluated the frequency of colorectal cancer screening for patients older than 75 years, as current Alberta guidelines only advocate screening for patients younger than 75 years (website last accessed 9 February 2017, [http://www.topalbertadoctors.org/download/1009/colorectal_guideline.pdf](http://www.topalbertadoctors.org/download/1009/colorectal_guideline.pdf)). Cost estimates were obtained from Alberta Health Services Data Integration Measurement and Reporting and reflect the costs incurred by the payer (Alberta Health Services) for each service (patients do not pay user fees in Alberta).

### Statistical analysis

In addition to reporting the frequency with which each low-value service was provided over the three study years, we determined the proportion of the population who would be ‘at-risk’ for that service (defined by age and clinical status) to determine the rate that each low-value service was provided per 1000 person years in Alberta. We explored our data per Alberta region (based on the patient’s home address, not the address where service was provided) using eight a priori defined regions (defined by Alberta Health Services independent of this study—two large metro centres with populations in excess of 1.2 million each, five regional centres with catchment areas exceeding 100 000 each, and the rural population). We used data from the College of Physicians and Surgeons of Alberta and Alberta Health Services to define the ratio of specialists to primary care physicians in each region. We used the administrative billing claims to determine the frequency (and type) of physician contacts in the 12 months prior to the provision of each of the low-value services. We performed multivariate logistic regression to examine the association between provision of each low-value service and patient age, sex (except for low-value services that were sex-specific), Charlson comorbidity score, region of residence, frequency and type of physician contacts in prior 12 months, median household income (based on the patient’s residence) and the specialist:primary care physician ratio in each region. SAS V.9.4 was used for all statistical analysis.

## Results

Between 2012 and 2015, 162 143 people (4% of all 3 814 536 adult Albertans and 5% of the 3 423 135 who saw a physician at least once in that time frame) received at least one of the 10 low-value services we studied, including 29.8% of Albertans older than 75 years who saw a physician at least once (57 811 of 194 068). There was reasonably high correlation within patients (r=0.54, p<0.0001)—in other words, the same patients often had multiple low-value investigations done. There was also a gradient across socioeconomic quintiles, with those in the highest quintile being more likely to receive at least one of the 10 low-value services than those adults in the lowest socioeconomic quintile (5.7% vs 4.6%, p<0.0001).

Of the 10 low-value services we examined, prostate-specific antigen (PSA) testing in men 75 years or older without a history of prostate cancer was the most commonly done (55.5% of at-risk population tested within the three study years, and 428 tests per 1000 person years at risk), and carotid artery imaging in asymptomatic patients without a history of cerebrovascular disease was least often done (0.1% of at-risk population over the three study years, and 0.3 tests per 1000 person years at risk) (table 1). The provision of low-value services varied significantly across eight regions defined by postal codes (table 1), but was not consistently higher in any one region.

View this table:
[Table 1](http://qualitysafety.bmj.com/content/27/5/340/T1)

Table 1 
Frequency of low-value services in Alberta between 1 April 2012 and 31 March 2015

This basket of low-value services accounted for $32 186 765 in spending over the 3 years studied (table 2). Taking into account the frequency of each test, the size of the at-risk population and the individual unit cost, the low-value services which resulted in the greatest costs to the healthcare system in our province were cervical cancer screening in women older than 65 without history of cervical dysplasia or genital cancer ($18.1 million), PSA testing in men older than 75 without history of prostate cancer ($5.5 million) and preoperative stress testing/cardiac imaging before non-cardiac surgery ($5.2 million).

View this table:
[Table 2](http://qualitysafety.bmj.com/content/27/5/340/T2)

Table 2 
Total cost of low-value services in Alberta between 1 April 2012 and 31 March 2015

In our multivariable regression models, we found that higher socioeconomic status (using our proxy of median neighbourhood income for the patient’s residence), increased frequency of specialist contact and higher ratio of specialists to primary care physicians in the patient’s region were associated with an increased risk of receiving the low-value services we examined. In table 3 we present the results for the two services accounting for the greatest costs to the Alberta Healthcare system (PSA testing and cervical cancer screening) and the two services with frequencies which differed the most between regions (overly frequent bone mineral density (BMD) testing and carotid artery imaging). While some low-value services (such as PSA testing in elderly men or cervical cancer screening in elderly women) were less common in older patients, those with higher Charlson scores or those who saw their primary care physician more often, other low-value services (too frequent BMD testing or asymptomatic carotid artery imaging) were more commonly done in these same patients. As expected, higher Charlson scores were correlated with increased frequency of physician visits (r=0.44, p<0.001).

View this table:
[Table 3](http://qualitysafety.bmj.com/content/27/5/340/T3)

Table 3 
Multivariate logistic regression of low-value services in Alberta between 1 April 2012 and 31 March 2015

## Discussion

We found that a basket of 10 low-value services (chosen because they have been studied in USA settings and represent a mix of services within the purview of primary care or specialist physicians) were frequently provided in our province, with substantial variation between tests and regions but without a consistent pattern of overuse in patient subgroups defined by age or comorbidity burdens. However, higher socioeconomic status, increased frequency of specialist contact and higher ratio of specialists to primary care physicians in the patient’s home region were all associated with an increased risk of receiving these low-value services in Alberta. Although our finding that 5% of adults who had seen a physician at least once received low-value care seems lower than the 8%6 to 11%5 reported in the USA, we only examined 10 services (compared with 165 and 286 in the US studies). Our finding that low-value care was provided at least once to 30% of Albertans over age 75 mirrors reports from the USA of 25%–42% in Medicare beneficiaries—again we only examine 10 services while the Medicare studies included another 16 services, which we could not reliably extract from administrative records.2 3 

Five of the low-value services we examined (PSA screening, asymptomatic carotid artery screening, too frequent BMD testing, cancer screening in elderly dialysis patients and hypercoagulability testing after first venous thromboembolic event) were done substantially more often in Alberta than in the USA, and three (colon cancer screening in the elderly, carotid artery imaging after syncope and homocysteine testing for cardiovascular screening) were less frequently performed.2 It is interesting to note that both high-cost and low-cost services are in the higher-than-US-frequency and the lower-than-US-frequency groups. Despite being done within a universal healthcare system with no financial barriers to care, our findings mirror studies from the USA demonstrating substantial variation across regions in frequency of low-value care and high correlation within patients (same patients getting multiple low-value investigations or therapies).2 11–13 Although we were not able to examine provider factors, American studies have reported high correlation within physicians (same physician ordering same low-value investigations or therapies for multiple patients independent of patient factors).2 11–13 While American studies have suggested that low-value care is more commonly provided in areas with higher specialist to primary care ratios, these areas also had higher per capita healthcare spending, reduced access to primary care and lower physician concentrations.1 3 12 Thus, our data extend those findings by demonstrating a similar association within a universal healthcare system with no financial barriers to care on the patient side or financial incentives to order tests on the provider side. Akin to data from the USA,6 we found that low-value services were more commonly provided to more advantaged individuals, despite Alberta having a universal-access, no-user-fee healthcare system.

Although none of the low-value services we investigated cause direct harm, they may indirectly confer harm by initiating a diagnostic cascade that will lead to excess costs and may carry risk if false-positives lead to provision of unnecessary therapy. An analysis of 135 Choosing Wisely recommendations revealed that for 40% increased costs were cited as the reason for inclusion on the list, while 49% were included because they were felt to potentially increase risk for patients.14 Others15–18 have pointed out that Choosing Wisely lists generally have not, at least to this point in time, focused on commonly used high-cost procedures, and certainly we found that some of the Choosing Wisely items were infrequently done (such as homocysteine testing for cardiovascular screening or carotid imaging for asymptomatic patients). Thus, one may well question why these tests even need to be listed by Choosing Wisely, or should organisations be instead highlighting more commonly ordered services?

Choosing Wisely initially focused on encouraging conversations between patients (informed by educational materials created by Consumer Reports) and their physicians, but we clearly need to move beyond that to a new era of active interventions. A recent study4 reported that only two of seven low-value services declined (and only marginally) after the launch of Choosing Wisely, leaving a ‘persistent disconnect between publicizing examples of waste and achieving value-based care in practice’.19 A recent analysis from the English National Health Service demonstrated that despite an explicit focus on reducing six operative procedures deemed low-value, only the three lowest cost procedures were significantly reduced.20 The quality improvement literature is rife with studies demonstrating practice variations and evaluating various knowledge translation strategies to modify physician behaviour, but most have focused on enhancing the adoption of new interventions into healthcare and very few have addressed de-implementation or abandonment of interventions shown to be harmful or to have poor cost-effectiveness ratios.21 22 It has long been known that passive knowledge transfer strategies, such as creating and disseminating clinical practice guidelines (or, in the context of this study, Choosing Wisely lists), have little or no effect on clinical practice and that active and multifaceted knowledge implementation strategies are needed to change practice, although the effects are often modest.21 23 Moreover, despite concerns otherwise, it does not appear that patient demands or preferences are a major driver of low-value service provision nor an obstacle to reducing low-value service usage,3 24 and the assumption that patients derive reassurance from normal test results is not necessarily true.25 

### Limitations

Although we used literature-based definitions for claims-based measures of low-value care, validation against a gold standard of clinical appropriateness is needed to make definitive statements about whether a test/procedure/therapy is indicated or not as this depends on the clinical context. Indeed, we would not want to see a 0% rate of low-value service provision as this would mean that sometimes these investigations or therapies are not being provided when they would likely be appropriate. In this context, it is worth noting that rates of low-value care do not need to be risk-adjusted since by definition they are unlikely to yield clinical value for any patient.

We likely undercaptured screening tests since we were using administrative claims data rather than actual clinical encounter data (such as an electronic medical record)—chart audits comparing actual screening rates with those captured in administrative claims confirmed that for some screening tests (such as cervical cancer screening), the undercapture rate in Alberta may be as high as 30% (C Cook, PhD, Director of Evaluation for Chinook Primary Care Network, Alberta Health Services, personal communication, 2016). Of note, this rate is actually lower than a report from the USA, where up to 40% of patients received laboratory and imaging services outside their home health maintenance organisation (and thus not captured in its electronic records).23 However, the direction of this bias serves to strengthen our findings since it would have led to an underestimate of the prevalence of low-value care. We were unable to examine the frequency for all 400+ items on the Choosing Wisely lists as many cannot be determined from administrative data.26 

## Conclusion

While much of the literature exploring the frequency of low-value care has been done in the USA, our study proves that the issue is just as relevant in integrated single-payer healthcare systems with universal coverage/access such as the Canadian province of Alberta. Although the measures we chose to examine only represent a small proportion of all potential low-value services, they do touch on multiple clinical areas. As recently pointed out by Newton and colleagues,27 92% of national clinical performance measures for outpatient care focus on underuse, thereby ‘fostering a culture of more is better and inadvertently encouraging overuse’. We believe that illustrating the extent to which low-value services are provided even in an integrated healthcare delivery system with universal coverage/access and no user charges at the point of care is a first step. Akin to studies in the USA, we found that higher socioeconomic status, increased frequency of specialist contact and a higher ratio of specialists to primary care physicians in regions were associated with an increased risk of receiving the low-value services we examined, while the associations with age, Charlson scores and frequency of primary care visits were inconsistent. Future studies need to develop performance measures and explore active knowledge translation interventions to promote deintensification in some areas of medicine.28 

Median neighbourhood income was calculated using each patient’s home postal code. The specialist/primary care ratio is a continuous variable provided by Alberta Health Services for each region studied—we assigned this to each patient based on their home postal code.

## Acknowledgments

This study is based in part on data provided by Alberta Health and Alberta Health Services Data Integration Measurement and Reporting. The interpretation and conclusions contained herein are those of the researchers and do not necessarily represent the views of the Government of Alberta nor Alberta Health Services. Neither the Government of Alberta nor Alberta Health Services express any opinion in relation to this study.

## Footnotes

*   Contributors FMA is the guarantor of this manuscript and has the right to grant, on behalf of all authors and does grant on behalf of all authors, an exclusive licence (or non-exclusive for government employees) on a worldwide basis to the BMJ Publishing Group Ltd to permit this article (if accepted) to be published in *BMJ* editions and any other BMJPGL products and sublicences such use and exploit all subsidiary rights, as set out in our licence. FMA conceived and designed the study and wrote the first draft, JB and ML obtained the data and conducted analyses, and all authors edited the manuscript and gave final approval.

*   Funding No project specific funding.

*   Competing interests None declared.

*   Ethics approval University of Alberta Health Research Ethics Board.

*   Provenance and peer review Not commissioned; externally peer reviewed.

*   Data sharing statement Due to the restrictions of the Alberta Health Information Act, we are not allowed to release ministry data to external repositories.

## References

1.   Colla CH , Morden NE , Sequist TD , et al . Choosing wisely: prevalence and correlates of low-value health care services in the United States. J Gen Intern Med 2015;30:221–8.[doi:10.1007/s11606-014-3070-z](http://dx.doi.org/10.1007/s11606-014-3070-z) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1007/s11606-014-3070-z&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=25373832&link_type=MED&atom=%2Fqhc%2F27%2F5%2F340.atom) 

2.   Schwartz AL , Landon BE , Elshaug AG , et al . Measuring low-value care in Medicare. JAMA Intern Med 2014;174:1067–76.[doi:10.1001/jamainternmed.2014.1541](http://dx.doi.org/10.1001/jamainternmed.2014.1541) 
    
    

3.   Schwartz AL , Chernew ME , Landon BE , et al . Changes in Low-Value Services in Year 1 of the Medicare Pioneer Accountable Care Organization Program. JAMA Intern Med 2015;175:1815–25.[doi:10.1001/jamainternmed.2015.4525](http://dx.doi.org/10.1001/jamainternmed.2015.4525) 
    
    

4.   Rosenberg A , Agiro A , Gottlieb M , et al . Early Trends Among Seven Recommendations From the Choosing Wisely Campaign. JAMA Intern Med 2015;175:1913–20.[doi:10.1001/jamainternmed.2015.5441](http://dx.doi.org/10.1001/jamainternmed.2015.5441) 
    
    

5.   Charlesworth CJ , Meath TH , Schwartz AL , et al . Comparison of Low-Value Care in Medicaid vs Commercially Insured Populations. JAMA Intern Med 2016;176:998–1004.[doi:10.1001/jamainternmed.2016.2086](http://dx.doi.org/10.1001/jamainternmed.2016.2086) 
    
    

6.   Reid RO , Rabideau B , Sood N . Low-Value Health Care Services in a Commercially Insured Population. JAMA Intern Med 2016;176:1567.[doi:10.1001/jamainternmed.2016.5031](http://dx.doi.org/10.1001/jamainternmed.2016.5031) 
    
    

7.   Berwick DM , Hackbarth AD . Eliminating waste in US health care. JAMA 2012;307:1513–6.[doi:10.1001/jama.2012.362](http://dx.doi.org/10.1001/jama.2012.362) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.2012.362&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=22419800&link_type=MED&atom=%2Fqhc%2F27%2F5%2F340.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000302538100025&link_type=ISI) 

8.   Ferrari R , Prosser C . Testing Vitamin D Levels and Choosing Wisely. JAMA Intern Med 2016;176:1019–20.[doi:10.1001/jamainternmed.2016.1929](http://dx.doi.org/10.1001/jamainternmed.2016.1929) 
    
    

9.   Ferrari R . Imaging studies in patients with spinal pain: Practice audit evaluation of Choosing Wisely Canada recommendations. Can Fam Physician 2016;62:e129–37.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiY2ZwIjtzOjU6InJlc2lkIjtzOjk6IjYyLzMvZTEyOSI7czo0OiJhdG9tIjtzOjE4OiIvcWhjLzI3LzUvMzQwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

10.  Quan H , Sundararajan V , Halfon P , et al . Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care 2005;43:1130–9.[doi:10.1097/01.mlr.0000182534.19832.83](http://dx.doi.org/10.1097/01.mlr.0000182534.19832.83) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/01.mlr.0000182534.19832.83&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=16224307&link_type=MED&atom=%2Fqhc%2F27%2F5%2F340.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000233268500010&link_type=ISI) 

11.  Baicker K , Chandra A . Medicare spending, the physician workforce, and beneficiaries' quality of care. Health Aff 2004;Suppl Web Exclusives:184–97.[doi:10.1377/hlthaff.w4.184](http://dx.doi.org/10.1377/hlthaff.w4.184) 
    
    

12.  Fisher ES , Wennberg DE , Stukel TA , et al . The implications of regional variations in Medicare spending. Part 1: the content, quality, and accessibility of care. Ann Intern Med 2003;138:273–87.[doi:10.7326/0003-4819-138-4-200302180-00006](http://dx.doi.org/10.7326/0003-4819-138-4-200302180-00006) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.7326/0003-4819-138-4-200302180-00006&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12585825&link_type=MED&atom=%2Fqhc%2F27%2F5%2F340.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000181019400001&link_type=ISI) 

13.  Lipitz-Snyderman A , Sima CS , Atoria CL , et al . Physician-Driven Variation in Nonrecommended Services Among Older Adults Diagnosed With Cancer. JAMA Intern Med 2016;176:1541–8.[doi:10.1001/jamainternmed.2016.4426](http://dx.doi.org/10.1001/jamainternmed.2016.4426) 
    
    

14.  Gliwa C , Pearson SD . Evidentiary rationales for the Choosing Wisely Top 5 lists. JAMA 2014;311:1443–4.[doi:10.1001/jama.2013.285362](http://dx.doi.org/10.1001/jama.2013.285362) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.2013.285362&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=24715077&link_type=MED&atom=%2Fqhc%2F27%2F5%2F340.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000333935000022&link_type=ISI) 

15.  Morden NE , Colla CH , Sequist TD , et al . Choosing wisely--the politics and economics of labeling low-value services. N Engl J Med 2014;370:589–92.[doi:10.1056/NEJMp1314965](http://dx.doi.org/10.1056/NEJMp1314965) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1056/NEJMp1314965&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=24450859&link_type=MED&atom=%2Fqhc%2F27%2F5%2F340.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000331187500003&link_type=ISI) 

16.  Lenzer J . Choosing Wisely: setbacks and progress. BMJ 2015;351:h6760.[doi:10.1136/bmj.h6760](http://dx.doi.org/10.1136/bmj.h6760) 
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNTEvZGVjMTVfOC9oNjc2MCI7czo0OiJhdG9tIjtzOjE4OiIvcWhjLzI3LzUvMzQwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

17.  Kerr EA , Chen J , Sussman JB , et al . Stress testing before low-risk surgery: so many recommendations, so little overuse. JAMA Intern Med 2015;175:645–7.[doi:10.1001/jamainternmed.2014.7877](http://dx.doi.org/10.1001/jamainternmed.2014.7877) 
    
    

18.  Colla CH , Sequist TD , Rosenthal MB , et al . Use of non-indicated cardiac testing in low-risk patients: Choosing Wisely. BMJ Qual Saf 2015;24:149–53.[doi:10.1136/bmjqs-2014-003087](http://dx.doi.org/10.1136/bmjqs-2014-003087) 
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjI0LzIvMTQ5IjtzOjQ6ImF0b20iO3M6MTg6Ii9xaGMvMjcvNS8zNDAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

19.  Parks AL , O’Malley PG . From Choosing Wisely to Practicing Value-More to the Story. JAMA Intern Med 2016;176:1571–2.[doi:10.1001/jamainternmed.2016.5034](http://dx.doi.org/10.1001/jamainternmed.2016.5034) 
    
    

20.  Coronini-Cronberg S , Bixby H , Laverty AA , et al . English National Health Service’s savings plan may have helped reduce the use of three ’low-value' procedures. Health Aff (Millwood) 2015:34381–9.
    
    

21.  Majumdar SR , McAlister FA , Furberg CD . From publication to practice in chronic cardiovascular disease – The Long and Winding Road. J Am Coll Cardiol 2004;43:1738–42.
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6MzoiUERGIjtzOjExOiJqb3VybmFsQ29kZSI7czo0OiJhY2NqIjtzOjU6InJlc2lkIjtzOjEwOiI0My8xMC8xNzM4IjtzOjQ6ImF0b20iO3M6MTg6Ii9xaGMvMjcvNS8zNDAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

22.  Howard DH , Shen YC . Comparative effectiveness research, technological abandonment, and health care spending. Adv Health Econ Health Serv Res 2012;23:103–21.
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=23156662&link_type=MED&atom=%2Fqhc%2F27%2F5%2F340.atom) 

23.  Lee VS , Kawamoto K , Hess R , et al . Implementation of a Value-Driven Outcomes Program to Identify High Variability in Clinical Costs and Outcomes and Association With Reduced Cost and Improved Quality. JAMA 2016;316:1061–72.[doi:10.1001/jama.2016.12226](http://dx.doi.org/10.1001/jama.2016.12226) 
    
    

24.  Gogineni K , Shuman KL , Chinn D , et al . Patient Demands and Requests for Cancer Tests and Treatments. JAMA Oncol 2015;1:33–9.[doi:10.1001/jamaoncol.2014.197](http://dx.doi.org/10.1001/jamaoncol.2014.197) 
    
    

25.  Rolfe A , Burton C . Reassurance After Diagnostic Testing With a Low Pretest Probability of Serious Disease. JAMA Intern Med 2013;173:407–16.[doi:10.1001/jamainternmed.2013.2762](http://dx.doi.org/10.1001/jamainternmed.2013.2762) 
    
    

26.  Bhatia RS , Levinson W , Shortt S , et al . Measuring the effect of Choosing Wisely: an integrated framework to assess campaign impact on low-value care. BMJ Qual Saf 2015;24:523–31.[doi:10.1136/bmjqs-2015-004070](http://dx.doi.org/10.1136/bmjqs-2015-004070) 
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjI0LzgvNTIzIjtzOjQ6ImF0b20iO3M6MTg6Ii9xaGMvMjcvNS8zNDAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

27.  Newton EH , Zazzera EA , Van Moorsel G , et al . Undermeasuring Overuse--An Examination of National Clinical Performance Measures. JAMA Intern Med 2015;175:1709–11.[doi:10.1001/jamainternmed.2015.4025](http://dx.doi.org/10.1001/jamainternmed.2015.4025) 
    
    

28.  Kerr EA , Hofer TP . Deintensification of Routine Medical Services: The Next Frontier for Improving Care Quality. JAMA Intern Med 2016;176:978–80.[doi:10.1001/jamainternmed.2016.2292](http://dx.doi.org/10.1001/jamainternmed.2016.2292)