Surrogates under scrutiny: fallible correlations, fatal consequences

Unproved benefits

“I think we’ve been far too cavalier in accepting favourable changes in a biomarker as a perfect proxy for patient benefit,” says Yale University public health professor Harlan Krumholz, who helped put together a recent report on surrogates for the United States Institute of Medicine.7 He says the focus on “knowing your numbers” and “treating to target” seems to play to everyone’s best interest—they are easy public health messages, they need only quick visits to doctors’ offices, and are a great boon for companies, which don’t have to do the larger long term studies of effects on clinically meaningful outcomes. Professor Krumholz believes the evidence shows “simple assumptions” about surrogates are often incorrect, and he argues we need to better inform people about potential dangers. “If a drug is approved only on the evidence of its impacts on a biomarker, there should be big clear warnings saying it has an unproven effect on patient health,” he says. “We need to convey the uncertainty.”

That recent report from the Institute of Medicine—Evaluation of Biomarkers and Surrogate Endpoints in Chronic Disease—is sobering reading.7 It details an often misplaced confidence in relying on surrogates to assess treatment benefits. “Remarkably, the cautionary voices speaking about the risks of using surrogate endpoints have been repeating the same messages for 20 years,” says the report. “What has been changing is the continually increasing amount of data supporting their arguments.” The report cites examples where treatments have benefited surrogate measures but harmed people, and it urges a far more rigorous evaluation of how these intermediate end points are used.

Even for widely used surrogates there’s more uncertainty than we might imagine. Although blood pressure is an extremely reliable proxy, questions arise from the fact that treatments similarly affecting blood pressure can have different effects on heart disease.7 In the face of the AIDS crisis, researchers discovered HIV-1 RNA was a valuable biomarker, enabling experimental drugs to be assessed quickly. However, in some contexts, short term changes in that biomarker proved to be a poor surrogate because partial suppression of the virus allowed the formation of drug resistant mutations and limited future usefulness of drugs.7 And although reduction in tumour size is sometimes a useful way to measure the effect of cancer treatment, even major shrinkage does not always represent meaningful improvement because, for example, in some cancers, smaller tumours tend to grow faster than larger ones.7 There are even doubts about the value of cholesterol, and it serves as a powerful case study of the need for a new way of thinking about surrogate markers.

Caution over cholesterol

Regulators, including the United States Food and Drug Administration, have officially sanctioned low density lipoprotein (LDL) cholesterol as a surrogate end point, allowing many drugs to be approved and marketed if they reduce this “bad cholesterol” without requiring proof they actually improve health. Yet despite the size of the global markets, there are myriad uncertainties surrounding cholesterol. According to the 2010 Institute of Medicine report, although the methods used to determine blood cholesterol are reliable and reproducible, they do not directly measure LDL cholesterol and so have “limitations.”7 In addition, though LDL cholesterol is “hypothesised” to have a causal role in the atherosclerotic disease process, it has “not been conclusively proven.”7 And reinforcing the general point about our over-confidence in surrogates, the report reminds us that lowering LDL cholesterol “does not always correlate with improved patient outcome.” The report points out there are over 200 coronary risk factors, and that cholesterol, while currently considered by many to be a valuable biomarker for heart disease, is only one of “multiple determinants” and “numerous other mediators.”

The benefits of long term preventive therapies like cholesterol lowering drugs are usually portrayed as relative reductions in risk, but when the risks are considered in absolute terms, a different picture emerges. For example, based on a Cochrane review of trials for primary prevention,8 there has been recent enthusiasm that for people without a history of heart disease statins can reduce premature deaths by 17%, coronary heart disease by 28%, strokes by 22%, and revascularisation by 34%.9 Yet a close reading of the tables from that systematic review suggests the estimated absolute risk reductions with around four to five years of drug taking are 0.5% for death, 1.9% for coronary heart disease, 0.5% for stroke, and 0.7% for revascularisation.8 The estimated number needed to treat for four to five years thus ranges from 50 to 200 depending on the outcome measure. So according to this evidence, most people taking long term statins for primary prevention gain no direct benefit.

While there is strong evidence of benefit, reinforced in other recent studies,10 the magnitude of that benefit in absolute terms is extremely small for those at low risk. There is also a question over the effects of industry funding on this evidence, as even a small amount of bias in the original trials could make the difference between an overall finding of benefit or an overall finding of no effect, in the systematic review’s summary. In fact, the Cochrane reviewers’ discussion of their results offers cause for great caution in interpreting the existing evidence about statins for primary prevention, and reinforces wider questions about the benefits of long term preventive medicines for otherwise healthy people.

The reviewers said they were unable to disaggregate composite outcome measures reported in the 14 statin trials they reviewed; that one third of those original trials reported outcomes selectively; and a majority did not even report on the drugs’ possible harms. Two of the larger cholesterol lowering trials included for review were stopped prematurely because of statistically significant benefits being achieved early, which “may lead to an over-estimation of treatment effects.” The Cochrane review also reported that all but one of the original trials had some form of drug company sponsorship, which has been shown to increase the likelihood of bias favouring the sponsor’s drug. “In primary prevention where world-wide the numbers of patients eligible for treatment are massive,” the reviewers wrote, “there might be motivations to use composite outcomes and early stopping to get results that clearly support intervention.” When an independent Canadian group recently reviewed similar evidence, they found that while these drugs undoubtedly lower cholesterol, “the claimed mortality benefit of statins for primary prevention is more likely a measure of bias than a real effect.”11

Related questions surround other commonly used surrogates which form the basis for profitable markets in drugs for people at risk of future events. A National Institutes of Health consensus conference on hepatitis B found in 2008 that while drugs improved virological or biochemical markers, there was only “low quality evidence” showing correlations between these intermediate outcomes and real world clinical outcomes, and “no conclusive evidence” that treatments reduced the risk of liver disease or death.12 The surrogate of bone mineral density has been controversial for many years, regarded by some researchers as accounting for only a minor proportion of a person’s overall risk of fracture.13 And while the search continues for reliable biomarkers to enable the prevention of Alzheimer’s, a recent conference found, “firm conclusions cannot be drawn about the association of any modifiable risk factor with cognitive decline or Alzheimer’s disease.”14 With type 2 diabetes, heavily promoted drug strategies to aggressively reduce blood sugar have raised people’s risks of heart disease and death,15 16 rather than lowered them, despite evidence that lifestyle changes are cheap and effective.17 Some argue the treatment of type 2 diabetes is a classic example of the medical profession ignoring unfavourable evidence that shows no benefit on meaningful outcomes, while favouring studies that show positive effects on surrogate numbers.18

History shows drugs help make diseases

Jeremy Greene, historian of science at Harvard University, has described “a radical restructuring of the normal and the pathological” that emerged in the second half of the 20th century as symptomless people at risk of future disease were increasingly classified as having medical conditions.1 In Prescribing by Numbers, Dr Greene charts how drug companies and their latest products have helped to shape and expand these new risk based conditions—including high blood pressure, type 2 diabetes, and high cholesterol: “Pharmaceuticals played a central and active role in the definition of these categories of illness,” he writes. Dr Greene argues this process of expanding categories to include people previously considered healthy can be seen as medicalisation. But rather than a monolithic or concerted strategy driven from the boardrooms of companies or professional medical associations, it demonstrates the “porous relationship between the science and the business of health care,” and the process carries benefits as well as risks.

That porous relationship between science and business is still evident in the more recent development of “quality measures,” which urge doctors to test for and treat the risk based conditions of the healthy. In the United States one of the main agencies developing quality measures, the National Committee for Quality Assurance, is directly funded by several drug companies, along with other sponsors.19 In the United Kingdom the process is overseen by a public institution, though there are criticisms of an overly narrow focus on numbers that can be modified by drug treatment.20 A long term critic of widening boundaries of illness, Iona Heath, president of the Royal College of General Practitioners, believes many thresholds have been set far too low. She argues quality measures may be encouraging tests that create anxiety and treatments that help surrogates rather than ameliorate the suffering of people.

In recent months we have seen yet more evidence confirming the risks of our over-reliance on surrogate numbers. In May a large trial of niacin was stopped. Though combining niacin with a statin boosted so called “good” cholesterol (high density lipoprotein) high doses were also associated with a small increase in strokes.21 In June a study in JAMA uncovered a pattern of over-estimating the strength of correlations between biomarkers and their respective diseases,22 highlighting the “thin line between hype and hope.”23

Shift from numbers to people

A major rethink of the role of surrogates in medicine is timely. Routinely approving and prescribing therapies on the basis of their effects on someone’s numbers, rather than their health, is increasingly seen as irresponsible and dangerous. And even when evidence suggests clinical benefits of popular “preventive” medicines for those at lower risks, a rational assessment reveals many people must be treated to prevent one adverse event, so most users gain no direct benefit despite years of treatment. The cost effectiveness of this approach is unsurprisingly in doubt.24 More disturbing still are the questions about whether some of the suggested clinical benefits are real or simply artefacts of sponsorship bias.

The rigour of the evidence informed approach to medicine has in recent decades helped us all understand the limitations of relying on surrogates, and for one of its key architects—McMaster University professor Gordon Guyatt—this problem is both historical and cultural. He argues that central to putting American medicine on a scientific basis was the assumption that an understanding of biological mechanisms would translate into improved management of patients. And while medical students over a century later are still taught to focus on fixing a person’s biological numbers—whether it’s cholesterol or bone density—what is urgently required is a new approach that provides genuine improvement for the person.

Understanding biological mechanisms and diagnosing by numbers has undoubtedly brought great benefits. Yet as the definitions of medical conditions relentlessly expand via that porous relationship between the science and business of healthcare, this fragmented reductionist approach is conferring multiple medical labels on vast swathes of healthy people, who are then treated with preventive drugs that won’t help most of them and may hurt many.25 The magic of numbers may help corporate profits and professional pride, but at what cost to the health of ordinary people who mistake a numerical benefit for a genuine one? Surely it’s time to ask if there might be a healthier new model for medicine based on far less harmful and costly ways to try to reduce human suffering.