The evidence base for new drugs

Marketing campaigns cast new drugs as “must haves.” In reality, the public evidence base for new drugs often leaves more questions than answers.1 2 While marketing authorization indicates that regulators judged the risk-benefit profile to be favorable, product labels rarely list and quantify these benefits and harms. Publications of premarketing trials might fill in some gaps, but readers beware: not all trials are published and those that are may be inaccurately or incompletely reported.3 4

Data transparency in clinical trials has emerged as a way to tackle the reporting biases that affect literature, enabling independent scrutiny of trials.5 But it is not enough. Even with open data we rarely know how new drugs compare with existing options or whether they improve patient centered outcomes. An analysis published in this issue (doi:10.1136/bmj.h796) suggests that new legislation in Germany may provide another piece of the puzzle.6

The German act on the reform of the market for medicinal products (AMNOG) was introduced in 2011 to inform drug pricing for all new drugs. To control costs, sponsors must submit a standardized dossier including evidence of the drug’s added benefit over already available drugs. This dossier is then reviewed by scientists, usually at the Institute for Quality and Efficiency in Health Care (IQWiG), who produce an assessment report. Their report, plus some (but not all) of the documents in the sponsor’s dossier, are subsequently made public. The completeness of trial methods and results in the new public “AMNOG documents” is assessed by Köhler and colleagues from IQWiG.6

Germany’s reforms seem to have missed the obvious. Clinical study reports—the long and complex documents submitted by sponsors to regulators, reporting the methods and results of randomized controlled trials—are not part of the AMNOG documents that are publicly released.2 Clinical study reports stay hidden. But Kohler and colleagues’ exhaustive description of the content of 15 submitted dossiers shows how what is made public still has considerable value, both in terms of breadth and depth.

Like clinical study reports, AMNOG documents contain far more detail than journal publications and other currently open sources (including registers and European public assessment reports). Köhler and colleagues found that about 90% of items were completely reported in AMNOG documents compared with 75% for methods and 52% for results items in non-AMNOG sources. This is good, but we would have also liked to know how clinical study reports scored.

Importantly, AMNOG documents contain a considerable amount of information that may be missing from clinical study reports of typical premarketing trials. These trials often fail to use clinically meaningful comparators and to study important subpopulations of patients. Such shortcomings may limit generalizabilty of findings. This problem is especially troublesome for health technology assessments, which seek to answer questions relevant to clinical use, such as the effects of a drug versus other treatments—and not inert comparators.

By contrast, AMNOG documents include post-marketing studies, trials with active drug comparators, extra information on methods, analyses in subpopulations tied to the approved indications, and a list of all randomized controlled trials of the drug for the approved indication (something not available from the European Medicines Agency).7

Most importantly for clinicians, Köhler and colleagues show that AMNOG documents are more clinically relevant than open sources because of their frequent inclusion of patient relevant outcomes. They may even be more useful on this score than clinical study reports. Particularly important are the systematic reviews by IQWiG assessing the potential added value of a new drug compared with a currently available comparator. These systematic reviews are likely to be far more authoritative than those based on open sources. For those engaged in assessing the performance of drugs in everyday use, as occurs in health technology assessment, the message is equally clear: meaningful evaluations of drugs require considerably more information than is presently available from open sources.

The public availability of AMNOG documents represents another important step towards the complete visibility of trial programmes and evidence about drugs’ effects in meaningful contexts.

However, there are major limitations to the usability—and possibly impact—of the AMNOG documents. Firstly, most of the documents are in German, limiting readership. Secondly, excluding clinical study reports because content is classified as “commercially confidential” seems out of date, particularly since the European Medicines Agency pledged to release clinical study reports in its possession. Although “relevant information” from these reports is made available in IQWiG’s assessment, we are reading information filtered by IQWiG, not the original. Selective reporting could be introduced, especially since IQWiG has only three months to assess the evidence, produce its own report, and assemble the AMNOG documents package. Viewed more optimistically, such filtering and interpretation could aid readers’ understanding in a similar way to the drug approval packages produced by the US Food and Drug Administration. The packages, which are freely available from Drugs@FDA, report the regulator’s scrutiny of the evidence submitted in support of registration—but not the evidence itself. While this limits usability and represents the views of the regulators and not those of the sponsors, from our experience of evaluating the evidence of neuraminidase inhibitors we have found that such commentary adds valuable insight to meaningfully interpret trial evidence.8

AMNOG dossiers are another piece of a puzzle that is far more complex and layered than any of us suspected a few years ago. One of the biggest unanswered questions is where all this leaves the Cochrane Collaboration and the evidence based medicine movement, which still rely on open sources that we now know are distorted by reporting bias.9 10 11

The puzzle may only be completed when all data generated by experiments on humans become available unconditionally. Meanwhile, we should cheer the German government and its staff for taking us one step closer.