Intended for healthcare professionals

Sensitivity analysis

Assessing concerns regarding the validity of three trials included in "Role of multivitamins and mineral supplements in preventing infections in elderly people: systematic review and meta-analysis of randomised controlled trials"

Alexander J Sutton, Alia El Kadiki

Department of Health Sciences, University of Leicester, 22- 28 Princess Road West, Leicester, LE1 6TP, UK

Alexander J Sutton

senior lecturer in medical statistics

Chemical Pathology Department, Royal Hallamshire Hospital, Glossop Road, Sheffield, S10 2JF, UK

Alia El Kadiki

specialist registrar

Correspondence to: Alexander J Sutton (ajs22{at}le.ac.uk)

 

Background

This article is a follow up to the systematic review and meta-analysis of the use of multivitamins and mineral supplements in preventing infections in elderly people published recently in the BMJ.1 Following the publication of this paper it came to the attention of both the BMJ and the authors that doubts regarding the validity of three of the trials2-4 included in the systematic review and meta-analyses had been raised publicly (see below). It is the aim of this article to assess the impact these allegations, if true, would have on the results of our systematic review and meta-analysis. We do not know whether the allegations made against the trials are true, but, after discussions with the editors of the BMJ, we were satisfied that there are enough published concerns relating to these studies to warrant the sensitivity analysis presented here.

Concerns about studies included in the systematic review/meta-analysis

Concerns have arisen about three trials included in our systematic review. Concerns were raised5 about the trial published by Chandra in the Lancet in 1992 2 relating to inconsistencies in the statistical analysis results and other anomalies. Although these were largely dismissed by Chandra in a letter of response,6 further concerns were raised7,8 regarding a follow-up paper by Chandra9 concerning the same trial but relating to different (cognitive) outcomes. As a result, Nutrition10 has recently retracted this follow-up paper.

A further trial3 by Chandra was also included in our meta-analysis. Specific concerns have been raised regarding this trial also,8 in addition to the fact that it was essentially a replication of a trial under scrutiny.

One of the rapid responses, following a news item in the BMJ11 highlighting concerns regarding Chandra’s now retracted article,9 was by Roberts and Sternberg12 (who had published some of the initial concerns regarding Chandra’s research). In this response they raised concerns relating to a trial report by (Amrit L) Jain4 which replicated Chandra’s 1992 Lancet results.2 This trial was published in Nutrition Research, which at the time was edited by Chandra. They also stated: (i) they were unable to find any other "trace" of Jain; (ii) the mailing address on the article was a rented mailbox not far from Memorial University where Chandra was employed; and (iii) they have been unable to locate any other trace of the institution affiliation given to Jain in his article. However, to date, none of the three trial reports2-4 included in our original systematic review have been retracted by the journals in which they were published.

Sensitivity analysis

Since none of the papers under scrutiny have been retracted by the journals they were published in, we are taking the stance that the analysis including such studies, as published in our original article, is still an appropriate primary analysis (see Discussion). However, had we known about the allegations at the time of writing the original paper, we would have conducted a careful sensitivity analysis investigating the influence such allegations would have, if true, on the evidence base. It is this sensitivity analysis we present here. We consider the three outcomes meta-analysed in the original paper in turn.

Outcome one: Mean difference in number of days spent with infection

Only the three trials under scrutiny contribute data to this outcome. Unaware of the allegations made by others, we independently commented in our original paper that the standard deviations looked very small for all studies reporting the outcome mean difference in number of days spent with infections on multivitamins and minerals compared with placebo over 12 months. We commented that these standard deviations may, in fact, be standard errors and performed a sensitivity analysis adjusting the analysis accordingly. In fact, Chandra, in a letter to the Lancet,6 had acknowledged this mistake had been made in his 1992 study but we were unaware of this letter at the time of the original analysis. As we reported previously,1 by inflating the errors in the study estimates in this way for all three trials, the pooled reduction in days of infection over 12 months changed from 17.5 (11 to 24) to 14 (10 to 18). (Figure 1)

Obviously, such a revised analysis is only appropriate if this were a reporting error in the three trials. The more serious allegations raised against these studies would seem to imply that the data were not legitimate. The impact of such accusations is clear. Since all studies individually show a statistically significant benefit then removing any one or two of them would still result in a suggested benefit for this outcome. Clearly, if all three trials were removed then no valid data would be available for this outcome.

Outcome two: Odds ratio of at least one infection in the study period

None of the trials under scrutiny contributed to this outcome, so the pooled odds ratio for one or more infections in the study period remains at 1.10 (0.81 to 1.50), suggesting little support for the benefit of multivitamins and minerals, with no sensitivity analysis required.

Outcome three: Incidence rate ratio for the difference in infection rates

The only trial under scrutiny that contributed to this outcome – the incidence rate ratio of infection for vitamins and minerals compared to placebo – was that by Jain.4 We now reanalyse this outcome removing this trial and combining the remaining three trials. In the revised analysis, the pooled incidence rate ratio is 1.00 (0.85 to 1.17) (Figure 2) compared to 0.89 (0.78 to 1.03) when all studies were included. Hence, removing the Jain trial (which had the most beneficial outcome), has had a large impact on the pooled estimate, with any suggestion of multivitamins being beneficial removed.

Summary

All four estimates of effect reported across the three outcomes in the three trials under scrutiny suggested a statistically significant benefit, while the six other estimates reported in the other five trials in the systematic review did not. If the allegations are true that these three studies are not reliable, then the remaining evidence base suggests no benefit for the use of multivitamins for preventing infections in the elderly. Under the less extreme situation that one or more of the three trials is valid, this would still provide some evidence that multivitamins may be beneficial and further investigation may be warranted.

Discussion

In this article we have aimed to explore the impact concerns about the validity of a number of studies would have on our previous meta-analysis, if they were true. If the three studies are indeed completely invalid, then this would change the evidence base from conflicting and heterogeneous to more homogeneous, with little suggestion that multivitamins and minerals are effective for reducing infection in the elderly. Also, the conclusions of our review potentially change, if the studies are invalid, since the case for concluding further research in this topic would be weak.

Implications for dealing with suspected fraudulent papers in future systematic reviews

It is unfortunate that the published concerns5,7,8,11,12 regarding the trials had escaped the attention of ourselves, the peer reviewer, and the editorial committee at the BMJ who dealt with our original paper.1 This situation did however mean we were not influenced by the mostly un-proven allegations. Since we used a search strategy designed to identify reports of randomised controlled trials, it is perhaps not surprising we missed the published discussion relating to them using this search. However in our re-assessment of the situation we acknowledge that Medline did provide a link that we missed to some of the correspondence in one of the trials’ 2 database entry (although this may have been added after we did our initial literature searches). Hence, although we hope that situations such as this are rare, we would like to flag up this issue for other systematic reviews so similar oversights are not made in the future. As well as using electronic databases to identify potential trials, and reading the abstracts available as an initial screening tool (as we did), the whole database record should be checked for links to publications relating to it, since this is information in the database record that is not available in the published paper. We strongly suggest this issue be highlighted in systematic review guidelines.13,14

As stated above, even if we had known about the allegations before publishing our initial paper, we would not have modified our primary analysis, since none of the three studies have been retracted, but we would have included the sensitivity analyses described here. We are unaware of any systematic review guidelines that provide guidance regarding how to deal with evidence from studies in which allegations of fraudulent research have been made. It would seem reasonable to assume that if a paper is formally retracted by a journal then it is no longer part of the published evidence base. It is less clear how unproven allegations should be dealt with. When we discovered the controversy that surrounded these studies, we wrote to the editors of the two journals in which the trial reports (Nutrition Research and the Lancet) were published to confirm that none of the papers had been retracted. They confirmed that these papers had not been retracted.

The most serious allegations suggest the three studies were essentially never done. This is well beyond the more usual concerns regarding study quality; that is, due to imperfect methodology, a study may be at risk of producing biased findings. Such accusations do not come into the usual quality appraisal of studies included in a systematic review. Using the Jadad score to measure quality, 15 both papers by Chandra scored 4 out of 5 while the study by Jain scored 2 out of 5. We considered the latter’s low score was at least in part attributable to the short length of the available trial report. Such instruments only really assess quality of reporting and not quality of the research and rely on the honesty of the trial investigators. Hence, the standard methods used for the selection of studies in a systematic review are inadequate under such extraordinary situations, which is why we have provided the sensitivity analysis described here.

Methods to detect fraudulent data in statistical datasets have been developed, but we know of no such methods to examine fraud across a series of meta-analyses of summary results. The fact that three out of a total of eight trials are under scrutiny, and these trials reported an outcome not reported by any other study, means using statistical methods to examine how extreme outlying trial results are is not possible. While the problem of "missing" studies, or publication bias, has received a lot of attention in meta-analysis, with methods developed to detect and address it, 16 the issue of having "extra" made-up studies is more unusual, with no formal methodology developed to address this situation.

Although we consider the sensitivity analysis conducted here to be very insightful, perhaps a more sophisticated alternative would be to use Bayesian statistical methods to provide a framework for including external information, such as expert beliefs regarding the likelihood that the studies are invalid.17 Unfortunately we did not have time to pursue this idea for this paper .

.Implications for research and practice

We appreciate it is unclear whose responsibility it is to further investigate and reach a verdict regarding the validity of the three trials. However, we do hope some verdict can be reached soon, since this sensitivity analysis has show that they have a crucial impact on the total evidence base and conclusions that can be drawn from it.

We would like to acknowledge Professor Saul Sternberg for initially bringing this issue to our attention, having read the original paper when first published on the BMJ website, and Dr Nicola Cooper for providing advice when writing this article.

Contributors: Analysis, interpretation, drafting and revising of the article were all carried out be AEK & AJS. AEK and AJS are joint first authors of the paper. AJS is the guarantor.Funding: None

Competing interests: Alex Sutton is an editor of the book, Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments, which will be published later this year for which he has received a payment and will receive a royalties.

Ethical approval: Not required.

1. El-Kadiki A, Sutton AJ. Role of multivitamins and mineral supplements in preventing infections in elderly people: systematic review and meta-analysis of randomised controlled trials. BMJ 2005;330:871-6

2. Chandra RK. Effect of vitamin and trace-element supplementation on immune responses and infecton in elderly subjects. Lancet 1992;340:1124-7.

3. Chandra RK. Influence of multinutrient supplement on immune responses and infection-related illness in 50-65 year old individuals. Nutrition Research 2002;22:5-11.

4. Jain AL. influence of vitamins and trace-elements on the incidence of respiratory infection in the elderly. Nutrition Research 2002;22:85-7.

5. Carpenter KJ, Roberts S, Sternberg S. Nutrition and immune function: a 1992 report. Lancet 2003; 361:2247.

6. Chandra RK. Author reply to Nutrition and immune function: a 1992 report. Lancet 2003; 361:2247-8.

7. Shenkin SD, Whiteman MC, Pattie A, Deary IJ. Supplementation and the elderly: dramatic results? Nutrition 2002;18:364-5.

8. Roberts S, Sternberg S. Do nutritional supplements improve cognitive function in the elderly? Nutrition 2003; 19:976-8.

9. Chandra RK. Effect of vitamin and trace-element supplementation on cognitive function in elderly subjects. Nutrition 2001;17:709-12.

10. Meguid MM. Retraction. Nutrition 2005;21:286.

11. White C. Three journals raise doubts on validity of Canadian studies. BMJ 2004;328:67.

12. Roberts SD, Sternberg S. How and what to investigate. BMJ – rapid response to "White C. Three journals raise doubts on validity of Canadian studies. BMJ 2004;328:67." http://bmj.bmjjournals.com/cgi/eletters/328/7431/67 [Accessed 20 April 2005]

13. Khan KS, Riet, G, Glanville J, Sowden AJ, Kleijen J. Undertaking systematic reviews of research on effectiveness. York: University of York, 2001.

14 The Cochrane Collaboration. Cochrane handbook for systematic reviews of interventions. Version 4.3.4 (March 2005) (http://www.cochrane.org/resources/handbook/handbook.pdf)

15. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ et al. Assessing the quality of reports of randomised clinical trials: Is blinding necessary? Controlled clinical trials 1996;17:1-12.

16. Sutton, A.J., Song, F., Gilbody, S.M. and Abrams, K.R. Modelling publication bias in meta-analysis: a review. Statistical methods in medical research 2000;9: 421-45.

17. Spiegelhalter DJ, Abrams KR, Myles JP. Bayesian approaches to clinical trials and health-care evaluation., Chichester: Wiley, 2003.

Figure 1 Random effects meta-analysis of outcome: mean difference in number of days of infection between multivitamin + supplement and placebo groups in 12 months assuming reported standard deviations for all trials were actually standard errors.


Footnote: the confidence interval round the pooled estimate is narrower after inflating the uncertainty in the study estimates. This may seem counter intuitive; however, this is because the statistical heterogeneity between study estimates is greatly reduced by such error inflation as this is reflected in the random effect pooled estimate.)

Figure 2 Fixed effect meta-analysis of outcome: incidence rate ratio for infection between multivitamin + supplement and placebo groups excluding the study by Jain