BMJ 2003;326:219 ( 25 January )

Education and debate

Statistics Notes

Interaction revisited: the difference between two estimates

Douglas G Altman, professor of statistics in medicine ^a, J Martin Bland, professor of medical statistics ^b. 

^a Cancer Research UK Medical Statistics Group, Centre for Statistics in Medicine, Institute for Health Sciences, Oxford OX3 7LF, ^b Department of Public Health Sciences, St George's Hospital Medical School, London SW17 0RE

Correspondence to: D G Altman doug.altman{at}cancer.org.uk

We often want to compare two estimates of the same quantity derived from separate analyses. Thus we might want to compare the treatment effect in subgroups in a randomised trial, such as two age groups. The term for such a comparison is a test of interaction. In earlier Statistics Notes we discussed interaction in terms of heterogeneity of treatment effect.^1-3 Here we revisit interaction and consider the concept more generally.

The comparison of two estimated quantities, such as means or proportions, each with its standard error, is a general method that can be applied widely. The two estimates should be independent, not obtained from the same individualsexamples are the results from subgroups in a randomised trial or from two independent studies. The samples should be large. If the estimates are E₁ and E₂ with standard errors SE(E₁) and SE(E₂), then the difference d=E₁E₂ has standard error SE(d)=[SE(E₁)² + SE(E₂)²] (that is, the square root of the sum of the squares of the separate standard errors). This formula is an example of a well known relation that the variance of the difference between two estimates is the sum of the separate variances (here the variance is the square of the standard error). Then the ratio z=d/SE(d) gives a test of the null hypothesis that in the population the difference d is zero, by comparing the value of z to the standard normal distribution. The 95% confidence interval for the difference is d1.96SE(d) to d+1.96SE(d).

We illustrated this for means and proportions,³ although we did not show how to get the standard error of the difference. Here we consider comparing relative risks or odds ratios. These measures are always analysed on the log scale because the distributions of the log ratios tend to be those closer to normal than of the ratios themselves.

In a meta-analysis of non-vertebral fractures in randomised trials of hormone replacement therapy the estimated relative risk from 22 trials was 0.73 (P=0.02) in favour of hormone replacement therapy.⁴ From 14 trials of women aged on average <60 years the relative risk was 0.67 (95% confidence interval 0.46 to 0.98; P=0.03). From eight trials of women aged 60 the relative risk was 0.88 (0.71 to 1.08; P=0.22). In other words, in younger women the estimated treatment benefit was a 33% reduction in risk of fracture, which was statistically significant, compared with a 12% reduction in older women, which was not significant. But are the relative risks from the subgroups significantly different from each other? We show how to answer this question using just the summary data quoted.

Because the calculations were made on the log scale, comparing the two estimates is complex (see table). We need to obtain the logs of the relative risks and their confidence intervals (rows 2 and 4).⁵ As 95% confidence intervals are obtained as 1.96 standard errors either side of the estimate, the SE of each log relative risk is obtained by dividing the width of its confidence interval by 2×1.96 (row 6). The estimated difference in log relative risks is d=E₁ E₂=0.2726 and its standard error 0.2206 (row 8). From these two values we can test the interaction and estimate the ratio of the relative risks (with confidence interval). The test of interaction is the ratio of d to its standard error: z=0.2726/0.2206=1.24, which gives P=0.2 when we refer it to a table of the normal distribution. The estimated interaction effect is exp(0.2726)=0.76. (This value can also be obtained directly as 0.67/0.88=0.76.) The confidence interval for this effect is 0.7050 to 0.1598 on the log scale (row 9). Transforming back to the relative risk scale, we get 0.49 to 1.17 (row 12). There is thus no good evidence to support a different treatment effect in younger and older women.

View this table: [in this window] [in a new window]   Calculations for comparing two estimated relative risks

The same approach is used for comparing odds ratios. Comparing means or regression coefficients is simpler as there is no log transformation. The two estimates must be independent: the method should not be used to compare a subset with the whole group, or two estimates from the same patients.

There is limited power to detect interactions, even in a meta-analysis combining the results from several studies. As this example illustrates, even when the two estimates and P values seem very different the test of interaction may not be significant. It is not sufficient for the relative risk to be significant in one subgroup and not in another. Conversely, it is not correct to assume that when two confidence intervals overlap the two estimates are not significantly different.⁶ Statistical analysis should be targeted on the question in hand, and not based on comparing P values from separate analyses.²

References

1.	Altman DG, Matthews JNS. Interaction 1: Heterogeneity of effects. BMJ 1996; 313: 486[Free Full Text].
2.	Matthews JNS, Altman DG. Interaction 2: Compare effect sizes not P values. BMJ 1996; 313: 808[Free Full Text].
3.	Matthews JNS, Altman DG. Interaction 3: How to examine heterogeneity. BMJ 1996; 313: 862[Free Full Text].
4.	Torgerson DJ, Bell-Syer SEM. Hormone replacement therapy and prevention of nonvertebral fractures. A meta-analysis of randomized trials. JAMA 2001; 285: 2891-2897[Abstract/Free Full Text].
5.	Bland JM, Altman DG. Logarithms. BMJ 1996; 312: 700[Free Full Text].
6.	Bland M, Peacock J. Interpreting statistics with confidence. Obstetrician and Gynaecologist (in press).

---

© 2003 BMJ Publishing Group Ltd
CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Related Articles

Inequity of access to investigation and effect on clinical outcomes: prognostic study of coronary angiography for suspected stable angina pectoris

Neha Sekhri, Adam Timmis, Ruoling Chen, Cornelia Junghans, Niamh Walsh, Justin Zaman, Sandra Eldridge, Harry Hemingway, and Gene Feder
BMJ 2008 336: 1058-1061. [Abstract] [Full Text] [PDF]

Antithrombin III in critically ill patients: systematic review with meta-analysis and trial sequential analysis

Arash Afshari, Jørn Wetterslev, Jesper Brok, and Ann Møller
BMJ 2007 335: 1248-1251. [Abstract] [Full Text] [PDF]

Misuse of confidence intervals threatens conclusions

Christopher J Cates
BMJ 2007 334: 1020. [Extract] [Full Text] [PDF]

Oral decontamination for prevention of pneumonia in mechanically ventilated adults: systematic review and meta-analysis

Ee Yuee Chan, Annie Ruest, Maureen O Meade, and Deborah J Cook
BMJ 2007 334: 889. [Abstract] [Full Text] [PDF]

Meta-analysis of frusemide to prevent or treat acute renal failure

Kwok M Ho and David J Sheridan
BMJ 2006 333: 420. [Abstract] [Full Text] [PDF]

Antibiotics and acute purulent rhinitis: There is no significant difference between antibiotics

Christopher J Cates
BMJ 2006 333: 395. [Extract] [Full Text]

Effect of hepatitis B immunisation in newborn infants of mothers positive for hepatitis B surface antigen: systematic review and meta-analysis

Chuanfang Lee, Yan Gong, Jesper Brok, Elizabeth H Boxall, and Christian Gluud
BMJ 2006 332: 328-336. [Abstract] [Full Text] [PDF]

Dummies and SIDS: Low response rates generate considerable uncertainty

Christopher J Cates
BMJ 2006 332: 178. [Extract] [Full Text]

Treatment of ocular hypertension and open angle glaucoma: meta-analysis of randomised controlled trials

Philip C Maier, Jens Funk, Guido Schwarzer, Gerd Antes, and Yngve T Falck-Ytter
BMJ 2005 331: 134. [Abstract] [Full Text] [PDF]

Psychosocial and psychological interventions for prevention of postnatal depression: systematic review

Cindy-Lee Dennis
BMJ 2005 331: 15. [Abstract] [Full Text] [PDF]

This article has been cited by other articles:

• Ferrante, G., Locca, D., Barlis, P. (2008). Is There Enough Evidence to Support Use of N-Acetylcysteine in Contrast-Induced Nephropathy?. ANN INTERN MED 149: 214-215 [Full text]  
• Moller, C. H., Penninga, L., Wetterslev, J., Steinbruchel, D. A., Gluud, C. (2008). Clinical outcomes in randomized trials of off- vs. on-pump coronary artery bypass surgery: systematic review with meta-analyses and trial sequential analyses. Eur Heart J 0: ehn335v1-16 [Abstract] [Full text]  
• Sekhri, N., Timmis, A., Chen, R., Junghans, C., Walsh, N., Zaman, J., Eldridge, S., Hemingway, H., Feder, G. (2008). Inequity of access to investigation and effect on clinical outcomes: prognostic study of coronary angiography for suspected stable angina pectoris. BMJ 336: 1058-1061 [Abstract] [Full text]  
• Bennett, C. L., Silver, S. M., Djulbegovic, B., Samaras, A. T., Blau, C. A., Gleason, K. J., Barnato, S. E., Elverman, K. M., Courtney, D. M., McKoy, J. M., Edwards, B. J., Tigue, C. C., Raisch, D. W., Yarnold, P. R., Dorr, D. A., Kuzel, T. M., Tallman, M. S., Trifilio, S. M., West, D. P., Lai, S. Y., Henke, M. (2008). Venous Thromboembolism and Mortality Associated With Recombinant Erythropoietin and Darbepoetin Administration for the Treatment of Cancer-Associated Anemia. JAMA 299: 914-924 [Abstract] [Full text]  
• Svilaas, T., Vlaar, P. J., van der Horst, I. C., Diercks, G. F.H., de Smet, B. J.G.L., van den Heuvel, A. F.M., Anthonio, R. L., Jessurun, G. A., Tan, E.-S., Suurmeijer, A. J.H., Zijlstra, F. (2008). Thrombus Aspiration during Primary Percutaneous Coronary Intervention. NEJM 358: 557-567 [Abstract] [Full text]  
• Afshari, A., Wetterslev, J., Brok, J., Moller, A. (2007). Antithrombin III in critically ill patients: systematic review with meta-analysis and trial sequential analysis. BMJ 335: 1248-1251 [Abstract] [Full text]  
• MEEWISSE, M.-L., REITSMA, J. B., DE VRIES, G.-J., GERSONS, B. P. R., OLFF, M. (2007). Cortisol and post-traumatic stress disorder in adults: Systematic review and meta-analysis. Br. J. Psychiatry 191: 387-392 [Abstract] [Full text]  
• Ho, K. M, Sheridan, D. J, Paterson, T. (2007). Use of intravenous magnesium to treat acute onset atrial fibrillation: a meta-analysis. Heart 93: 1433-1440 [Abstract] [Full text]  
• Bonovas, S., Filioussi, K., Flordellis, C. S., Sitaras, N. M. (2007). Statins and the Risk of Colorectal Cancer: A Meta-Analysis of 18 Studies Involving More Than 1.5 Million Patients. JCO 25: 3462-3468 [Abstract] [Full text]  
• Kuderer, N. M., Dale, D. C., Crawford, J., Lyman, G. H. (2007). Impact of Primary Prophylaxis With Granulocyte Colony-Stimulating Factor on Febrile Neutropenia and Mortality in Adult Cancer Patients Receiving Chemotherapy: A Systematic Review. JCO 25: 3158-3167 [Abstract] [Full text]  
• Lansberg, M. G., Thijs, V. N., Hamilton, S., Schlaug, G., Bammer, R., Kemp, S., Albers, G. W., on behalf of the DEFUSE Investigators, (2007). Evaluation of the Clinical-Diffusion and Perfusion-Diffusion Mismatch Models in DEFUSE. Stroke 38: 1826-1830 [Abstract] [Full text]  
• May, J., Evans, J. A., Timmann, C., Ehmen, C., Busch, W., Thye, T., Agbenyega, T., Horstmann, R. D. (2007). Hemoglobin Variants and Disease Manifestations in Severe Falciparum Malaria. JAMA 297: 2220-2226 [Abstract] [Full text]  
• Cates, C. J (2007). Misuse of confidence intervals threatens conclusions. BMJ 334: 1020-1020 [Full text]  
• Chan, E. Y., Ruest, A., Meade, M. O, Cook, D. J (2007). Oral decontamination for prevention of pneumonia in mechanically ventilated adults: systematic review and meta-analysis. BMJ 334: 889-889 [Abstract] [Full text]  
• Bjelakovic, G., Nikolova, D., Gluud, L. L., Simonetti, R. G., Gluud, C. (2007). Mortality in Randomized Trials of Antioxidant Supplements for Primary and Secondary Prevention: Systematic Review and Meta-analysis. JAMA 297: 842-857 [Abstract] [Full text]  
• Gami, A. S., Witt, B. J., Howard, D. E., Erwin, P. J., Gami, L. A., Somers, V. K., Montori, V. M. (2007). Metabolic Syndrome and Risk of Incident Cardiovascular Events and Death: A Systematic Review and Meta-Analysis of Longitudinal Studies. J Am Coll Cardiol 49: 403-414 [Abstract] [Full text]  
• Lim, E., Carballo, S., Cornelissen, J., Ali, Z. A., Grignani, R., Bellm, S., Large, S. (2007). Dose-Related Efficacy of Aspirin After Coronary Surgery in Patients With PlA2 Polymorphism (NCT00262275). Ann. Thorac. Surg. 83: 134-138 [Abstract] [Full text]  
• Schneider, V., Levesque, L. E., Zhang, B., Hutchinson, T., Brophy, J. M. (2006). Association of Selective and Conventional Nonsteroidal Antiinflammatory Drugs with Acute Renal Failure: A Population-based, Nested Case-Control Analysis. Am J Epidemiol 164: 881-889 [Abstract] [Full text]  
• Ho, K. M, Sheridan, D. J (2006). Meta-analysis of frusemide to prevent or treat acute renal failure. BMJ 333: 420- [Abstract] [Full text]  
• Cates, C. J (2006). Antibiotics and acute purulent rhinitis: There is no significant difference between antibiotics. BMJ 333: 395-395 [Full text]  
• Ho, K. M., Litton, E. (2006). Use of chlorhexidine-impregnated dressing to prevent vascular and epidural catheter colonization and infection: a meta-analysis. J Antimicrob Chemother 58: 281-287 [Abstract] [Full text]  
• Casas, J. P, Shah, T., Cooper, J., Hawe, E., McMahon, A. D, Gaffney, D., Packard, C. J, O'Reilly, D. S, Juhan-Vague, I., Yudkin, J. S, Tremoli, E., Margaglione, M., Di Minno, G., Hamsten, A., Kooistra, T., Stephens, J. W, Hurel, S. J, Livingstone, S., Colhoun, H. M, Miller, G. J, Bautista, L. E, Meade, T., Sattar, N., Humphries, S. E, Hingorani, A. D (2006). Insight into the nature of the CRP-coronary event association using Mendelian randomization. Int J Epidemiol 35: 922-931 [Abstract] [Full text]  
• de Craen, A. J.M., Stott, D. J., Westendorp, R. G.J., Khare, A., Lopez, M., Gogtay, J., Quinlivan, E. P., Gregory, J. F. III, Refsum, H., Smith, A. D., Wang, X., Demirtas, H., Xu, X., Tomlinson, D. R., Lang, D., Lewis, M. J., Lonn, E., the HOPE-2 Investigators, , Bonaa, K. H., Tverdal, A., Ueland, P. M. (2006). Homocysteine, B vitamins, and cardiovascular disease.. NEJM 355: 205-206 [Full text]  
• Salpeter, S. R., Buckley, N. S., Ormiston, T. M., Salpeter, E. E. (2006). Meta-Analysis: Effect of Long-Acting {beta}-Agonists on Severe Asthma Exacerbations and Asthma-Related Deaths. ANN INTERN MED 144: 904-912 [Abstract] [Full text]  
• Quiroga, I., McShane, P., Koo, D. D. H., Gray, D., Friend, P. J., Fuggle, S., Darby, C. (2006). Major effects of delayed graft function and cold ischaemia time on renal allograft survival. Nephrol Dial Transplant 21: 1689-1696 [Abstract] [Full text]  
• Sood, A, Ford, E S, Camargo, C A Jr (2006). Association between leptin and asthma in adults. Thorax 61: 300-305 [Abstract] [Full text]  
• Park-Wyllie, L. Y., Juurlink, D. N., Kopp, A., Shah, B. R., Stukel, T. A., Stumpo, C., Dresser, L., Low, D. E., Mamdani, M. M. (2006). Outpatient Gatifloxacin Therapy and Dysglycemia in Older Adults. NEJM 354: 1352-1361 [Abstract] [Full text]  
• Lee, C., Gong, Y., Brok, J., Boxall, E. H, Gluud, C. (2006). Effect of hepatitis B immunisation in newborn infants of mothers positive for hepatitis B surface antigen: systematic review and meta-analysis. BMJ 332: 328-336 [Abstract] [Full text]  
• Murphy, V E, Clifton, V L, Gibson, P G (2006). Asthma exacerbations during pregnancy: incidence and association with adverse pregnancy outcomes. Thorax 61: 169-176 [Abstract] [Full text]  
• Cates, C. J (2006). Dummies and SIDS: Low response rates generate considerable uncertainty. BMJ 332: 178-178 [Full text]  
• Bonovas, S., Filioussi, K., Tsavaris, N., Sitaras, N. M. (2005). Use of Statins and Breast Cancer: A Meta-Analysis of Seven Randomized Clinical Trials and Nine Observational Studies. JCO 23: 8606-8612 [Abstract] [Full text]  
• Maier, P. C, Funk, J., Schwarzer, G., Antes, G., Falck-Ytter, Y. T (2005). Treatment of ocular hypertension and open angle glaucoma: meta-analysis of randomised controlled trials. BMJ 331: 134- [Abstract] [Full text]  
• Dennis, C.-L. (2005). Psychosocial and psychological interventions for prevention of postnatal depression: systematic review. BMJ 331: 15- [Abstract] [Full text]  
• Murphy, V. E., Gibson, P. G., Smith, R., Clifton, V. L. (2005). Asthma during pregnancy: mechanisms and treatment implications. Eur Respir J 25: 731-750 [Abstract] [Full text]  
• Greer, J. B., Modugno, F., Allen, G. O., Ness, R. B. (2005). Androgenic Progestins in Oral Contraceptives and the Risk of Epithelial Ovarian Cancer. Obstet Gynecol 105: 731-740 [Abstract] [Full text]  
• Eskenazi, B., Kidd, S.A., Marks, A.R., Sloter, E., Block, G., Wyrobek, A.J. (2005). Antioxidant intake is associated with semen quality in healthy men. Hum Reprod 20: 1006-1012 [Abstract] [Full text]  
• Bruynesteyn, K, Wanders, A, Landewe, R, van der Heijde, D (2004). How the type of risk reduction influences required sample sizes in randomised clinical trials. Ann Rheum Dis 63: 1368-1371 [Abstract] [Full text]  
• Pedone, C., Pahor, M., Carosella, L., Bernabei, R., Carbonin, P., for the GIFA Investigators, (2004). Use of Angiotensin-Converting Enzyme Inhibitors in Elderly People With Heart Failure: Prevalence and Outcomes. J. Gerontol. A Biol. Sci. Med. Sci. 59: M716-M721 [Abstract] [Full text]  
• Als-Nielsen, B., Gluud, L. L, Gluud, C. (2004). Non-absorbable disaccharides for hepatic encephalopathy: systematic review of randomised trials. BMJ 328: 1046- [Abstract] [Full text]  
• Habib, A. S., El-Moalem, H. E., Gan, T. J. (2004). The efficacy of the 5-HT3 receptor antagonists combined with droperidol for PONV prophylaxis is similar to their combination with dexamethasone. A meta-analysis of randomized controlled trials: [L'efficacite de la combinaison des antagonistes des recepteurs 5-HT3 et du droperidol ou de la dexame- thasone est similaire pour prevenir les NVPO. Une meta-analyse d'essais randomises et controles]. Canadian J. Anesthesia 51: 311-319 [Abstract] [Full text]  
• Lim, E., Ali, Z., Ali, A., Routledge, T., Edmonds, L., Altman, D. G, Large, S. (2003). Indirect comparison meta-analysis of aspirin therapy after coronary surgery. BMJ 327: 1309- [Abstract] [Full text]  
• Chinn, S (2003). Asthma and obesity: where are we now?. Thorax 58: 1008-1010 [Full text]  
• Hind, D., Calvert, N., McWilliams, R., Davidson, A., Paisley, S., Beverley, C., Thomas, S. (2003). Ultrasonic locating devices for central venous cannulation: meta-analysis. BMJ 327: 361- [Abstract] [Full text]  
• McAlister, F. A., Straus, S. E., Sackett, D. L., Altman, D. G. (2003). Analysis and Reporting of Factorial Trials: A Systematic Review. JAMA 289: 2545-2553 [Abstract] [Full text]  

Rapid Responses:

Read all Rapid Responses

Another evidence to report results in terms of relative risks

Jose L. M. Mesa
bmj.com, 5 Feb 2003 [Full text]