Effects of a low carbohydrate diet on energy expenditure during weight loss maintenance: randomized trial

The substance of Professor Polychronakos' criticisms have been addressed in our manuscript, but we will take this opportunity to clarify a few points.

Non-adherence is a methodological limitation in any outpatient feeding study, and to some degree inpatient studies, too (if any participant is ever out of sight or allowed visitors). We considered this issue in depth in our Discussion section. We provide detailed sensitivity analyses that show the robustness of our primary outcome to substantial non-adherence, even with unfavorable assumptions (eTables 6 and 7).

We disagree that the low carbohydrate diet would be inherently most unpalatable. Process measures indicated high levels of acceptability of all test diets in our study. Indeed, drop-out rates from the low carbohydrate group were nominally lower than for the high carbohydrate group.

Polychronakos has reversed the strength of the findings between the Intention-to-Treat and Per Protocol analyses. The diet effect was stronger among the latter – including those who had objective evidence of better adherence. As Polychronakos recognizes, the opposite would be expected if non-adherence played a substantive role in the primary outcome. While baseline insulin secretion status could theoretically be a marker for selective non-adherence, a more plausible explanation is offered by the Carbohydrate-Insulin Model (that individuals with the highest insulin secretion on a high carbohydrate diet experience the most adverse metabolic consequences) [1].

Our preliminary assessment of energy intake was consistent with the primary outcome, but not statistically significant in the full Per Protocol group. We discussed reasons for the major imprecision in this measure in our manuscript. Interestingly, and as related to the point above, the difference in energy intake among the high insulin secretors was significant by diet, arguing against non-adherence as a material explanation. A variety of secondary outcomes of relevance to this discussion are the subject of ongoing analyses by our group.

The biological mechanisms relating nutrition to metabolism and body weight regulation are inherently complex. Every study addressing a major scientific debate in this field will likely have methodological and interpretive limitations. But cautious interpretation is warranted not only for our study, but also for the small, short-term feeding studies that dominant the literature (see our Introduction section for the special limitations of such studies).

We strongly disagree that “the only credible studies on nutritional intervention on humans are the ones performed on hospitalised subjects.” In the modern era, few volunteers would be willing to spend more than a few weeks in a hospital research unit. Even if volunteers could be found, the costs for a long-term, large-scale study would be prohibitive. Conservatively assuming $1000 per hospital day in the U.S., the food and residential costs alone for our study would exceed $40 million. A further concern with inpatient studies is the artificial environment and physical confinement, which would inevitably confound energy expenditure and other metabolic variables. Rather, high quality research with a variety of designs – ward studies, outpatient feeding trials, behavioral trials and prospective observational studies – will be needed to answer questions that have bedeviled the field of obesity for more than a century.

1. Ludwig DS, Ebbeling CB. The Carbohydrate-Insulin Model of Obesity: Beyond "Calories In, Calories Out". JAMA Intern Med. 2018;178(8):1098-1103

In the previous response, Mr. Moulthrop claims that he has identified a serious error in Dr. Hall's analysis. He suggests that because Dr. Ludwig's study "gives a highly significant result" and because it was the primary outcome … "random chance is very unlikely" and that Hall's reasoning for the result being random chance is not as compelling as Ludwig's interpretation.

There are several misconceptions here regarding P-values and hypothesis testing. The P-value is the probability of getting a test statistic at least as extreme as what was observed if every model assumption is correct [1]. Some key assumptions include:

- Randomization (sampling, assignment)
- No uncontrolled sources of systematic error in the results
- That the test model (often a null model) is correct

When we make these assumptions (that are often not correct) we are engaging in a thought experiment, so that all results that deviate from what we expect under the model are thought to be a result of pure random error. These are assumptions, not truths. Again, by this logic, all results under these assumptions are via random error.

A highly significant result indicates a few things.

- A dimensional violation specified by the test hypothesis has been detected by the test.
- The data are not very compatible with the test model and test hypothesis.
- In a Neyman-Pearson framework, we are allowed to reject the test hypothesis based on a prespecified alpha.

What a highly significant result does not indicate:

- That the results are meaningful
- That random chance is unlikely
- That the test hypothesis is most likely to be false (significance can indicate a violation of assumptions, it doesn't tell us *which* assumption has been violated)

Furthermore, an inflated familywise error rate (which is important in a Neyman-Pearson framework) does not simply occur as a result of multiple comparisons and is not solved by specifying the primary endpoint in advance. There are several other factors that can inflate this error rate, where it is far more likely in the long run to produce a significant result that is a false positive [2,3].

Rather than fixate only on P-values and whether or not they are "highly significant", we can interpret observed p (the realization of the random variable P)[4], as a continuous measure of compatibility and look to the width of the compatibility (confidence) interval to see what effects are compatible with the test model and its assumptions. Producing a likelihood function [5] and a P-value function [6,7] where every compatibility interval is graphed, will foster a much more nuanced interpretation of the data.

Trying to produce explanations as to why results gave "highly significant results" is highly misguided. Statistical significance and hypothesis testing have utility in certain areas where one is trying to automate decisions and control errors in the long run [8]. This may be useful in areas where replication is feasible, such as psychology [9]. In nutrition, where experiments similar to this are not always feasible, it may not hold as much utility.

The fixation on trying to explain highly significant results from a test of the null hypothesis truly is reflective of “cargo-cult statistics” [10].

References

1. Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol 2016;31:337–50. doi:10.1007/s10654-016-0149-3
2. Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci 2011;22:1359–66. doi:10.1177/0956797611417632
3. Wicherts JM, Veldkamp CLS, Augusteijn HEM, et al. Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking. Front Psychol 2016;7:1832. doi:10.3389/fpsyg.2016.01832
4. Greenland S. Valid P-values behave exactly as they should: Some misleading criticisms of P-values and their resolution with S-values. Am Stat 2018;18.
5. Royall R. Statistical evidence: a likelihood paradigm. Routledge 2017. https://www.taylorfrancis.com/books/9781351414562
6. Poole C. Beyond the confidence interval. Am J Public Health 1987;77:195–9.https://www.ncbi.nlm.nih.gov/pubmed/3799860
7. Rothman KJ, Greenland S, Lash TL, et al. Modern epidemiology. Published Online First: 2008.https://www.annemergmed.com/article/S0196-0644(08)01394-2/abstract
8. Lehmann EL, Romano JP. Testing Statistical Hypotheses. Springer Science & Business Media 2006. https://market.android.com/details?id=book-K6t5qn-SEp8C
9. Lakens D, Adolfi FG, Albers CJ, et al. Justify your alpha. Nature Human Behaviour 2018;2:168–71. doi:10.1038/s41562-018-0311-x
10. Stark PB, Saltelli A. Cargo-cult statistics and scientific crisis. Significance 2018;15:40–3. doi:10.1111/j.1740-9713.2018.01174.x

We would like to clarify procedures related to data acquisition and maintenance of the study group assignment masking (blind).

The final analysis plan, with a change in specification of the baseline for total energy expenditure (TEE, the primary outcome) was approved by the institutional review board and posted in September 2017, prior to unmasking diet group assignment, as stated in the main manuscript.

At that time, the final outcome data for the third and main cohort (comprising 50% of participants) had not yet been received by the Boston investigators. However, data for the prior cohorts had been received.

Therefore, the relevant sentence in the online Supplement (page 16) would more accurately include the word “complete” as follows (1): “Submitted Final Data Analysis Plan (version 2017.06.14) to IRB and obtained approval prior to receiving the complete primary outcome data and breaking the randomization blind."

Dr. William Wong in Houston sent raw isotopic data periodically in batches to Boston during the study. Data cleaning and modeling (i.e., curve fitting to allow for calculation of the primary outcome) were performed in late 2017 after posting the final analysis plan. The statistician broke the blind and delivered the initial results to the principal investigators in January 2018. The statistician maintained the only depository for the primary outcome until posting the full database with publication of the manuscript in November 2018.

1. Supplemental Information (online): https://www.bmj.com/content/bmj/suppl/2018/11/13/bmj.k4583.DC1/ebbc04619...

The question of whether the ratio of dietary carbohydrates to fat substantially impacts total energy expenditure (TEE) or body fat has been investigated for decades, with most studies pointing to no clinically meaningful effect 1. However, a recent study by Ebbeling et al. reported substantial differences in TEE between diets varying in their ratio of carbohydrate to fat 2.

The original pre-registered statistical analysis plan for the primary study outcome of Ebbeling et al. addressed the question of whether the reduction in TEE during weight loss maintenance compared to the pre-weight loss baseline depended on the dietary carbohydrate to fat ratio – a design similar to a previous study by many of the same authors 3. However, the final analysis plan was modified to make the diet comparisons with the TEE measurements collected in the immediate post-weight loss period rather than at the pre-weight loss baseline. As fully described in a manuscript available on the bioRxiv pre-print server (https://www.biorxiv.org/content/early/2018/11/28/476655), reanalyzing the data according to the original analysis plan of Ebbeling et al. found that the TEE differences were no longer statistically significant between the diet groups and the nominal diet differences of ~100 kcal/d were much smaller than the ~250 kcal/d differences reported in the final publication. In other words, when conducting the analysis originally planned by the authors we found that the significant increases in TEE with the low carbohydrate diet that were reported by Ebbeling et al. disappeared. Furthermore, the significant TEE effect modification by baseline insulin secretion also disappeared.

As justification for our reanalysis, we note that most of the history of the study (7 of 8 versions of the protocol spanning from 2014-2016) the planned primary outcome calculations used the pre-weight loss TEE baseline as the anchor point for the subsequent diet comparisons during weight loss maintenance. Prior to unmasking the randomization blind, but after all cohorts had completed the trial, the final protocol amendment in 2017 altered the previously planned statistical analysis to use the post-weight loss TEE measurement as the anchor point to make the subsequent diet comparisons.

The reasons for the change in the analysis plan were not provided in the protocol amendment or the final statistical analysis plan, but the Supplemental Materials in the final publication provided three reasons. First, the post-weight loss TEE measurement was chosen as the new anchor point because it occurred closer to the point of diet randomization. Second, the pre-weight loss TEE measurement would be “strongly confounded by weight loss”. How this might happen and why the post weight loss measure would not be similarly affected is difficult to imagine. Finally, Ebbeling et al. argued that the pre-weight loss baseline would have been inappropriate because the TEE measurements were insufficiently accurate or precise and therefore the study would be under-powered. However, the power calculations in the protocol were based on TEE data from a pilot study using the pre-weight loss TEE measurements as the basis for comparing how different diets affected the absolute reduction in TEE during weight loss maintenance 3. The pilot study did not measure TEE in the period immediately post-weight loss and therefore could not have been used to power the recent study in question.

Despite a request by the BMJ Editors to report the results of their original analysis plan, Ebbeling et al. refused because they were “concerned that the additional analysis would provide no meaningful biological insights – that is, no useful information about the nature of the relationship between dietary composition and energy expenditure. Rather, inclusion of the additional analysis would tend to elevate and give undue attention to an error, and therefore potentially cause confusion.”

We believe that the revised analysis plan has caused confusion and that the original statistical analysis plan that used pre-weight loss TEE as the anchor point is preferable for several reasons. First, it specifically addresses the question of whether the typical reduction in TEE that accompanies maintenance of lost weight depends on the carbohydrate to fat ratio of the weight loss maintenance diet. Second, the revised plan is potentially confounded by the substantial adaptive thermogenesis that occurs immediately post-weight loss that typically becomes less severe after a period of energy balance and weight loss maintenance 4 5. Finally, the pre-weight loss baseline TEE measurements were obtained in the situation where the doubly labeled water method is routinely employed: free-living people maintaining their habitual weight. Ideally, a post-weight loss TEE measurement should have first stabilized subjects at the lower body weight for several weeks prior to dosing with doubly labeled water 6. In contrast, the post-weight loss TEE measurements conducted by Ebbeling et al. were obtained during the same 2-week weight stabilization period when diet calories were being progressively increased at a rate determined by each individual subject’s recent rate of weight loss. While the doubly labeled water method generally provides a robust and valid estimate of TEE, this situation of simultaneous refeeding immediately post-weight loss potentially introduces uncertainty into the conversion of CO2 production into TEE. For example, the daily respiratory quotient during this period was clearly not equal to the food quotient as was assumed by Ebbeling et al. While such an effect can be appropriately modeled 7, this was not done in their TEE calculations.

In conclusion, when analyzed using the original statistical plan that was not confounded by the immediate post-weight loss period, the data of Ebbeling et al. do not support the conclusion that the ratio of dietary carbohydrate to fat affects the reduction in TEE during weight loss maintenance. While there are many reasons people could benefit from consuming healthy low carbohydrate diets 8, such diets are unlikely to help offset the usual reduction in TEE during maintenance of lost weight.

References
1. Hall KD, Guo J. Obesity Energetics: Body Weight Regulation and the Effects of Diet Composition. Gastroenterology 2017;152(7):1718-27 e3. doi: 10.1053/j.gastro.2017.01.052 [published Online First: 2017/02/15]
2. Ebbeling CB, Feldman HA, Klein GL, et al. Effects of a low carbohydrate diet on energy expenditure during weight loss maintenance: randomized trial. Bmj 2018;363:k4583. doi: 10.1136/bmj.k4583 [published Online First: 2018/11/16]
3. Ebbeling CB, Swain JF, Feldman HA, et al. Effects of dietary composition on energy expenditure during weight-loss maintenance. Jama 2012;307(24):2627-34.
4. Hall KD. Computational Modeling of Energy Metabolism and Body Composition Dynamics. In: Krentz AW, Heinemann L, Hompesch M, eds. Translational Research Methods for Diabetes, Obesity and Cardiometabolic Drug Development. London: Springer-Verlag 2015:265-82.
5. Weinsier RL, Nagy TR, Hunter GR, et al. Do adaptive changes in metabolic rate favor weight regain in weight-reduced individuals? An examination of the set-point theory. Am J Clin Nutr 2000;72(5):1088-94.
6. Bhutani S, Racine N, Shriver T, et al. Special Considerations for Measuring Energy Expenditure with Doubly Labeled Water under Atypical Conditions. Journal of obesity & weight loss therapy 2015;5(Suppl 5) doi: 10.4172/2165-7904.S5-002 [published Online First: 2016/03/11]
7. Hall KD, Guo J, Chen KY, et al. Methodologic Issues in Doubly Labeled Water Measurements of Energy Expenditure During Very Low-Carbohydrate Diets. bioRxiv 2018 doi: 10.1101/403931
8. Hall KD, Chung ST. Low-carbohydrate diets for the treatment of obesity and type 2 diabetes. Curr Opin Clin Nutr Metab Care 2018;21(4):308-12. doi: 10.1097/mco.0000000000000470 [published Online First: 2018/04/21]