[superqd]

So I studied as a scientist (physicist) and have a high bar for what constitutes a meaningful scientific study and scientific result. A food frequency questionnaire, of the sort this study's results are based on, means very little scientifically. No meaningful result can be drawn, and I consider it borderline immoral, given the type of affect this could have on people's behaviors, to assert a conclusion based on such poorly conducted research.

Asking people to quantify their eating habits over the previous several years and expect to have meaningful quantitative results is impossible. I can barely quantify my eating habits over the last few months. There are just far too many variables that cannot be controlled for in such research as to render any conclusion impossible.

We need meaningful nutritional research, desperately, and this just isn't it. It angers me deeply that this passes as science that will get reported on, used as "proof" in justifying recommendations, etc.

[cycomanic]

I'm also a physicist and I would first say that generally (except for some specific sub-fields) we are actually not well trained on experiments (like this) that require significant statistics.

In particular the law of large numbers applies (which you should know about as a physicist). I would argue that despite the noise in the reporting the conclusions from this study are likely much more significant than a tightly controlled randomized trial with a small group (~100 people).

Self reported questionnaires are certainly a well established method in these sort of studies and the short-comings are well understood, that does not mean their results are meaningless.

In particular, I would argue that many (most) people can give meaningful quantitative answers to their eating habits over several years as their habits rarely change. Certainly most people here would be able to answer approximately with how many meals per week they eat meat, how much vegetables they eat on average, how often they drink alcohol etc.. We are only interested in rough averages.

Regarding your argument that there are too many variables that can't be controlled for, ok lets hear them name at least 10 and make a valid statistical argument why they need to be controlled for in a sample size of ~80 000.

[superqd]

In this case, a "well established method" simply means common-place methodology, not necessarily a robust scientific protocol.

The problem is the humans. We are not reliable witnesses, even of our own pasts, and unless we are following very strict diets, most of us essentially follow a mostly random walk with our food intake across a narrow range of food groups (even caloric intake can be wildly inconsistent over time). I've met enough people who completely underestimate their consumption of desserts, e.g., that they mentally reduce the occurrence of such events in their personal historical narratives. That alone could put many from both low-carb vegans and low-carb meat eaters into the high-carb category.

Which is the point, the worst failure of this methodology is that there is no evidence that human memory, regarding food consumption, provides an accurate measure over long periods of time. If you cannot establish the trustworthiness of the most basic tool of your methodology, then almost by definition it's meaningless. The error in self reporting could be high enough such that the true classification of 80% of the low carb group is actually high carb. Or the vegan group could be nearly half high-carb, etc.

Without a thorough vetting of the accuracy of the instrumentation used in the research (FFQ in this case), we cannot know the margins of error. But we have apriori reasons for believing the accuracy to be significantly high. Adding more people to the group doesn't increase the accuracy in this case, it only increases the volume of the noise. Is the True Low Carb ratio of each group 10% or 90%? Who knows.

If we can't validate the accuracy of the tools, it makes the results essentially a random choice between the possible conclusions. It means they haven't disproven the null hypothesis.

Which is why I favor some form of sequestration of smaller populations of similar genetic makeup (multiple groups for each type of diet), etc. But that is mostly impractical.

[quanto]

I agree with your ethos; too often poor science, regardless of kind, is being passed as "soft" or "applied" scientific research.

That said, I do think even noisy studies (noisy as in inaccurate measurements) like this at least show that there is no glaring big correlation between, say, low-carb diet and mortality. If low-carb diet people started to die decades younger than average, then even noisy studies would show. In this sense, studies like this have some value, albeit little.

[superqd]

I just think there is far too much noise in this sort of study to draw any sort of conclusion. An effect has to be very strong to manifest in such noise. Yeah, if the difference in life expectancy was 3 decades, say, then perhaps it would show in such noise.

[didibus]

What do you consider the "noise" here? That people would inaccurately assess if they were eating low-carb or not?

[chx]

> We need meaningful nutritional research, desperately,

I am now wearing a constant glucose monitor and tried half a cup red lentil - semolina flour pasta. Should be OK, shouldn't it? My blood sugar jumped from below to 100 to above 140. My target is 120 and I am almost always below. Most often, well below. 140 is bad. I tried a "keto burrito". 135. I simply cut all starches and now I am good.

All in all, ever since I am wearing my CGM I can attest almost all nutritional advice given to diabetic people at least does not apply to me and I suspect I am not alone.

I am not surprised. At all. First of all, the entire science reeks because it's not genetics based. We know (although, do we? I no longer trust anything) at least lactose intolerance is caused by a gene. Is it too far fetched to say reaction to other foods could also be varied based on genetics? Second, there is a huge problem with scientific research itself. There are books and how to say, counter-research done on this. Basically, you take a grant, run a study and then produce some result, if your end result is nothing which is ought to be default, well that doesn't look good. So you produce some result... even if it's cherry picked data. And then some statisticians a decade later get hold of the raw data and prove you have found nothing. It happens all the time. Third, this is especially a problem with drugs but can happen with nutrition research too. Surely you have seen the ads where they are looking for volunteers? Well, who do you think they are going to get? No one produces a study with the title "our study among people experiencing homelessness and abject poverty"...

Anyways, I am vaccinated against covid. I hope this one is not botched. But nutritional advice? Nah. I refuse to talk to a nutritionist, I refuse to accept any advice, all of them are invalid -- for me.

[hattmall]

Kind of questionable perhaps, but a good place to do these studies would be on prisoners. You could potentially do it by simply monitoring their intake and not actually changing anything in the offering. It should be fairly reasonable to get a reliable intake report.

[SV_BubbleTime]

Proposal and other issues aside, I’m not sure you rely on food served to be food consumed. Prisons in most places still buy food at commissary.

But I’ll agree that you might be able to quantitative compare prison A vs prison B if you can get a control prison and account for differences. Just a diet of 20% more carbs between A/B would be interesting.

I imagine someone has “done science” on prisons in other areas already. Way more ethical than the typical proposals I see for prisoners/prisons on the internet.

[superqd]

Every time I've considered how to really do nutrition research, it essentially amounts to a prison, even if voluntary. You would need to be able to lock different populations away from any source of food than what you provide, and control their daily activities, etc, for at least 2 years, though you could likely get some results after only 6 months.

[superqd]

Yeah, the ethics would be tough, but essentially to really do nutrition research, you MUST control the food intake. Control for age, weight, calories and activity, and vary only the dietary makeup to varying degrees, etc.

[flukus]

> Yeah, the ethics would be tough

What sort of ethical issues are you thinking? I would have thought higher grade meals than you'd find in a standard prison would entice all the volunteers you could want. There's probably other groups were some good quality meals would be just as enticing, like college students.

A bigger practical issue in prisons would be things like the high prevalence of hepatitis and drug use along with all the confounding factors (like poverty) that lead them to end up in prison.

[ghancock]

In the US, federally funded research institutions have to follow certain rules when doing human subjects research. When I was a researcher the training I took said prisoners generally could not be used as subjects just because they were convenient, and that the two situations where they could typically be studied were cases where prisons, crime, or something else that was inherently specific to prisoners was being studied and cases where some condition very highly over represented among prisoners was what was being studied (the example given was HIV).

[superqd]

I just don't know how ethical it would be to force a particular diet that might affect someone's life expectancy without their consent. I mean, I suppose you could get volunteers...

[gremlinsinc]

Why not self-imposed prisoners... like at a rehab-type facility. You can leave whenever you want but you don't get paid...then pay them like $5k for 1 month, or 5k/month for however long the study goes... Then it's not prison -- it's controlled but "at will"

[lrem]

How do you think a good research setup for this would even look like?

And remember: anything that relies on current technology won't bring results until 2040s or 2050s.

[9wzYQbTYsAIc]

> Previous validation studies revealed reasonably good correlations between energy-adjusted nutrients assessed by the FFQ and multiple food records completed over the preceding year in both NHS(21) and HPFS(20).

[jallen_dot_dev]

> I consider it borderline immoral, given the type of affect this could have on people's behaviors, to assert a conclusion based on such poorly conducted research.

Eating more vegetables?

[vixen99]

Based on the massive amount of evidence linking 'chemicals from plants' (polyphenols, glycosinolates and others) with (in the main) markedly positive effects on human biochemistry & physiology, this appears to be a very good idea.

It's noteworthy that the 'five (vegetables/fruit) a day' mantra is a familiar one in the UK and I guess elsewhere. But rarely does anyone explain precisely why this is a recommendation given that it's quite possible to supply the required fat, protein, carbohydrate, vitamins and minerals without consuming significant amounts of fresh fruit and vegetables (using supplements).

[superqd]

That particular behavioral change likely wouldn't be bad itself, but that was more a rant on these sorts of studies overall. Poor research has real world consequences in this case (nutrition research), and we really need sound insight into what diet works best for the individual.

[CTmystery]

Even after you control for all the things you propose controlling for, you still have a huge confounding factor in genetic makeup. So then you need prisons with thousands of identical twins, each pair living out the same life in all ways but diet (which is meticulously recorded).

You're just not going to get an experiment like you can lay out in physics.

I think the point of the author mentioning major limitations in the paper is to acknowledge that this is sadly the best we've got to work with

[superqd]

Oh, yeah, the biochemistry of individuals is what makes so especially challenging, but not insurmountable. Maybe we are in a simulation to find out whether eating fish is good for you...

[didibus]

> Poor research has real world consequences in this case (nutrition research), and we really need sound insight into what diet works best for the individual

I think your claim is unsubstantiated. You're for some reason under the belief that nutrition has a large impact on people's life expectancy and health, but I don't know what accurate experiment or data you're basing this off, while also pointing out that this research was not good enough.

All the nutrition studies are like this, and so if you disregard them, you're left with no reason to believe nutrition even affect health or life expectancy for the average person in the first place.

[otabdeveloper4]

> We need meaningful nutritional research, desperately

Not gonna happen.

a) Too much vested interest everywhere. b) We can't experiment on humans for ethical reasons anyways.

[SV_BubbleTime]

It seems you want science. Sorry, the world seems to want headlines and diet fad debate.

We do desperately need actual research. I think the obesity crisis is far worse than most anyone realizes. I can think of a dozen reasons for people to actively ignore it.

[superqd]

Yeah, it bugs me deeply that this passes as science (and other such "research"). I realize it's difficult to conduct meaningful science in this area, given how different the biochemistry of every person is, but it's something I believe we can do. Given how important our health is, it's something I feel we really need to do.