[aresant]

A very cool idea @ the high level, but one fatal flaw:

"At the end of each day you will fill out a short survey with a few questions evaluating the effects of the pills."

Self-reporting is an unsuitable mechanism to draw out scientific results.

There's an excellent detailed explanation available (1) but in TL;DR here are four of the most compelling factors at work:

1 - Honesty/Image management

2 - Introspective ability

3 - Understanding / Question Interpretation

4 - Response bias

Take Image Management & Response Bias - participants know that they will be able to see their results vs. the control group and it's not a leap to realize how easily our ego and even subconscious need for validation could dramatically skew the full study results.

(1) http://www.sciencebrainwaves.com/the-dangers-of-self-report/

[j42]

I'm not sure I see that as a fatal flaw... ultimately, the user is sitting in front of a digital box that can guide and prompt them in all the same ways a researcher can--the only limit is in performing tests that require a medical professional to assess biomarkers. If the pace of medical device development continues, it's even reasonable to think something like a Theranos-that-works could commoditize the process while being intrinsically tamper-resistant.

Regardless, users can be prompted to perform any software action (knowingly or unknowingly, to affect bias) and that action can be measured by the system. It may so happen that every critical measurement occurs unbeknownst to the user, before they self-report anything (if at all). As we are currently undergoing a period of sensor-proliferation (fitness/health devices, wearables, internet of things, etc...) it's not unrealistic to think we will soon be able to instantly correlate data from a smartphone camera, blood/tissue, and the cloud.

Now there's always the problem of intentional fraud/deception, but I think the aggregate nature solves that problem. A small percentage will try to "break" the system, and that small percentage will never surpass a critical threshold with enough volume. In terms of ML/SVM's, we're now very good about filtering outliers or "misrepresented data"... while the responsibility is on you to develop a reliable classifier (for data-consistency more than arbitrary measurement), I imagine at scale you could infer trends with the same relative accuracy of traditional academia and research.

It's a really fascinating new direction--even if only an adjunct to traditional research--and I'll definitely be keeping an eye on the project.

[chrisrxth]

Just read through that. I agree with the points about self-reporting being unreliable.

Even with the bias of self reporting, differences between real and placebo will still be meaningful. You can't entirely deceive yourself if you don't know what you just took.

Also, there is some degree of honesty required for this experiment. If someone really wanted to, they could open up the pills and taste them to see if it's placebo or not. They could also intentionally lie. But the goal here is to learn about yourself through experimentation, so I expect most people that participate to have some level of desire for truth and honesty. I would also argue that while these biases do exist, even large randomized clinical trials have the same pitfalls, so we are not necessarily worse off than what's already out there.

[meta_AU]

AFAICT, there isn't a control group. You have 4 weeks of pills and each week it could be real or fake. So you are comparing against yourself.