[Yvain]

What kind of response bias are you expecting here and how would you fix it?

I mean, obviously there is a bias in favor of people who use more nootropics and find nootropics more effective. But since all I was doing was ranking nootropics relative to one another, that should mostly cancel out. I'm not sure why respondents would, for example, be expected to be a group disproportionately likely to have a stronger response to modafinil relative to bacopa compared to the general population.

More pressingly, I'm not sure how I can avoid it besides gathering hundreds of people together and forcing them to try nootropics one by one in controlled conditions, for which I don't have the resources. Even professional studies that select their subjects ahead of time have atrociously high dropout rates. If you have an implementable idea for avoiding response bias on next year's survey I'm happy to go ahead with it, but "not collect any survey data at all because available methodology isn't perfect" seems strictly inferior.

I feel like I hedged myself pretty appropriately: " we can't get good data without way more effort and resources than anyone is willing to put in, but if we ran a survey we could at least get better anecdata", "This survey does not intend to rectify [the confusion as to whether nootropics work at all]", "a pessimist might say that [this is] extracting patterns from random noise", et cetera. But given the choice between giving up because it wasn't perfect and going ahead with hedges, I figured I might as well go ahead.