[notafraudster]

(See bio for my background) All of the replies you've gotten so far are very good and I upvoted all of them.

In particular:

cuchoi acknowledges the publication bias version of the risk here. Let's say your average effect is 1 unit with a confidence interval that is 0.9 units at a desired level of confidence. We can interpret this confident interval two different ways: one is that assuming 1 unit is the true effect, then repeated sampling would produce a sampling distribution of estimate effects that span 0.1 to 1.9 at the desired level of confidence. Another is assuming that, say, 0.1 was the true effect, effects as large as the one we see (1 unit) would occur a non-trivial portion of the time. Now, imagine many researchers do this experiment and the true effect is 0.1. Some researchers find negative effects, some find small effects that are not significant, others do larger studies and find small effects that are significant, others find larger effects. Now, imagine the journal will only publish effects that are both statistically significant and substantively interesting. The only person that submits for publishing is the version of the study that finds the large effect (1 unit). cuchoi is very correct to suggest that when your design can only find large effects, the published effect will likely be overestimated.

fpoling and sandgiant highlight the sensitivity risk argument. Suppose that the outcome is heavily sensitive to some confounders (socioeconomic status, nutrition, smoking status, race, etc.) And suppose poor people are slightly more likely to get treatment, just from coin flip chance. Because poverty correlates with both the effect and the probability of being treated (even though you tried to assign randomly), some of the visible effect is the relationship between poverty and treatment, not effect and outcome. There are designs other than simple randomization that try to explicitly deal with known confounders, but they can't deal with unknown confounders. Larger sample sizes mitigate the risk of imbalance of both known and unknown confounders.

Everyone is doing great!