[mlyle]

One reason we believe this is that we can typically detect virus quantities way too small to reliably replicate in culture, and believe that in vivo infection is even harder than in culture (because of innate immune response and the body not being composed just of the most susceptible cells).

You generally can't measure infectious dose directly without a highly unethical challenge study. You can sometimes know concentrations that did or didn't result in infection in various real world scenarios, and sometimes you have circumstantial evidence (e.g. you can know how much virus a person sheds, and what proportion of a room with certain ventilation quantities got infected).