That is true of "manufactured homes", a.k.a. "trailers", but not of "prefab" or "modular" homes, which are built using the same components as conventional homes.
They're the same components but not. If you would frame a building and then build the wall assemblies traditionally your ability to integrate parts of the structure and seal it is different than when you put together assemblies and try to seal the thick boundaries between them. You also don't have any quality control - whatever shows up on site is installed.