most bacteria have an outer lipid layer that is clung to by the hydrophobic end of soaps and pulled apart by the hydrophilic forces clinging to water on the other end.
Additionally, the soap itself is binding to oils (which often frees up dirt if it was trapping in or under an oily layer, which being hydrophobic will stop water from washing the dirt off). Soap itself doesn’t do a whole lot against dirt.