It's not about killing bacteria, its about leaving fluoride ions in a slightly basic or at least nonacidic environment for long enough to convert hydroxyapatite to fluorohydroxyapatite. This chemical reaction takes time, but considerably strengthens your teeth against corrosion and prevents cavities.
Brushing is mainly supposed to mechanically take the bacteria off of your teeth, like washing your hands, not kill them chemically. The fluoride exists to help keep enamel intact.