Because it means waves bounce off the bottom faster (less distance travelled) and much more importantly, with far more energy (square cube law works against you). So the waters far more choppy far faster, since you have 50% less water volume to absorb all the energy.
If it's too shallow the swimmers arms hit the bottom ...
Slightly deeper and there's drag from the floor as their arms barely miss it. That effect persists until it doesn't .. now it's deep enough.
It needs to be deep enough that vortex's created by swimmers have disapated by the time they reach bottom and reflect back to the surface so as to not interfere with following swimmers or swimmers returning.
Well before that the hydrogen atoms would fuse into helium and release vast amounts of energy, killing all the swimmers, spectators, and many people in Paris.
Edge effects affecting the flow field around the swimmer. I suppose the floor might trap turbulence near the surface rather than dissipating into the depths.