Seems like there should be some inexpensive way to buy/built a BTLE “kill switch” that hooks onto your belt. (And indeed a simple search turns up such a product for about $200.)
Killing the power can help in some situations, but nowhere near all of them. A small boat moves very differently than a person treading water. It drifts in the wind. A swimmer doesn't. Even with the engine off, there is a good chance that he wouldn't get back on. And a one-foot wave isn't so small when your eyes are only a couple inches above the water. You need a positive connection between you and boat.