[rwmj]

In the UK you can set up a direct debit with just the bank account details (not quite the same as withdrawing money, but still enough to scam someone). The rest of your comment is quite correct though.

[IanCal]

Although there is at least the safety of the direct debit system, where if you flag it then the bank must immediately pay you what was taken.