You'd also need the routing number for the bank as well as the name on the account, at a minimum. And what you're thinking of is ACH, which goes between institutions, not between individuals.
The routing and account numbers are on all paper checks you write. You can use them to set up ACH transfers to/from that account, with pretty minimal checks.
Those transfers have to go through heavily regulated institutions, however, all of whom have strict KYC laws. Of all the ways to try and steal someone's money, this would be fairly low on my list.