AllowUsers user1 user2...
And it's probably best to disable password logins and use keys instead (http://www.google.com./search?hl=en&q=ssh+authorized_key...).