[tptacek]

You're describing the simplest possible challenge-response scheme. It has two problems, both severe enough that you shouldn't recommend people waste time implementing it:

* First, because no browser bakes this crypto protocol in, you have to deliver it over Javascript. The protocol basically stipulates that you don't have SSL/TLS. So all you've done is move the goalposts. No matter what kind of dance you do (for instance: Meebo actually delivers a JS implementation of RSA!), the action is now in the JS delivery, which is trivially compromised.

* Second, secure authentication schemes aren't vulnerable to trivial dictionary attacks. This one is: the attacker is stipulated to have access to your traffic. She sees the nonce the server sends and the hash the client responds with. She can solve for the password by (very fast) brute force against a wordlist.

[dryicerx]

To your first point, yes js would be doing the crypt, if the js delivery can be compromised, then the login html delivery can be equally compromised (which would send the login information somewhere else)

I agree with your second point, a eavesdropped can use a dictionary attack. It makes it just a tiny bit harder for them since they need to generate their own cleartext-crypttext and cannot use a pregenerated table.

I am curious, is there a better way to do this (other than SSL or using RSA)

[tptacek]

In that first sentence, you need to take the we word "if" out. The exact same attack that motivated you to come up with the challenge-response scheme works against the JS delivery.

In the third sentence, take the "or RSA" out. There's no way to get a browser to safely do RSA authentication without SSL.

I have good news for you. The answer to this problem doesn't involve complex technology. What security practitioners are going to recommend to you is, just put up a login page, and send usernames and passwords. I have just released you from having to waste time and energy thinking about this.