[iams]

  User-agent: googlebot
  #User-agent: bing
  User-agent: yandex
  Disallow: /

why would it be interpreted as a blank line? If you remove everything after the #, that includes the new line characters at the end of the line. Leaving:

  User-agent: googlebot
  User-agent: yandex
  Disallow: /

[franze]

here the problem comes in, as people write

  User-agent: googlebot
  User-agent: bing    #behaved-badly
  User-agent: yandex
  Disallow: /

if # removes everythin afer the #, then it would be

  User-agent: googlebot
  User-agent: bing    User-agent: yandex
  Disallow: /

which results into the whole line " User-agent: bing User-agent: yandex" beeing thrown out as malformed, so only googlebot would be disallowed.

[eyelidlessness]

It's ambiguous, and this is why significant whitespace can be so frustrating. Unless it's specified, some people will interpret a full-line comment as a blank line with a comment ending it (`^#.$`), and others will interpret it as you have (`^#.?\n`). Neither is obviously correct (even if it's obvious to you).

Edit: I don't know how to escape in HN formatting. Obviously there are italics where literal asterisks should be.

[traes]

  *** You can just use three asterisks. ***

* You can just use three asterisks. *

   Unfortunately you need something after them though. ***

Unfortunately you need something after them though.

[eyelidlessness]

Thanks, that's good to know!

[DoreenMichele]

If you indent it two spaces, I think that should solve that problem:

   *(`^#.$`), and others will interpret it as you have (`^#.?\n`)*

(Not sure where you intended those asterisks. I made my best inference.)

[eyelidlessness]

Yeah I was hoping to keep them inline, but indenting surely would have helped. I intended them after the dots (`.*`, dot asterisk in case I get the escaping wrong, is 0 or more characters in regex).