In short, this is because JS does not treat NaN and Infinity as numerical constants but as pre-defined, mutable variables; this way backward-compatible parsing of hypothetical sane JSON with eval would be vulnerable to injection.
Nevertheless, many JSON codecs have their own idea what to do with it, so this stuff can get really nasty.