[josefx]

By default XML is either UTF-8 or 16, any other encoding has to be identified either through metadata or an explicit declaration in the document itself.

If you are guessing it is because someone failed to properly store or transmit the document.

[derriz]

By "through metadata", you mean with some external separate mechanism unrelated to the XML spec? That's exactly the flaw of what's supposed to be a universal and general document representation.

Both UTF-8 or UTF-16 are supported, yes, but that doesn't solve the problem because XML parsers have to deal with other encodings. A document that initially looks like UTF-8 could be almost any of literally hundreds of encodings including all the common ISO 8859 encodings. Which is why XML processing/parsing code has to go through multiple phases: a "try to determine the encoding" phase (using heuristics like those described in https://www.w3.org/TR/REC-xml/#sec-guessing) and then switch to a "parsing" phase after rewinding to the start of the document.

I did some work for a bank once which were in the process of moving to using XML as the universal messaging format (message bus architecture) and this wasn't some theoretical issue - it was a genuine pain as the message sources and sinks included a bunch of machines with different native encodings including a bunch of IBM machines (EBCDIC), old Windows machines, proprietary Unices like HPUX, along with Linux, etc.

This is a well known problem with XML.

[josefx]

> By "through metadata", you mean with some external separate mechanism unrelated to the XML spec?

The code generating the xml text may not be aware of the final encoding used to store/transmit it and a library writing text to an encoded stream might not be aware that the text it is sending contains xml. So I would say allowing xml to be parsed using an externally specified encoding makes at least some sense.

> A document that initially looks like UTF-8 could be almost any of literally hundreds of encodings including all the common ISO 8859 encodings

Your own link shows that those "hundreds of encodings" are all based on ASCII, which is enough to get the concrete encoding from the declaration.

> and then switch to a "parsing" phase after rewinding to the start of the document.

So one call to fseek(stream, 0, SEEK_SET)?