PS: Did some research and while it is true that a ZWNBSP will break up ligatures in many systems, that’s not its intended use. It’s there to prevent line breaks (up to Unicode 3.2), nowadays only kept for compatibility outside of BOM. Modern Unicode uses U+2060 WORD JOINER to express the same intention.
The old Unicode rules for ZWNBSP are also quite tricky: you are not supposed to ignore the first ZWNBSP if you already know the encoding. Which means to express an initial actual ZWNBSP you need to write two of them if the encoding is unknown to the receiver and only one of them if known.
> Where the character set information is explicitly marked, such as in UTF-16BE or UTF-16LE, then all U+FEFF characters, even at the very beginning of the text, are to be interpreted as zero width no-break spaces. Similarly, where Unicode text has known byte order, initial U+FEFF characters are also not required and are to be interpreted as zero width no-break spaces. For example, for strings in an API, the memory architecture of the processor provides the explicit byte order. For databases and similar structures, it is much more efficient and robust to use a uniform byte order for the same field (if not the entire database), thereby avoiding use of the byte order mark.
Systems that use the byte order mark must recognize that an initial U+FEFF signals the byte order; it is not part of the textual content. It should be removed before processing, because otherwise it may be mistaken for a legitimate zero width no-break space. To represent an initial U+FEFF ZERO WIDTH NO-BREAK SPACE in a UTF-16 file, use U+FEFF twice in a row.
The first one is a byte order mark; the second one is the initial zero width no-break space.
No, it's only allowed at the start. How would that even work? If an app sees a UTF-32 BOM halfway through a file should it interpret the rest of the file as UTF-32? Are there any apps that handle text files like this?