[sillysaurus3]

Joel Spolsky said that ten years ago. The problem is that devs are afraid to learn unicode. They treat it like learning a foreign language. It's not even a fun problem, like learning a new programming language, so nobody makes time for it. The only people who learn it are those who make it a point of pride to implement something correctly and handle corner cases.

Unicode isn't even hard: Use UTF-8. Don't try to measure the length of a string unless you're rendering that string and measuring the length in screen units like pixels. If you do those two things, that's 90% of the effort of making Unicode-safe software.

I think both views are valid. Those who don't know how to write Unicode-safe software shouldn't feel shamed into learning Unicode before releasing open source work. Those who already know Unicode should feel happy that they're making other people's lives easier.

[tracker1]

But these are file formats that may well not be encoded in UTF-8.. the formats already exist.. it isn't like he's creating a new spreadsheet format here. Some of them may well be encoded to something that works fine against unicode/utf-8, others not so much.

[sillysaurus3]

So you write FooToUTF8() and UTF8ToFoo(), where Foo is whatever the encoding is in the external format. Done.

As far as I know, UTF-8 will work 100% of the time, and is almost always the best internal representation for software you write due to how simple and uniform it is. If something is encoded in some other format, you can probably find a conversion function online.

[tracker1]

Okay, so why don't you fork the project, and create your simple Foo/UTF8 methods, and confirm that they are the correct Foo/UTF8 methods for each of the document formats supported.

I'm not saying that it's really all that hard, but there are multiple document formats, and versions of those formats. The author obviously didn't need unicode support, so didn't test for it. I'm sure test cases, and a pull request would be welcome.