[masklinn]

> How does using UTF-8 to represent strings help? Because you can represent invalid strings: just leave them as-is and don't try to decode them unless you have to.

That’s not UTF8. That’s a bag’o bytes which might be UTF8. Very different thing.

> Sure, you can't decode them as code points, but that's actually a pretty unusual thing to do.

It’s not, any unicode-aware text processing does it implicitly. This means any such processing has to either perform its own validation that the input is valid, or it may fly off the rails entirely if fed nonsense. This also increases risks if security issues, either outright UBs, or the ability to smuggle payloads through overlong encoding.

[StefanKarpinski]

> That’s not UTF8.

True; I was careful not to call it that, but treating strings as UTF-8 by convention does make sense.

> It’s not, any unicode-aware text processing does it implicitly. This means any such things processing has to either perform its own validation that the input is valid, or it may fly off the rails entirely if fed nonsense.

In theory, but that's just not how most string operations actually work. If you have two UTF-8 strings and you want to concatenate them, you just concatenate the bytes. It would be ridiculously inefficient to decode the code points in each string and then re-encode them back into a destination buffer. If you have two UTF-8 strings and you want to see if one is a substring of the other and at what byte index, you just look for the bytes of one as a "substring" of the bytes of the other. Again, it would be ridiculously inefficient to decode the code points in each and do matching on code points. But what if the strings aren't valid UTF-8?! Both of those operations work just fine even if the strings aren't valid and produce sensible, intuitive results.

If you're implementing a browser or a terminal that has to actually display UTF-8 as characters then sure, you have to actually decode characters. Similarly, if you're parsing text somehow, then you have to decode characters. But many program only do concatenation and search and other operations like that which are actually implemented in terms of byte sequences, not characters.

[masklinn]

> True; I was careful not to call it that

You specifically called it UTF8, repeatedly. The very comment I quoted asserts that "Utf8 would help deal with the issue [of garbage inputs]" (in its denial of the opposite assertion). You also did it in https://news.ycombinator.com/item?id=33986421

> If you have two UTF-8 strings and you want to concatenate them, you just concatenate the bytes.

That's not a unicode-aware operation, it's mostly a unicode-irrelevant operation (though unicode awareness can be useful in edge cases because of special grapheme clusters, but that's very task-specific).

> But what if the strings aren't valid UTF-8?! Both of those operations work just fine even if the strings aren't valid and produce sensible, intuitive results.

If your content is not actually UTF-8, you can end up with UTF-8, thus changing the semantics of the content. You can also end up with overlong UTF-8, which also changes the semantics of the content in a worse way.

[StefanKarpinski]

The comment that you're quoting wasn't mine. In the comment you link to says "UTF-8 by convention". If either string is valid, then the result is as expected. If you're concatenating two strings that are both invalid UTF-8, there's not much you can do that's better than just concatenating the bytes together... which is exactly what treating them as byte arrays would end up doing (but it's less convenient). If you're worried about invalid UTF-8 you can check for validity (which again, is exactly what you end up doing if you use byte arrays).

[masklinn]

> The comment that you're quoting wasn't mine.

The comment I originally quoted was yours. The second quote is adapted from a statement you denied. It is thus your statement. Let me put both sections together since you seem unwilling to do so:

>> Utf8 would not help with the issue in the article in any way.

> It's not at all obvious how it helps, but it does.

So you are stating, unambiguously, that "Utf8 would help deal with the issue [of garbage inputs]".

> In the comment you link to says "UTF-8 by convention".

The comment I link says:

> Treating paths as UTF-8 works very well

Which is either

1. wrong

or

2. nonsensical, given the later statement that you should not "require your UTF-8 strings to be valid", which would make them not UTF-8

> If you're concatenating two strings that are both invalid UTF-8, there's not much you can do that's better than just concatenating the bytes together... which is exactly what treating them as byte arrays would end up doing

But that's the point innit? You're asserting semantics which don't hold and which you break with no regard.

> (but it's less convenient).

Is it now? Here's the concatenation of two strings:

    a + b

here's the concatenation of two byte arrays:

    a + b

You're right, the inconvenience makes me shudder. What horror. What indignity.

[adgjlsfhk1]

The problem with using strict UTF-8 for paths is that paths aren't guaranteed to be valid UTF-8. How do you want to write a program that opens a path who's name is invalid UTF-8?

[masklinn]

> The problem with using strict UTF-8 for paths is that paths aren't guaranteed to be valid UTF-8.

Ok but I’m not saying to do that. I’m saying if you have not-utf8 strings don’t call them UTF8.

> How do you want to write a program that opens a path who's name is invalid UTF-8?

That’s not my problem given I’m not advocating for that.

[StefanKarpinski]

The issue is that when you're implementing something like a programming language or a robust general purpose utility, then simply not being able to open—or list or remove or stat—paths with invalid names is not really acceptable.

[masklinn]

Do you actually read comments before replying?