[maxdamantus]

But there's no option to just construct the string with the invalid bytes. 3) is not for this purpose; it is for when you already know that it is valid.

If you use 3) to create a &str/String from invalid bytes, you can't safely use that string as the standard library is unfortunately designed around the assumption that only valid UTF-8 is stored.

https://doc.rust-lang.org/std/primitive.str.html#invariant

> Constructing a non-UTF-8 string slice is not immediate undefined behavior, but any function called on a string slice may assume that it is valid UTF-8, which means that a non-UTF-8 string slice can lead to undefined behavior down the road.

[gf000]

How could any library function work with completely random bytes? Like, how would it iterate over code points? It may want to assume utf8's standard rules and e.g. know that after this byte prefix, the next byte is also part of the same code point (excuse me if I'm using wrong terminology), but now you need complex error handling at every single line, which would be unnecessary if you just made your type represent only valid instances.

Again, this is the same simplistic, vs just the right abstraction, this just smudges the complexity over a much larger surface area.

If you have a byte array that is not utf-8 encoded, then just... use a byte array.

[kragen]

There are a lot of operations that are valid and well-defined on binary strings, such as sorting them, hashing them, writing them to files, measuring their lengths, indexing a trie with them, splitting them on delimiter bytes or substrings, concatenating them, substring-searching them, posting them to ZMQ as messages, subscribing to them as ZMQ prefixes, using them as keys or values in LevelDB, and so on. For binary strings that don't contain null bytes, we can add passing them as command-line arguments and using them as filenames.

The entire point of UTF-8 (designed, by the way, by the group that designed Go) is to encode Unicode in such a way that these byte string operations perform the corresponding Unicode operations, precisely so that you don't have to care whether your string is Unicode or just plain ASCII, so you don't need any error handling, except for the rare case where you want to do something related to the text that the string semantically represents. The only operation that doesn't really map is measuring the length.

[xyzzyz]

> There are a lot of operations that are valid and well-defined on binary strings, such as (...), and so on.

Every single thing you listed here is supported by &[u8] type. That's the point: if you want to operate on data without assuming it's valid UTF-8, you just use &[u8] (or allocating Vec<u8>), and the standard library offers what you'd typically want, except of the functions that assume that the string is valid UTF-8 (like e.g. iterating over code points). If you want that, you need to convert your &[u8] to &str, and the process of conversion forces you to check for conversion errors.

[maxdamantus]

The problem is that there are so many functions that unnecessarily take `&str` rather than `&[u8]` because the expectation is that textual things should use `&str`.

So you naturally write another one of these functions that takes a `&str` so that it can pass to another function that only accepts `&str`.

Fundamentally no one actually requires validation (ie, walking over the string an extra time up front), we're just making it part of the contract because something else has made it part of the contract.

[kragen]

It's much worse than that—in many cases, such as passing a filename to a program on the Linux command line, correct behavior requires not validating, so erroring out when validation fails introduces bugs. I've explained this in more detail in https://news.ycombinator.com/item?id=44991638.

[kragen]

That's semantically okay, but giving &str such a short name creates a dangerous temptation to use it for things such as filenames, stdio, and command-line arguments, where that process of conversion introduces errors into code that would otherwise work reliably for any non-null-containing string, as it does in Go. If it were called something like ValidatedUnicodeTextSlice it would probably be fine.

[amluto]

> command-line arguments

Command-line arguments on Windows are their own special disaster.

[adastra22]

I'd agree if it was &[bytes] or whatever. But &[u8] isn't that much different from &str.

[kragen]

Isn't &[u8] what you should be using for command-line arguments and filenames and whatnot? In that case you'd want its name to be short, like &[u8], rather than long like &[bytes] or &[raw_uncut_byte] or something.

[xyzzyz]

It's actually extremely hard to introduce problems like that, precisely because Rust's standard library is very well designed. Can you give an example scenario where it would be a problem?

[kragen]

Well, for example, the extremely exotic scenario of passing command-line arguments to a program on little-known operating systems like Linux and FreeBSD; https://doc.rust-lang.org/book/ch12-01-accepting-command-lin... recommends:

  use std::env;

  fn main() {
      let args: Vec<String> = env::args().collect();
      ...
  }

When I run this code, a literal example from the official manual, with this filename I have here, it panics:

    $ ./main $'\200'
    thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "\x80"', library/std/src/env.rs:805:51
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

($'\200' is bash's notation for a single byte with the value 128. We'll see it below in the strace output.)

So, literally any program anyone writes in Rust will crash if you attempt to pass it that filename, if it uses the manual's recommended way to accept command-line arguments. It might work fine for a long time, in all kinds of tests, and then blow up in production when a wild file appears with a filename that fails to be valid Unicode.

This C program I just wrote handles it fine:

  #include <unistd.h>
  #include <fcntl.h>
  #include <stdio.h>
  #include <stdlib.h>

  char buf[4096];

  void
  err(char *s)
  {
    perror(s);
    exit(-1);
  }

  int
  main(int argc, char **argv)
  {
    int input, output;
    if ((input = open(argv[1], O_RDONLY)) < 0) err(argv[1]);
    if ((output = open(argv[2], O_WRONLY | O_CREAT, 0666)) < 0) err(argv[2]);
    for (;;) {
      ssize_t size = read(input, buf, sizeof buf);
      if (size < 0) err("read");
      if (size == 0) return 0;
      ssize_t size2 = write(output, buf, (size_t)size);
      if (size2 != size) err("write");
    }
  }

(I probably should have used O_TRUNC.)

Here you can see that it does successfully copy that file:

    $ cat baz
    cat: baz: No such file or directory
    $ strace -s4096 ./cp $'\200' baz
    execve("./cp", ["./cp", "\200", "baz"], 0x7ffd7ab60058 /* 50 vars */) = 0
    brk(NULL)                               = 0xd3ec000
    brk(0xd3ecd00)                          = 0xd3ecd00
    arch_prctl(ARCH_SET_FS, 0xd3ec380)      = 0
    set_tid_address(0xd3ec650)              = 4153012
    set_robust_list(0xd3ec660, 24)          = 0
    rseq(0xd3ecca0, 0x20, 0, 0x53053053)    = 0
    prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=9788*1024, rlim_max=RLIM64_INFINITY}) = 0
    readlink("/proc/self/exe", ".../cp", 4096) = 22
    getrandom("\xcf\x1f\xb7\xd3\xdb\x4c\xc7\x2c", 8, GRND_NONBLOCK) = 8
    brk(NULL)                               = 0xd3ecd00
    brk(0xd40dd00)                          = 0xd40dd00
    brk(0xd40e000)                          = 0xd40e000
    mprotect(0x4a2000, 16384, PROT_READ)    = 0
    openat(AT_FDCWD, "\200", O_RDONLY)      = 3
    openat(AT_FDCWD, "baz", O_WRONLY|O_CREAT, 0666) = 4
    read(3, "foo\n", 4096)                  = 4
    write(4, "foo\n", 4)                    = 4
    read(3, "", 4096)                       = 0
    exit_group(0)                           = ?
    +++ exited with 0 +++
    $ cat baz
    foo

The Rust manual page linked above explains why they think introducing this bug by default into all your programs is a good idea, and how to avoid it:

> Note that std::env::args will panic if any argument contains invalid Unicode. If your program needs to accept arguments containing invalid Unicode, use std::env::args_os instead. That function returns an iterator that produces OsString values instead of String values. We’ve chosen to use std::env::args here for simplicity because OsString values differ per platform and are more complex to work with than String values.

I don't know what's "complex" about OsString, but for the time being I'll take the manual's word for it.

So, Rust's approach evidently makes it extremely hard not to introduce problems like that, even in the simplest programs.

Go's approach doesn't have that problem; this program works just as well as the C program, without the Rust footgun:

  package main

  import (
          "io"
          "log"
          "os"
  )

  func main() {
          src, err := os.Open(os.Args[1])
          if err != nil {
                  log.Fatalf("open source: %v", err)
          }

          dst, err := os.OpenFile(os.Args[2], os.O_CREATE|os.O_WRONLY, 0666)
          if err != nil {
                  log.Fatalf("create dest: %v", err)
          }

          if _, err := io.Copy(dst, src); err != nil {
                  log.Fatalf("copy: %v", err)
          }
  }

(O_CREATE makes me laugh. I guess Ken did get to spell "creat" with an "e" after all!)

This program generates a much less clean strace, so I am not going to include it.

You might wonder how such a filename could arise other than as a deliberate attack. The most common scenario is when the filenames are encoded in a non-Unicode encoding like Shift-JIS or Latin-1, followed by disk corruption, but the deliberate attack scenario is nothing to sneeze at either. You don't want attackers to be able to create filenames your tools can't see, or turn to stone if they examine, like Medusa.

Note that the log message on error also includes the ill-formed Unicode filename:

  $ ./cp $'\201' baz
  2025/08/22 21:53:49 open source: open ζ: no such file or directory

But it didn't say ζ. It actually emitted a byte with value 129, making the error message ill-formed UTF-8. This is obviously potentially dangerous, depending on where that logfile goes because it can include arbitrary terminal escape sequences. But note that Rust's UTF-8 validation won't protect you from that, or from things like this:

  $ ./cp $'\n2025/08/22 21:59:59 oh no' baz
  2025/08/22 21:59:09 open source: open 
  2025/08/22 21:59:59 oh no: no such file or directory

I'm not bagging on Rust. There are a lot of good things about Rust. But its string handling is not one of them.

[gf000]

Then [u8] can surely implement those functions.

[adastra22]

I don’t understand this complaint. (3) sounds like exactly what you are asking for. And yes, doing unsafe thing is unsafe.

[maxdamantus]

> I don’t understand this complaint. (3) sounds like exactly what you are asking for. And yes, doing unsafe thing is unsafe

You're meant to use `unsafe` as a way of limiting the scope of reasoning about safety.

Once you construct a `&str` using `from_utf8_unchecked`, you can't safely pass it to any other function without looking at its code and reasoning about whether it's still safe.

Also see the actual documentation: https://doc.rust-lang.org/std/primitive.str.html#method.from...

> Safety: The bytes passed in must be valid UTF-8.

[xyzzyz]

> If you use 3) to create a &str/String from invalid bytes, you can't safely use that string as the standard library is unfortunately designed around the assumption that only valid UTF-8 is stored.

Yes, and that's a good thing. It allows every code that gets &str/String to assume that the input is valid UTF-8. The alternative would be that every single time you write a function that takes a string as an argument, you have to analyze your code, consider what would happen if the argument was not valid UTF-8, and handle that appropriately. You'd also have to redo the whole analysis every time you modify the function. That's a horrible waste of time: it's much better to:

1) Convert things to String early, and assume validity later, and

2) Make functions that explicitly don't care about validity take &[u8] instead.

This is, of course, exactly what Rust does: I am not aware of a single thing that &str allows you to do that you cannot do with &[u8], except things that do require you to assume it's valid UTF-8.

[maxdamantus]

> This is, of course, exactly what Rust does: I am not aware of a single thing that &str allows you to do that you cannot do with &[u8], except things that do require you to assume it's valid UTF-8.

Doesn't this demonstrate my point? If you can do everything with &[u8], what's the point in validating UTF-8? It's just a less universal string type, and your program wastes CPU cycles doing unnecessary validation.

[matt_kantor]

> except things that do require you to assume it's valid UTF-8

That's the point.

[maxdamantus]

But no one has demonstrated an actual operation that requires valid UTF-8. The reasoning is always circular: "I require valid UTF-8 because someone else requires valid UTF-8".

Eventually there should be an underlying operation which can only work on valid UTF-8, but that doesn't exist. UTF-8 was designed such that invalid data can be detected and handled, without affecting the meaning of valid subsequences in the same string.

[amluto]

> UTF-8 was designed such that invalid data can be detected and handled, without affecting the meaning of valid subsequences in the same string.

But there is not a canonical response to invalid data. So literally every operation that might need to make a choice of what to do when presented what invalid data should either (a) accept a parameter asking what to do on error and potentially fail or (b) take a parameter type that forces errors to be handled in advance.