[tialaramex]

Speed is one of those "Quantity has a quality all its own" things. We use very fast tools in a qualitatively different way, even though all that changed was how long it takes in seconds.

It is interesting that nobody was writing these tools in C or in C++. There are obvious ergonomic reasons, but perhaps also it matters that Rust cares a lot more about types than either of those languages.

[kibwen]

> It is interesting that nobody was writing these tools in C or in C++

This one's easier to explain. People interested in tooling for a specific language probably want to write that tooling in that language (hence pip, poetry, mypy, jedi, etc). Normally that would be the end if it, if Python wasn't 10-100x slower than a natively compiled language. And going from Python to Rust is an order of magnitude easier than going from Python to C or Python to C++, because the compiler is so good at identifying silly mistakes. Rust is just a friendlier language.

[f311a]

It's just very hard to write such systems in C/C++. Even all these Rust versions are segfaulting and panicking quite a lot. So many corner and edge cases that can be found in Python code, and the memory handling is also hard.

The author of Zuban started writing it back in 2020 or 2021, so it took him more than 4 years to complete it. And he is the author of Jedi, so he had prior experience already.

[tialaramex]

I get why they'd panic, but why have enough segfaults that you noticed? So, I went and read the code.

Zuban seems to have a bunch of scary "I'm not sure if this is correct" unsafe blocks, which to me would be a red flag. I mean, it's better that there's a comment expressing the doubt, but my experience is that if you're not sure whether it's correct, it's probably not correct.

[estebank]

While acknowledging the risk of causing a pile-on (which I don't want!), would it be possible to have a link to them or a description of what the unsafe blocks accomplish? I'm intrigued if they are for performance or API ergonomics, if they are due to limitations of the borrow checker, the stdlib or crate dependencies.

For anyone reaching for unsafe, there are in many cases either an existing API (split_at_mut comes to mind). For others, using zero-copy or bytemuck instead of unsafe is a good idea too.

None of that is to say "never write unsafe", unsafe existing is pretty much one of the reasons for Rust to be :)

[tialaramex]

I skimmed these, so this is nothing close to a survey, much less a comprehensive review of the software, however

For example in crates/zuban_python/src/file/diagnostics.rs:

"TODO this unsafe feels very wrong, because a bit lower we might modify the complex/ points."

or crates/zuban_python/src/database.rs:

"Points are guarded by specific logic and if they are overwritten by something that shouldn't it should not be that tragic."

I saw nothing where I was like "ZOMG this is definitely busted" but I definitely did not get the robust "Oh, I see now why this is correct" that I like from a good unsafe rationale comment, and these aren't tiny things like the small unsafe bit twiddling transmutes which are probably either actually correct or in any case will do what you expected at compile time and so any surprises are priced in without a rationale text.