[rbehrends]

You can also have the exact opposite problem with RAII, where a resource survives the end of a transaction, because there is still a live reference to it hidden away somewhere (say, due to some debugging code holding on to it).

This is a classical liveness vs. safety dualism. "Something good will eventually happen" and "nothing bad will ever happen" are promises whose solutions are often in conflict with one another.

The general problem — to make transactional state changes and transactional control flow (i.e. expectations about these state changes) match up precisely — is not easy to solve in the general case, especially once you move on to things that are less trivial than simple resource acquisition/release matching.

[hawkice]

Oddly, Rust's ownership system really does solve these problems, and Non-lexical lifetimes should eliminate accidental scope-broadening. Unless you are doing some mega-schenanigans, an e.g. MutexGuard gets released precisely when you think.

Your point about this being difficult to solve in the general case is true, it's just worth pointing out Rust intends to do that hard thing anyway.

[steveklabnik]

Non-lexical lifetimes do not affect something that implements Drop, so the MutexGuard will, by default, last till the end of its lexical scope.

You can still call drop on it manually to release it earlier, though.

[hawkice]

Huh! Learn something new every day -- so are lifetimes still exactly describing when things get dropped? And anything implementing Drop just doesn't get optimized lifetimes?

[oconnor663]

Lifetimes and destructors aren't as directly related as it might feel at first. Destructors of course are a thing that happens at runtime, and sometimes the order in which things get destructed has important observable side effects. (Maybe it prints something, or maybe you care about the order locks are released in. Apart from memory safety questions.)

Lifetimes really have no runtime effect at all. They only exist to prove things about the program at compile time. So all the types and function signatures get assembled together, and then a constraint solver gets run over the whole thing. As long as it returns "yes a solution exists", then no ones really cares about the details of the solution. The benefit of non-lexical lifetimes is to weaken the constraints on the system, so that code that used to appear invalid now appears valid. But I believe it will have no effect on any existing code. (There's a Rust compiler reimplementation somewhere that doesn't even check lifetimes, since you can always use the standard compiler in testing.)

[steveklabnik]

> so are lifetimes still exactly describing when things get dropped?

It depends on exactly what you mean by this.

In a nutshell, "non-lexical lifetimes" means "things go out of scope when after their last use. Drop implies a use at the end of the current lexical scope."

Dropping Drop types earlier ("eager drop") was desired, but has significant problems, including "a large body of unsafe code exists in production which relies on knowing when Drop types go out of scope and changing this behavior may cause a ton of unsoundness in existing code."

[dagenix]

Not really.

Lifetimes are a language you use to help the compiler prove that all of your references will be valid. If it's unable to prove that, it will throw up an error. That doesn't prove that your references were wrong - it just says that they _might_ be wrong, and the compiler won't allow that possibility. Non-lexical lifetimes just provide the ability to prove more refences and thus allow more code to compile - code that was already fine, but, the compiler couldn't figure out that it was fine.

[rbehrends]

> Oddly, Rust's ownership system really does solve these problems

No. Rust's ownership problem solves it for trivial cases, at the cost of making it hard to do other things (such as sharing references past the lifetime of the owner without resorting to Rc<T> or Arc<T>, at which point you don't really have lifetime guarantees anymore).

The essential limitation of Rust is that (without resorting to Rc<T> and Arc<T>, which would put you back to square one) it is conceptually limited to the equivalent of reference counting with a maximum reference count of 1. In order to make this work, Rust needs move semantics and the ability to prove that an alias has a lifetime that is a subset of the lifetime of the original object) and may even sometimes have to copy objects, because it can never actually increase the (purely fictitious) reference count after object creation.

This inherent limitation makes a lot of things hard (or at least hard to do without copying or explicit reference counting). Structural sharing in general, hash consing, persistent data structures, global and shared caches, cyclic data structures, and so forth.

In short, you have the problem with shared references less, because Rust makes it hard to share data in the first place, for better or worse. (Again, unless you resort to reference counting, and then you get the issue back in full force.)

[steveklabnik]

> it is conceptually limited to the equivalent of reference counting with a maximum reference count of 1.

This is a thing people say, but I think it's misleading. Reference counting can increase the lifetime of an object, but borrowing cannot. I've seen this really trip up beginners.

> This inherent limitation makes a lot of things hard

It can make them different, which can be hard, but these things are already hard. And some people think it can make them easier or even better; see Bodil Stokke's work on persistent data structures in Rust.

[bunderbunder]

> This is a thing people say, but I think it's misleading. Reference counting can increase the lifetime of an object, but borrowing cannot. I've seen this really trip up beginners.

I'm not sure I follow.

The only reference-counted language I've used is (pre-ARC) Objective-C. There, it was a very common idiom to "borrow" objects - so common that it didn't even have a name. There was just objects you "retained" (that is, staked a claim on), and ones you didn't.

Maybe there's a pitfall to how the "automatic" part of automatic reference counting is typically implemented?

[steveklabnik]

It has been years since I've written objective-c, so I'll write out some psuedo-code. This may be wrong, please correct me! (It should map to C++ pretty directly, and certainly does in unsafe Rust.)

* You have an object. You call retain on it. You have a count of one.

* You also have a pointer to that object. The "borrow" in your analogy.

* You return this pointer, and stash it somewhere. The object still has a count of one, so it's still live, so this is okay.

* Later in your program, you use that pointer to call release.

Here, we've only ever had a reference count of one, but our object has lived across arbitrary inner scopes. In Rust, this would not work, unless you dropped into unsafe.

Obviously, with Arc and autoretain this kind of code doesn't get written anymore, I would hope. And even without, it wouldn't be guaranteed, so you'd want the "borrow" to actually bump the refcount. But Rust is about guaranteeing that it can't.

[bunderbunder]

Ah, I think I follow.

So, it sounds to me like it's not necessarily that Rust's model is fundamentally different from "ref counting with a limit of 1", at least in terms of how you should be managing your memory, so much as that the language doesn't let you some things that you really shouldn't be doing in the first place.

Sometimes it felt like Objective C wouldn't just let you point a gun at your foot, it would actively cheer you on while you did it.

[rbehrends]

> This is a thing people say, but I think it's misleading. Reference counting can increase the lifetime of an object, but borrowing cannot. I've seen this really trip up beginners.

"With a maximum reference count of 1." As the reference count becomes 1 upon object creation, it cannot really be increased further. Hence, only operations that keep the (virtual) reference count at 1 or reduce it to 0 are allowed.

My point here is that you inherently cannot do things where you cannot prove that this virtual reference count can be capped at 1.

[steveklabnik]

That doesn't change much; the point is that (in many languages), variables going into or out of scopes don't fiddle with the ref count[1], and so people assume that something will live until they make the count go down explicitly.

It also only refers to ownership, not borrowing, and both are equally important.

Beyond that, what I'm saying is something more meta: It doesn't really matter if this analogy is spot-on or not; it's got enough wiggle room in it that I've seen it trip up beginners. Maybe that's because they misunderstand the analogy, but given that its point is to convey understanding, that means that it isn't a great analogy, in my experience. YMMV.

1: directly, of course; this also depends on the language.

[rbehrends]

> It also only refers to ownership, not borrowing, and both are equally important.

I addressed borrowing above. Borrowing is proving lifetime subset properties and that you therefore can avoid increasing the virtual reference count.

And this is not about whether this is useful for beginners. It is to illustrate inherent limitations of the approach.

[burntsushi]

> Rust's ownership problem solves it for trivial cases, at the cost of making it hard to do other things

Your analysis of the trade offs is fine, but you claim that Rust only solves this problem for "trivial" cases. If that's true, then most of the Rust code I've written is trivial. To me, that pretty thoroughly weakens your dismissal here, at least in my case.

[rbehrends]

I am talking about just using basic references & borrowing. Once you introduce reference counting (Rc<T> and Arc<T>) and copying, you open up a lot more options, of course, but at this point you also can't make any lifetime guarantees anymore, because objects can escape the "owner's" scope at will.

[burntsushi]

> I am talking about just using basic references & borrowing.

... yes, I know. And is presumably what you referred to as "trivial." But this in fact comprises the vast majority of Rust code I've written. So you can call it trivial if you want, but as I said, it significantly reduces the weight of your dismissal.

There's plenty of Rust code I've written that makes use of Arc/Rc, specifically for cases you've called out (global caches, structural sharing, etc.) but it's nowhere near ubiquitous. So what I'm trying to say is that your representation of the problems that Rust solves is at best misleading, as supported by my experience writing a not insignificant amount of Rust.

So in other words, sure, you can call most of my code "trivial," but on the other hand, I can say that the problems posed by you in your top-level comment are actually solved in most of my code, regardless of whether you think it's trivial or not.

[tcoff91]

Yes but that isn’t a problem because you will still be protected from all the things that rust is trying to protect you from.

[jstimpfle]

That's also a problem with garbage collection, by the way. GC means memory safety, but not necessarily correctness. In fact, it invites sloppiness, and a kind of sloppiness that sits at a more conceptual level, which could be harder to fix.

[Ooveex2C]

At least in java the recommended way is to only use GC as safety net for resource management. Print a warning when it's cleaned up due to finalizers (or cleaner-refs since v9) and use scope-based cleaning instead where possible.

[fjsolwmv]

Can you give in an example?

Keeping a debug reference at end of transaction (via a shared-reference type, since a non-shared RAII reference type could never get into that state) isn't a coding error, it's a design error -- the development intentionally requested contradictory things. That is solved by using weak references if you don't want a debug tool to force an object to stay alive.

[rbehrends]

It is not a design error. You are making the mistake of equating reachability of a reference with making it legal to access it. And the problem is that the two often aren't one and the same.

As a simple example, you may still want to access a resource after it has been released. Closing a network connection, for example, does not mean that accessing it is invalid. The connection may still have state (such as statistics collected or whether a non-blocking close was clean) that is perfectly legal to access after release (and in fact may only be consistent/observable afterwards).

The Eiffel FILE class [1], for example, allows you to call `is_closed` at any time (as well as the various `is_open` functions). This is necessary because `not is_closed` is evaluated as a precondition for many other operations.

A more complex example is a resource that is shared by many threads. Whether that resource is valid is often not a question of whether a reference is reachable, but a function of complex distributed state. Sometimes this can be solved by atomic reference counting, but even then atomic reference counting is expensive.

[1] https://archive.eiffel.com/products/base/classes/kernel/file...