[gardaani]

I'm having problems fulfilling this requirement in my libs: "Crates providing libraries should never use functions or instructions that can fail and cause the code to panic."

The Rust standard library Vec, HashMap etc. can cause a panic in Rust, if the device (such as a mobile phone with a small memory) runs out of memory.

C and C++ standard libraries (malloc, std::vector, std::map..) can handle those situations by returning null or throwing an exception.

I wish Rust had some easy way to recover from out-of-memory situations when using the standard library. I have been considering writing my own out-of-memory safe Vec, HashMap etc, but it can't be the right way to do it..

[_0w8t]

https://www.youtube.com/watch?v=ARYP83yNAWk mentioned that the notion that C++ could handle out-of-memory was a purely theoretical wishful thinking. In practice it could not.

[masklinn]

Yes that is one of the primary failures of Rust at the moment: to my knowledge it currently has no good way to safely manage allocation failures (it also has serious issues with stack overflows).

This is an issue with all heap-allocating construct, not just collections but also Box or Rc.

> I wish Rust had some easy way to recover from out-of-memory situations when using the standard library. I have been considering writing my own out-of-memory safe Vec, HashMap etc, but it can't be the right way to do it..

Maybe look at the embedded space there? There might be no_std third-party libraries which handle these issues. Possibly on top of alloc as the (unstable[0] and obviously unsafe) `Alloc` trait does have a concept of allocation failure.

[0] https://github.com/rust-lang/rust/issues/32838

[steveklabnik]

> Yes that is one of the primary failures of Rust at the moment: to my knowledge it currently has no good way to safely manage allocation failures

So, sort of yes and sort of no.

The data structures that allocate in the standard library do not let you handle allocation failure. However, if you write your own, the global allocator lets you determine if failure happened, and then you can do whatever you want with it.

[rectang]

I think Rust got it right, here.

Dealing with allocation failure gracefully is hard and requires a lot of extra code.

For most applications the best default is to panic and handle it up the stack rather than pay the programming overhead of handling allocation failure explicitly in every last nook and cranny. The Rust standard library rightly optimizes for this use case.

For the embedded or critical-safety application spaces, where you really do want to handle allocation failure gracefully, you need something other than the standard library. Letting that "something" develop slowly out in the community is a good call.

[cesarb]

The problem with allocation failure is that it's non-local. Suppose thread A does a huge allocation of several hundred megabytes. The best case scenario is that this allocation fails cleanly; the worst case scenario is that this allocation succeeds but causes a small 1024-byte allocation in an unrelated thread B to fail (and it doesn't have to be multiple threads, this can happen even within a single thread). I don't know of any solution for this problem, other than statically reserving the memory which will be used for each part of the system.

[a1369209993]

The horrible part is that this isn't even limited to threads; a competely separate process (even, eg, a browser JS engine running 'completely' untrusted remote code) can cause your program to hit a allocation failure, and then not have enough memory to preform nontrivial error-handling.

[gardaani]

Actually, I have found FallibleVec written by Mozilla [1] but I haven't found anything for other containers, yet.

[1] https://github.com/mozilla/mp4parse_fallible

[baby]

Honestly all C programs that fail to allocate panic as well. What else can you do?

[masklinn]

> Honestly all C programs that fail to allocate panic as well.

I'd say that most C programs that fail to allocate panic (if only because e.g. they're running over-committing anyway so allocations never fail), but it's not true that all of them do. There are systems which do handle heap allocation failures "properly", usually ones which run on systems which are both resource-constrained enough that allocation failure is a possibility yet large enough that heap allocation is a possibility e.g. modern embedded-ish systems running with dozens (!) or even hundreds (!!!) of megabytes of RAM.

Or the software might be designed for running with a low ulimit for some reason (I think that also triggers allocation failures, I remember that being a case where Python throws MemoryError).

Even on larger machines, I'd also guess it is / was also more common on 32b systems where address space exhaustion could be a real concern.

> What else can you do?

You might just abort the current task because it's not that important when resource constrained, or you might have caches you can clear. Or possibly (if that's a possibly routine issue e.g. a background task which might run in cases of both plentiful and constained resources) you might switch to a less CPU efficient but more memory-efficient algorithm.

[baby]

I've looked at these types of systems and they would still crash, actually even if you want to gracefully handle failed allocations it is possible that your program will crash due to lack of memory

[Cyph0n]

Oh, I never even considered this... But how would Rust ever be able to signal allocation failures in such cases?

[steveklabnik]

Same as any other failure; functions that may try to allocate memory and fail will return a Result<T, E> instead of just a T.

[swiftcoder]

Which would be pretty painful for everyone not working in a memory-constrained environment (at a guess, that's most developers). Much like std:println! panicking instead of returning an error, ergonomics are important too a lot of the time.

[masklinn]

It obviously wouldn't be the default (it couldn't be, in fact, since the existing APIs are set in stone short of memory unsafety).

The way it'd work is either you'd have two different collections entirely, one which'd signal and one which'd panic (the latter possibly being built atop the former) or every method which can panic on allocation failure would also have a panic-safe alternative.

Example: https://github.com/rust-lang/rust/issues/48043 similarly Vec could gain `try_push`, `try_append`, `try_insert`, `try_split_off`, …

[__s]

Which is why you offer two APIs: one that panics, & one which returns Result or Option

[Sharlin]

Eg. with Vec, every operation that may trigger reallocation should return Result.