The problem is that a function cannot know whether or not the caller can recover from a particular error, so there's no point in making that distinction in the first place.
It's part of your API design, if they are recoverable or not. You should lean towards recoverable: if the caller can't recover, they can always do something with it. But some kinds of problems are non-recoverable.
Say you have a library which has some kind of internal state, and the consistency of said internal state is important for memory safety, as other operations rely on it.
Using assert! to verify that your state is consistent is useful, in case you have a bug. But since it's all internal, it's not something that the caller can really recover from, either. It's not their fault, it's yours.
Well here's my issue with this explanation that I've heard a lot: the caller may or may not be able to recover from it, but the caller may still be doing something with the error. Maybe the caller doesn't want to expose the error to its caller, or maybe the caller wants to build a new message to explain the cause of your error. But to panic! takes away a lot of options for the caller.
Worst of all, it leads to untestable code, because the calling test case can't properly check the error against some known result.