That's why Cell<T> and RefCell<T> are a thing. Both allow you to mutate shared references, but disable shared access across threads. The qcell crate even includes a version of Cell/RefCell that's "branded" by a region/lifetime, just like in this proposal.
As I remember, Cell only allows moving/copying data from/to cell so if you have a 128-byte object inside do you have to copy it to modify? Or this can be optimized?
RefCell does do runtime checks, but the cost is checking the counter, a conditional branch, then incrementing/decrementing the counter twice.
Because the counter is non-atomic and non-volatile the optimiser can sometimes optimise out the actual modification of the counter. It's not free, but it's not also not a huge expense.
If you're talking about stack objects, that's what lifetimes are for.
If you're talking about heap, you can accomplish that by restricting that the references can't be used to free or invalidate the memory. But you could still allow them to be mutable.