I've run into similar issues and found that the `thiserror` crate (https://crates.io/crates/thiserror) combined w/ anyhow makes a lot of that pain go away
For those reading this that aren't super familiar, common Rust advice is "use thiserror for libraries and anyhow for applications," as they make slightly different tradeoffs and so are useful, especially together.