In my particular case, I was absolutely sure it was safe without having to test it. Would it remain so forever? Probably not because software changes over time.
In those cases what you are supposed to do is write your logic using unsafe, and wrap it with lifetime safe APIs. That way your "smartness" joins efforts with rusts, and not-so-smart people can use your APIs without having to be as smart.