Self signed != inherently insecure. If you have the public key, verified it, and you want to make sure that you're getting content directly from the server, you can do just that. It's just not got a chain of trust with root certs that are built into the browser.
My point is how do you get the public key if you can't trust what has been published? You can't trust the gemini site, you'd have to use another protocol, such as HTTPS signed by a CA in order to verify the public key you are being given was actually signed by the author and not someone in the middle rewriting the authors gemini content.