[GeoAtreides]

>Doesn't it seem implausible that in the world of Dune they reached their level of technology without AGI?

This implies their technology is unfathomably complex, too complex for the human mind to understand. Doesn't seem to be the case.

>Seems like if it was adopted as a policy, there would have to be many gray areas where people pretend that they aren't really using AI - the ship's computer is just a domain specific finetuned version, after all, etc.

Ships are using Navigators, who are trained in "higher mathematics" and are also prescient. They have no need for AIs, nor do they wish to court destruction if they are found out (I can only imagine the BG will have the Spacing Guild by the balls if they knew).

> Why would the dangers of an incomprehensible machine mind be worse than the dangers of a transhuman being that was bred and optimized for generations? Who solved the alignment problem for these superhumans?

The books address these questions, especially the one about a certain God-Emperor...