If hammer companies were suddenly the most valuable international companies, and spent millions on ad campaigns and lobbying about trusting the hammer interface, then you can assume a large amount of people might trust the hammer interface
Even if your tool learns to talk and to make decisions, it's still a tool, not a person. You're the person and the one responsible for the decisions you make based on your tools.
Going back from the analogy, the problem is that we conflated software <engineers> with "coders".
A lot of people thought their job was to create code, we gave them a tool to generate a lot of code fast, and they truly think that "more code" = "more good"
Literally saw a video ad the other day which went like "I've always been cautious using Google's AI because it sometimes gets things wrong, but this time, it got it right!"
> I thought it had potential but I don't know how you'd fund it.
The same way we fund other social services here in Europe. If an individual is incapable of caring for themselves, the state is expected to care for them.