Hacker News new | ask | show | jobs
by hexadecimal 371 days ago
I loop through all Q&As at the end of the billing period and ask gemini if this users query was solved.

Something fair for customers, thats the goal. But right now my priority is seeing if anyone gives a f** about this to begin with.

Regarding hallucinations: It's not 100% solvable, but can be done pretty well in my opinion with prompting (aka only answer based on facts you see in the documentation)

P.S. if you have docs and want a more "live demo" you can sign up and just put in the docs url and you'll have a live demo without installing the script onto the docs site.
1 comments
tom1337 371 days ago
Interesting approach using Gemini for checking whether the answer helped. I wonder how that would work if the response is hallucinated.

As an example: Stripe has sandboxes, I wanted to know how to create them via the CLI but found nothing in the docs. Asked the AI and it was like "Just do stripe sandboxes create <name>". The command does not exist and the whole functionality is not planned to be added. If another AI now reads this it could interpret it as answered successfully?

Just thinking - but I like the idea of the tool. Would love to use it but currently don't have any public docs to test with. Good luck with your project :)

link
hexadecimal 371 days ago
As an example: Stripe has sandboxes, I wanted to know how to create them via the CLI but found nothing in the docs. Asked the AI and it was like "Just do stripe sandboxes create <name>". The command does not exist and the whole functionality is not planned to be added. If another AI now reads this it could interpret it as answered successfully?

--> I would say so. Not quite sure what Stripe is using on their docs, but sounds like its been given free reign and not grounded in actual docs.

Easiest way is to weed out hallucinations, hasnt been a problem for me at all to be honest.

P.S. For anyone reading, happy to refund all answers that you deem shouldnt be billed.

link
tom1337 371 days ago
Just tested the tool on an open source documentation. Looks and feels great but somehow after the initial Preview in the onboarding flow I cannot get the Preview to work. It seems to always answer with the wrong dataset and suddenly says "The provided documents describe Agionic, an Al tool for documentation, and do not contain details about UI component libraries like Grid." Not sure whats happening here.

https://imgur.com/a/XiVmosl

link
hexadecimal 371 days ago
I'm guessing the page was refreshed and the preview lost context to the right customer ID...

Thanks for trying it!

link