[pscanf]

> The intention of this is to reduce hallucination on information extraction, right?

Correct.

> Also, how do you convert your docs / information into JSON documents?

Right now you have to add it yourself to the database. The idea is that you use Superego as the software in which you record your expenses / income / whatever, so the data is naturally there already.

But I'm also working on an "import from csv/json/etc" feature, where you drop in a file and the AI maps the info it contains to the collections you have in the database.

[iamspoilt]

Have you looked at qmd by any chance? Thoughts on that for sqlite vs qmd in this case? https://github.com/tobi/qmd

[pscanf]

Oh, interesting, I didn't know about this project, thanks for sharing!

I tried to implement something like this for the search functionality, but ended up going with "old school" lexical search instead.

Mostly because, in my experimentation, vector search didn't perform significantly better, and in some case it performed worse. All the while being much more expensive on several fronts: indexing time (which also requires either an API or a ~big local model), storage, search time, and implementation complexity.

And Superego's agent actually does quite well with the lexical search tool. The model usually tries a few different queries in parallel, which approximates a bit semantic search.