[CSMastermind]

Some initial thoughts:

- It would be significantly more helpful if it could ingest my database schema.

- Why make me manually pick files instead of indexing them all and doing some RAG magic?

- Having used ChatGPT for similar tasks, I'm skeptical that step 2 you have, where it generates a high-level solution, will work all that well. Right now, LLMs work best for me when I'm giving them much more specific tasks and keeping the high-level plan in my head.

- This isn't really a technical design doc in the sense that I use the term? This might just be me using language incorrectly but I typically call a tech design doc a document that explains the inner workings of a system, normally with something like a UML or similar diagram. This just seems like todo list or subtask generation for writing a feature.

- Have you considered feeding the AST directly into the LLM? Honestly just curious how well it would understand the code if you did that vs. giving it the raw source.

Either way, congrats on the launch!

[ivanovm]

We'll queue up database schemas, good call.

On RAG, I have a lot of thoughts - it's my pet project to make it work really well. We started out down this road but in practice getting good results out of RAG has been shockingly hard on bigger, more complicated codebases. In many cases when we put this in front of devs that know their code, the RAG part has been a lossy approximation of what they already knew the relevant code to be, so they'd sit there and watch "is this thing going to find the code that I already know I need for this task"

I think the key here will be a robust, language-agnostic symbol graph that maps out high-resolution dependencies, so when you are retrieving `foo`, you can also efficiently retrieve the references to and from of `foo` to some depth (often greater than 1). Some languages have better tools for this kind of symbol resolution than others, but in many cases they're highly bespoke. Github's stack-graphs and Sourcegraph's SCIP are the best attempts I've seen at solving this in a generic way, but neither of them have gotten all the way there yet.

I've been half-seriously thinking about running neovim in headless mode and using it as an interface to all the LSP's for each language, and resolving all the edges of the symbol graph that way - I might still try it one of these days

On design doc convention - I think you're right, it's not completely 1-1 with the term today, and there's not a single definition that everyone agrees on. We're using it as the initial mental model of where we want to plug in, but there might be a better succinct way to describe it

Haven't tried the raw ast yet, curious how that goes

[robmck]

Thanks so much for the thoughts! Happy to respond.

-Schema: We agree. After seeing how many times it has asked to see the database schema when working on a task, that is definitely something we will be adding.

-Manual / RAG: We did some RAG magic in the first two iterations of our product and found it was really hard to get right when the task was not well defined and as the size of the codebase grew. We found that our users were watching it do the search for the files and basically saying, "Let's see if it finds the code file I already know we need." So, we decided to go manual so at least the user isn't unpleasantly surprised by the results while we determine the right path forward with RAG. Or maybe even combine both where the user gives an "entry point," and the AI figures out the rest of the files

-I think you're right to be skeptical. In our experience, the LLMs are really best at breaking down a task to be just one notch more specific. So, I think the output here wouldn't be the best for a more high-level task. Our plan to handle this is to allow plans to be solved recursively, where you continually create a tree of subtasks, and only when you get to a specific enough "leaf" do you actually go through with generating code.

-You're right; I think we used the term "technical design doc" more generally to steer people's thinking toward software design in English rather than code. I like that you mention diagrams, though, because we would love to incorporate diagrams into our documents as our tool and LLM's capabilities increase!

-This is a very interesting idea! We used a version of this (a select set of elements of the AST that we tried to rank in importance) in our initial RAG system to give the AI a "starting point" for the RAG code search. It worked pretty well. My concern would be that the quality of the search might then depend on the naming/structure of the code (I'm thinking some 1000-line mega functions or something)

Thanks so much for the thoughtful questions!

EDIT: I see Maks also responded and I think we both offer similar messaging with slightly different delivery :)

[StarlaAtNight]

In case it helps as a tool recommendation, I’ve had some success having ChatGPT writing & rendering MermaidJS (diagrams as code)

[robmck]

This looks amazing thanks for the recommendation! Much better than the LaTeX diagram thread I was starting to pull on.

[seveibar]

FWIW I run codegen for outputting db schema documention via CI and would recommend that for any project, it makes database exploration/diffs a lot simpler and of course can easily be ingested by AI.