Almost no one knows if a project/business idea will be successful or not, so it's not much use asking. It's more productive to ask smart, experienced people how to best validate and execute an idea. People generally give useful and actionable feedback based on their experiences. Just make sure you understand who you're talking to when evaluating someone's advice.
"understand who you're talking to when evaluating someone's advice."
Good you mentioned this, found out to this is a crucial part as well: Always perceive the advice you get depending on that person's background and interests (e.g. your target group, or domain-foreign expert).
I think that people suggest RAG, also because the models develop so fast that very probably the base model you finetune on will be obsolete in a year or so.
If we are approaching diminishing returns it makes more sense to finetune. As the recent advances seem to happen by throwing more compute to CoT etc maybe the time is close or has already come.
There are so many chain types it is easier to do the abbreviations. Basically extend a RAG to have a graph to influence how to either critisize itself or perform different actions. It has gotten to the point where there are libraries for define them. https://langchain-ai.github.io/langgraph/tutorials/introduct...
Fine tuning to a specific codebase is a bit strange. It's going to learn some style/tool guidance which is good (but there are other ways of getting), at the risk of unlearning some generalization it learned from looking at 1,000,000x more code samples of varied styles.
In general I'd suggest trying this first:
- Large context: use large context models to load relevant files. It can pickup your style/tool choices fine this way without fine tuning. I'm usually manually inserting files into context, but a great RAG solution would be ideal.
- Project specific instructions (like .cursorrules): tell it specific things you want. I tell it preferred test tools/strategies/styles.
I am curious to see more detailed evals here, but the claims are too high level to really dive into.
In generally: I love fine tuning for more specific/repeatable tasks. I even have my own fine-tuning platform (https://github.com/Kiln-AI/Kiln). However coding is very broad. Good use case for foundation large models with smart use of context.