[diggan]

Looks interesting and useful if the accuracy numbers are as told. Kind of sad it's only available via a remote API though, makes the product more like a traditional SaaS-API. The marketing keeps talking about "models" yet the actual thing you use is only the API, would have been nice to be able to run locally. Although I do understand that it's harder to make money in that case.

I got curious about what datasets you used for training the models? Figured the easiest would be to scrape git repositories for commits from there, but seems there are also quality issues with an approach like that.

[eborgnia]

Open source git repos are a really good place to get data -- it requires a lot of munging to get it into a useful format, but that's the name of the game with model training.

It's on the roadmap to make public evals people can use to compare their options. A lot of the current benchmarks aren't really specialized for these prompt-to-app use cases