[ignoramous]

Those charts show pass@k metric (expectation at least k generated samples are correct out of n) on OpenAI and Octopack problem evals for code.

WaveCoder: https://arxiv.org/abs/2312.14187 (section 3.2)

Octopack: https://github.com/bigcode-project/octopack

[srikanth235]

While testing internally, Mistral worked well. But these models are just starting points. Will add support for the models WaveCoder-Ultra-6.7B, WizardCoder-33B, Magiccoder-S-DS-6.7B, etc soon.