[george3d6]

Looks quite interesting, already pinned this in the relevant slack channel :)

To be honest I'm rather happy with how the internal benchmark suite is turning out, but to some extent you are inviting bias by creating them yourself. On top of that, it doesn't hurt to have more benchmarks.

At the end of the day it's a combination of: * How much work is it to integrate (easy to measure) * How visible is it, i.e if we actually find something interesting will be visible and legible to others (ify to mesure, citations, stars, etc are some invitation) * How useful it is to "improve" the library (hard to measure, and what we aim to be good at is a moving target)

So realistically that's the equation I have to judge in terms of adding a new benchmarks suite, and it's very annoying because you'll note the most important things are the hardest to measure.

Would you want people to integrate with this now or would you rather wait a few weeks/months/years until it matures more? If the former, can you give a few details regrading where to start (README is fairly barren), if the later please ping me ([e-mail]) when you think it could be ready to try.

Anyway, any open benchmark library is a step in the right direction, thanks for working on this :)

[cweill]

Thanks for your feedback! Based off the description of how you already do things, I'd say you're ahead of the curve as far as rigorous model quality benchmarking. You should absolutely hold off of using nitroml for a few months until it's more mature. It's very much pre-prerelease in a build-in-the-open sense. :) I'll shoot you an email once it's ready for anyone to try out. When the time comes, we'll have a blog post to announce it, and will include proper documentation.

And, congrats on the launch!