[itishappy]

It's a strong signal of unknown quality. Caveat emptor. Closed-source makes that extra difficult.

We are used to this, hence why folks here point it out.

[jloveless]

For QC this is a pretty solid tester (run on L or other compatible runtime) - focused on performance as it pushes multiple edges.

https://github.com/l-labs/master-benchmark

Which can also be run with minimal modifications on ngn and others.

For database the h2o.ai for duckDB really is a solid hard bench IMHO:

https://github.com/l-labs/db-benchmark/tree/main/l

[jloveless]

Even deeper is the full test suite ala

https://github.com/l-labs/rust_ipc/blob/master/tests/ipc_tes...

Which is ~6k tests (Rust is actually what I use to drive release testing)

[dragonwriter]

“It's a strong signal of unknown quality.”

Unknown quality is the absence of information, you can’t have a signal for it.

If you are using “unknown” as a euphemism for “poor”, which is the most obvious thing that makes the signal meaningful in the direction that seems apparent from context, why?.

[cygx]

Phrased differently, they claimed that the entropy of the distribution of some quality metric tends to be higher on the domain of projects with vibecoded sites compared to the domain of projects with handcrafted sites.

[naasking]

I don't believe for one second that this metric has been rigourously evaluated, it's also just based on vibes.

[cygx]

Sure. Doesn't mean the statement itself was nonsense, as the post I replied to implied...

[itishappy]

I explicitly do not mean "poor," that's why I choose "unknown." A better word would have been "variable." The point I intend to convey is that you need to use other signals to judge.

That's also why "closed-source" was mentioned in the same sentence. Two missing signals.

That's also why the author keeps pointing to the benchmarks and test suite, which is the signal they'd like us to judge by. Seems entirely reasonable to me!