[aspenmartin]

That’s ok but at what point is this getting into conspiracy territory? You have just said there is nothing you would believe to the contrary, but then by definition that’s not exactly a very thoughtful or insightful position.

[OtomotO]

I never said that I am not willing to believe the contrary.

I am not willing to believe the contrary from strangers on the interwebs or PR departments of companies who want to sell me something.

If people I genuinely trust tell me about their experiences, I am willing to try again.

But yes, if it doesn't work for me (for whatever reason, could be that I am holding it wrong), then I can accept that it works for everyone but me and still not use it.

Also "scientific" doesn't mean what it used to mean. When the n is small or it's just anecdotes (I am aware of the irony) blown out of proportion I really can't take the data and conclusions seriously

[aspenmartin]

N isn’t small, science means what it’s always meant, statistics is a thing, and what you’re describing is just putting your trust in a very poor quality benchmark. You said you would not trust any data that indicates something that contradicts your opinion. Benchmarks are not PR they are designed by a variety of institutions completely outside the control of frontier labs. Again congratulations on your conspiracy theory.

[OtomotO]

> Again congratulations on your conspiracy theory.

I am neither impressed nor offended by any kind of argumentum ad hominem. I sincerely hope you have a wonderful day!

> Benchmarks are not PR they are designed by a variety of institutions completely outside the control of frontier labs.

I don't give a crap about how good a shovel may be in a theoretical experiment when it's digging in sand, when I work with hard earth.

The ones I had a look at are mostly absolutely meaningless to my actual work.

> and what you’re describing is just putting your trust in a very poor quality benchmark.

And here is where we disagree fundamentally, so we can leave it at that.

Ex falso quodlibet

[aspenmartin]

> I don't give a crap about how good a shovel may be in a theoretical experiment when it's digging in sand, when I work with hard earth.

I don't know what this means, benchmark tasks are pretty hard and pretty in domain.

> The ones I had a look at are mostly absolutely meaningless to my actual work.

You've looked at 100,000 benchmarks?

> And here is where we disagree fundamentally, so we can leave it at that.

Yes we do disagree, yet one of us has statistics and rigor and one of us doesn't.

[OtomotO]

> You've looked at 100,000 benchmarks?

What about "The ones I had a look at" was unclear?

> Yes we do disagree, yet one of us has statistics and rigor and one of us doesn't.

Yup, that's true. So again, have a nice life!