[malfist]

> nearly as accurate as a human book keeper

Anything to avoid using the metric system.

Though seriously, what is this metric? Why would I care if an LLM is accurate as a human bookkeeper? Humans aren't exactly known for perfect recall.

[adamkurkiewicz]

Hey, author of the blog post here.

I was one of the human book-keepers for this benchmark (the preparer; my co-founder verified the VAT submission once ready), and given that at the time of doing this I knew I was eventually going to use this data for evaluating the models, I was super careful. So I guess this is a "good book-keeper". In the previous company our book-keepers made lots of mistakes; some serious enough that we had to restate our company's accounts.

[wat10000]

You'd care if you had a human bookkeeper and you were considering replacing them with this company's AI bookkeeper.

[altruios]

Why would I want both a less accurate book keeper and to incur all of the liability of doing the books myself?!

[onraglanroad]

It's not less accurate. As commented above, the control knew they were being tested against the machine, so made sure to be super careful.

In everyday life the human is less careful, and the machine costs 1% of the human.

[wat10000]

Because it's way cheaper. Not saying it's a good tradeoff, but that appears to be their pitch.

[murderfs]

Presumably your current book keeper is not your slave, and you have to pay them...

[altruios]

...again: liability is the key issue. Cost savings a not exactly an isolated issue here.

[murderfs]

Then why only have one human bookkeeper? Surely two would be better, since you can compare their results. But then, perhaps you should hire three, so you can figure out which one is right.

[deno]

But what if the third bookkeeper is malicious? You need at least four bookkeepers to achieve byzantine fault tolerance with f=1.

[altruios]

Regardless of how correct they are, they assume liability: which is a metric that you do not improve with more bookkeepers.

[CamperBob2]

Got some bad news for you about that "liability" thing: it's always been on you. Read the fine print on your tax forms sometime.

[infecto]

It’s a service that provides an AI bookkeeper so it’s a pretty relevant metric.