Hacker News new | ask | show | jobs
by youssefabdelm 1168 days ago
TL;DR?

(Yes, GPT-4 is prob better... but by how much and on what?) A table would've been easier

3 comments

You just couldn't scroll to the bottom of the page, eh?

The conclusion is that neither 3.5 nor 4 are good enough because for anything none trivial they generate code that is often subtly wrong. Might still speed up somebody new to the language/project/learning or I would say: with additional tooling/plugins/"prompt engineering"/tinkering the author might get useful results.

> So, should you use GPT to generate your OpenAPI validations? Probably not... yet... I'm looking forward to repeating this experiment with GPT-6, and maybe GPT-7 will be able to generate an JSONSchema compiler and replace this library altogether.

from https://github.com/E-xyza/Exonerate/blob/master/bench/report...

(I believe the author is significantly underestimating the pace of progress)

Specific numbers are at https://github.com/E-xyza/Exonerate/blob/master/bench/report.... GPT-4 does significantly better.

Not the creator, but as it's GitHub, feel free to repeat the experiment and submit a pull request with a better method; peer review.
Not everyone has the time or expertise to create a pull request. But there are issues allowed on this repo! Create an Issue if you'd like the author to address something, in my opinion.
I’ve seen a good number of things get addressed simply because an Issue crossed a threshold of votes/comments over time. Really can make voices heard on the anon internet. Big respect to anyone working for free who takes them seriously, that takes integrity.