Hacker News new | ask | show | jobs
by pseudony 5 days ago
And what do you base this on ?

How does one objectively quantify how it stacks upnto another model ?

Or even, what is your subjective evaluation based on ?

I really wonder - because I have just finished a fully vibe-coded gtk/rust/lua application with me basically writing 7% of the code (all in one module) and GLM 5.1 writing the rest. We haven’t had regressions, confusion or anything else. And I am pretty damned sure I couldn’t manage this one year ago with claude code and Sonnet.
1 comments
lejalv 5 days ago
What harness, if you don't mind sharing?
link
pseudony 4 days ago
Course not :)

I use pi (pi.dev).

I suspect some of the issue id that some harnesses are over-optimized for particular models and their preferences (tool calling, instructions to soften their deficiencies etc).

Pi is much more minimalist - probably a fairer point of comparison.

A different suspicion of mine is that some people over-specialize in a given model - or maybe become lazy with their prompts or suffer from skill issues.

Fwiw - I generally maintain a specs/ folder as I code.

I never use “plan” mode - I just tell the LLM to make no code changes, but discuss design with me.

At some point I am happy (I typically ask it to summarize and write the actual spec), I review; correct misunderstandings, ask for follow-up questions, we incorporate the additional details into the spec and move on.

I often have TODO’s/tasks in those specs too and I regularly update progress on them. It also happens that I ask the LLM to review my code (actual) against the spec and search for differences- we then resolve them. Sometimes by modifying the code; sometimes by modifying the spec.

For starters, I write an overview spec - nail down the big concepts and architectural choices at a high level. Moderately complicated facets of the application get their own spec - we write these as and when it gets relevant.

I think it helps the model a lot because I can refer to specs I feel relevant in drafting new specs or when solving tasks. And LLMs are generally better at proactively consulting these specs when getting an overview of the application and its design ahead of implementation.

link