[maccard]

Have you any example materials that show a harness that an agent can be pointed at an app with some sort of telemetry tool, the results it gained and the cost of doing so? Because my experience is the same as the parents - the LLM goes on massive tangents, and the more tangents it goes on the worse the results get.

[ndriscoll]

Not sure what materials your hoping for, but I've pointed codex at pprof application endpoints and just told it to identify hotspots and propose lowest hanging fruit/highest ROI items to fix, which I then approved it to open PRs for. Reduced application CPU usage by ~30% on loaded servers.

It's not magic. Fairly obvious oversights that I or someone else on the team could've found and fixed if we had looked, but it was never a business priority so we never did until I curiously spent 5 minutes asking it to do it for me.

[maccard]

Ideally a blog post on an open source project that shows harness, model, prompt and the before/after measurements plus the PR.

[colechristensen]

Ultimately the harness is me and the experience is like managing an unruly toddler. You have to pay attention to what it does and issue corrections. Skills and prompts and AGENTS.md do some, nested sets of agents do some, but over it all is me keeping track of what it's doing and needs to do.

You have to size the unit of work to its useful attention span, you have to have code architecture that is conducive to units of work, and you have to have tools like ticket managers, etc. that keep the big picture and smaller pictures in mind. Reverting from agile practices to more pre-planning whole project documentation and architecture decisions helps.

But in the end it's you. LLMs have their limits and need humans to direct them.

[maccard]

Gotcha, so no.

[bmurphy1976]

You don't need a "blog post." You need experience managing projects, tracking progress, coordinating conversations, mentoring less experienced engineers, etc. That experience comes with practice and time.

If you really need something to latch onto, there's plenty of educational material available about how to be an effective scrum master. Start there.

[colechristensen]

Indeed. I'm saying the human inside the engineering loop is essential. If a person needs training, particularly if the set of high level advice I set out in a reply is rejected, a blog post isn't going to cut it anyway.

Some AI skeptics paradoxically insist humans are irreplaceable and then get almost mad(?) when you discuss how humans are necessary in the loop. At least that's my interpretation of what is going on, it's hard to tell.

When building with LLMs I'm mostly product manager and QA, the whole task is having the judgement to evaluate and correct course. You can share advice but it's not a copy-paste situation where one set of prompts/harnesses/whatever solve all your problems forever. They have to be adjusted for a particular situation.

[maccard]

> If a person needs training, particularly if the set of high level advice I set out in a reply is rejected, a blog post isn't going to cut it anyway.

My problem with the “high level advice” is that the results it gives are bad. Sure, sometimes it gets it right, and it helps but I’d wager about half the time the results are just bad. Hence why I’m asking people to show their homework here. I think that the anthropic rust rewrite is great example - it shows yes you can do a lift and shift assuming you’re ok with those constraints.

> Some AI skeptics paradoxically insist humans are irreplaceable.

But yet in this comment thread we have people saying “just point it at your code and let it go” like [0], you saying “just follow the vague instructions and if you can’t get the results that I’m telling you you’ll get them it’s your problem. But it’s too circumstantial for me to be able to tell you how to work”

> where one set of prompts/harnesses/whatever solve all your problems forever

I don’t want forever, I just want to know what is actually working today, or this week, or this month.

[0] https://news.ycombinator.com/item?id=49122616

[colechristensen]

>But yet in this comment thread we have people saying

I'm writing for myself, not including myself in any group of people.

>I just want to know what is actually working today,

With LLMs I have facilitated the writing of about a million SLOC over the past 9 months.

- A GitHub "clone" with many enhancements which all of my development is hosted on

- A task manager as though kanban had a baby with a FIFO stack

- (in progress) a CAD kernel potential Parasolid competitor

- reverse engineered a vehicle diagnostics tool to diagnose and fix some car problems I was having more easily

- plenty of other small, half baked, or abandoned projects for one reason or the other

LLMs amplify me, they don't do things "for" me. They are working very well for me, but I have definitely hit their limitations in many places and transitioned back to the hardest problems to solve being on the human side of the equation.

What they can do for any particular person depends on the person and if they can come around to really grokking the human-machine relationship.

[simonw]

Tell the coding agent to install or build its own performance and profiling tools.

A few lines of Python with a time.time() call is enough to start iterating on performance improvements, and the really good models know how to use much more advanced existing tools.