[Sammi]

- don't learn from what you tell them

Whenever I have a model fix something new I ask it to update the markdown implementation guides I have in the docs folder in my projects. I add these files to context as needed. I have one for implementing routes and one for implementing backend tests and so on.

They then know how to do stuff in the future in my projects.

[Clent]

They still aren't learning. You're learning and then telling them to incorporate your learnings. They aren't able to remember this so you need to remind them each day.

That sounds a lot like '50 First Dates' but for programming.

[embedding-shape]

> They aren't able to remember this

Yes, this is something people using LLMs for coding probably pick up on the first day. They're not "learning" as humans do obviously. Instead, the process is that you figure out what was missing from the first message you sent where they got something wrong, change it, and then restart from beginning. The "learning" is you keeping track of what you need to include in the context, how that process exactly works, is up to you. For some it's very automatic, and you don't add/remove things yourself, for others is keeping a text file around they copy-paste into a chat UI.

This is what people mean when they say "you can kind of do "learning" (not literally) for LLMs"

[bloppe]

While I hate anthropomorphizing agents, there is an important practical difference between a human with no memory, and an agent with no memory but the ability to ingest hundreds of pages of documentation nearly instantly.

[averageRoyalty]

That is true, but does it actually matter if the outcome is the same? GP is saying they don't need to remind them.

[troupo]

The outcome is definitely not the same, and you need to remind them all the time. Even if you feed the context automatically they will happily "forget" it from time to time. And you need to update that automated context again, and again, and again, as the project evolves

[Sammi]

They document how to do something they just figured out. They store/memorise it in a file.

It's functionally working the same as learning.

If you look at it like a black box, then you can't tell the difference from the input and output.

[_flux]

I believe LLMs ultimately cannot learn new ideas from their input in the same way as they can learn it from their training data, as the input data doesn't affect the weights of the neural network layers.

For example, let's say LLMs did not have examples of chess gameplay examples in their training data. Would one be able to have an LLM play chess by listing the rules and examples in the context? Perhaps, to some extent, but I believe it would be much worse than if it was part of the training (which of course isn't great either).

[latentsea]

50 first new Date()

[MangoToupe]

Ah, so it's like you have a junior developer that can't learn

[kvirani]

Can this additional prompt from you also be automated? I do this too but I forget sometimes. I don't know if a general rule will be enough ?

[troupo]

> I add these files to context as needed.

Key words are these.

> They then know how to do stuff in the future in my projects.

No. No, they don't. Every new session is a blank slate, and you have to feed those markdown files manually to their context.

[latentsea]

The feeding can be automated in some cases. In GitHub copilot you can put it under .github/instructions and each instructions markdown file starts with a section that contains a regex of which files to apply the instructions to.

[blktiger]

You can also have an index file that describes when to use each file (nest with additional folders and index files as needed) and tell the agent to check the index for any relevant documentation they should read before they start. Sometimes it will forget and not consult the docs but often it will consult the relevant docs first to load just the things it needs for the task at hand.

[troupo]

So, again, they don't learn.

[latentsea]

Do you want them to?

[troupo]

I would. Getting tired of redirecting them in correct directions from scratch every time

[MangoToupe]

Doesn't this also consume context

[theshrike79]

Having a good prompt file ("memory") is an artform.

The AI hype folks write massive fan fiction style novellas that don't have any impact.

But there's middle ground where you tell the agent the specific things about your repo that it doesn't know based on its training. Like if your application has a specific way to run tests headless or it's compiled a certain way that's not the default average.

[troupo]

This works surprisingly well for Claude: https://github.com/obra/superpowers (in the context of rather small side projects in Elixir).

Unless, of course, the phase of the moon is wrong and Claude itself is stupid beyond all reason

[latentsea]

Yes.

[theshrike79]

https://agents.md/

AGENTS.md exists, Codex and Crush support it directly. Copilot, Gemini and Claude have their own variants and their /init commands look at AGENTS.md automatically to initialise the project.

Nobody is feeding aything "manually" to Agents. Only people who think "AI" is a web page do that.

[troupo]

Ah yes. Agents.md is a magical file that just appears out of thin air. No one creates it, no one keeps it updated, and LLMs always, without fail, not only consult it but never forget it, and in every new session know precisely what changed in the project and how to continue.

All of them often can't even find/read relevant docs in a new session without prompting

[theshrike79]

Literally every single CLI-based Agent will show you a suggestion to run /init at startup.

And of course it's up to the developer to keep the documentation up to date. Just like when working with humans. Stuff don't magically document itself.

Yes "good code is self-documenting", but it still takes ages to find anything without docs to tell you the approximate direction.

It's literally a text file the agent can create and update itself. Not hard. Try it.

[troupo]

> Just like when working with humans. Stuff don't magically document itself.

Humans actually learn from codebases they work with. They don't start with a clean slate every time they wake up in the morning. They know where to find information and how to search for them. They don't need someone to constantly update docs to point to changes.

> but it still takes ages to find anything without docs to tell you the approximate direction.

Which humans, unsurprisingly, can do without wiping their memory every time.