Hacker News new | ask | show | jobs
by mercutio93 153 days ago
The persistent identity files are interesting but there's a cost problem. A recent paper (arxiv 2602.11988 https://arxiv.org/html/2602.11988v1) found context files increase inference cost by 20%+ with marginal performance gains; LLM-generated ones actually decreased success rates slightly.

Four identity files per agent injected every session feels like monkey patching coherence with context. Context isn't memory, it's just more tokens. The hard unsolved problem is cross-session learning without the bloat.

Curious if you've measured the token overhead of the identity files vs the performance gain they provide.
2 comments
selridge 153 days ago
That paper is hard to evaluate because their modal example of a “context file” is a bad practice that arose from early attempts at text-based agent guidance before we recognized that contextual instruction was the aim, not just “context.”

With a “context file” you’re almost guaranteed to add bloat without useful behavior change because it’s just a pre-set list of things that could be good to know about.

So the results of the study don’t generalize to every text file used for instruction or even most.

link
tarasyarema 152 days ago
100%, my tldr from that paper is: never use /init basically
link
selridge 152 days ago
It is WILD how much better things got when I ripped out all the "best practices" from https://agents.md/
link
tarasyarema 153 days ago
Yeah, saw that paper. And I have the following notes on it:

1. Agents update those files themselves, but currently with my oversight and guidelines (from the UI you can even see it's contents)

2. Measuring this is extremely hard, if not impossible. One of the goals of the swarm is to help me on random tasks that can span a lot of different pieces, not just implementing a feature.

Before last week, we did not have the memory and identity files. And, from an empirical pov, I can say that the general feel improved a lot. I see that in similar situations it does not perform the same mistakes. Also, what is stored in those files generally is something that the agent CAN NOT find using it's tools (like the paper suggest to avoid) which actually helps.

In any case, the swarm created a research on this topic a few days ago https://github.com/desplega-ai/agent-swarm/pull/86 maybe I'll iterate on it and see what we can get :D

link