[Bibabomas]

Hey, codebase-memory-mcp and semble are not exactly the same, but it's an interesting comparison, I'll put it on the todolist to check that out and add it to our benchmarks if feasible. If you ever get a chance to do this same comparison with semble it would be super useful feedback since these "real" scenarios are hard to benchmark/replicate.

[freakynit]

So, I just tested with semble. Your MCP integration did not work, and kept throwing error (Failed to connect to "semble": MCP error -32000: Connection closed) though I installed using documented manner (tried both: pip and ux methods).

Anyways, I made it work by making it generate relevant doc (using semble init), and then copying this into AGENTS.md, and then prompting it with this line:

""" Start by reading AGENTS.md in current folder. Now, the task::: `Explore the ingestion and search paths. Do not read README.md at all`. Prefer to use `semble` search for code search. Do not do new installation. semble is already available at `/Users/nitinbansal/.local/bin/semble` . """

The results are much better. Even better than my own setup, but, vary a lot. I did 4 runs:

95k/2.9k

25k/2.7k

71k/2.9k

37k/4.0k

[Bibabomas]

Sorry to hear about the MCP integration, that's definitely something we'll look into. If you have any info about your system or how to reproduce it please let me know. Very nice to hear about the results, thanks for checking this! The variance is interesting to see, that's probably non-determinism in the LLM rather than semble since semble is deterministic. But I'm guessing we can make that better with the prompt, I'll look into this.

[stavros]

uv failed for me once, but then worked the second time. I think it has to do with uv just taking a while to install the first time. Maybe if you pre-run the installation it'll work better.