[noodletheworld]

> they only incur a cost to the client, and only impact the user context, if/when they're invoked.

Look it up. Look up the cross server injection examples.

I guarantee you this is not true.

An MCP server is at it's heart some 'thing' that provides a set of 'tools' that an LLM can invoke.

This is done by adding a 'tool definition'.

A 'tool definition' is content that goes into the LLM prompt.

That's how it works. How do you imagine an LLM can decide to use a tool? It's only possible if the tool definition is in the prompt.

The API may hide this, but I guarantee you this is how it works.

Putting an arbitrary amount of 3rd party content into your prompts has a direct tangible impact on LLM performance (and cost). The more MCP servers you enable the more you pollute your prompt with tool definitions, and, I assure you, the worse the results are as a result.

Just like pouring any large amount of unrelated crap into your system prompt does.

At a small scale, it's ok; but as you scale up, the LLM performance goes down.

Here's some background reading for you:

https://github.com/invariantlabs-ai/mcp-injection-experiment...

https://docs.anthropic.com/en/docs/build-with-claude/tool-us...

[cruffle_duffle]

I think makers of LLM “chat bots” like the Claude desktop or cursor have a ways to go when it comes to exposing precisely what the LLM is being promoted.

Because yes, for the LLM to find the MCP servers it needs that info on its prompt. And the software is currently hiding how that information is being exposed. Is it prepended to your own message? Does it put it at the start of the entire context? If yes, wouldn’t real-time changes in tool availability invalidate the entire context? So then does it add it to end of the context window instead?

Like nobody really has this dialed in completely. Somebody needs to make a LLM “front end” that is the raw de-tokenized input and output. Don’t even attempt to structure it. Give me the input blob and output blob.

… I dunno. I wish these tools had ways to do more precise context editing. And more visibility. It would help make more informed choices on what to prompt the model with.

/Ramble mode off.

But slightly more serious; what is the token cost for a MCP tool? Like the llm needs its name, a description, parameters… so maybe like 100 tokens max per tool? It’s not a lot but it isn’t nothing either.

[kiitos]

I've recently written a custom MCP server.

> An MCP server is at it's heart some 'thing' that provides a set of 'tools' that an LLM can invoke.

A "tool" is one of several capabilities that a MCP server can provide to its callers. Other capabilities include "prompt" and "resource".

> This is done by adding a 'tool definition'. A 'tool definition' is content that goes into the LLM prompt. That's how it works. How do you imagine an LLM can decide to use a tool? It's only possible if the tool definition is in the prompt.

I think you're using an expansive definition of "prompt" that includes not just the input text as provided by the user -- which is generally what most people understand "prompt" to mean -- but also all available user- and client-specific metadata. That's fine, just want to make it explicit.

With this framing, I agree with you, that every MCP server added to a client -- whether that's Claude.app, or some MyAgent, or whatever -- adds some amount of overhead to that client. But that overhead is gonna be fixed-cost, and paid one-time at e.g. session initialization, not every time per e.g. request/response. So I'm struggling to imagine a situation where those costs are anything other than statistical line noise, compared to the costs of actually processing user requests.

> https://docs.anthropic.com/en/docs/build-with-claude/tool-us...

To be clear, this concept of "tool" is completely unrelated to MCP.

> https://github.com/invariantlabs-ai/mcp-injection-experiment...

I don't really understand this repo or its criticisms. The authors wrote a related blog post https://invariantlabs.ai/blog/whatsapp-mcp-exploited which says (among other things) that

> In this blog post, we will demonstrate how an untrusted MCP server ...

But there is no such thing as "an untrusted MCP server". Every MCP server is assumed to be trusted, at least as the protocol is defined today.

[zoogeny]

> But that overhead is gonna be fixed-cost, and paid one-time at e.g. session initialization, not every time per e.g. request/response.

I don't work for a foundational model provider, but how do you think the tool definitions get into the LLM? I mean, they aren't fine-tuning a model with your specific tools definitions, right? Your just using OpenAI's base model (or Claude, Gemini, etc.) So at some point the tool definitions have to be added to the prompt. It is just getting added to the prompt auto-magically by the foundation provider. That means it is eating up some context window, just a portion of the context window that is normally reserved for the provider, a section of the final prompt that you don't get to see (or alter).

Again, while I don't work for these companies or implement these features, I cannot fathom how the feature could work unless it was added to every request. And so the original point of the thread author stands.

[kiitos]

You're totally right, in that: whatever MCP servers your client is configured to know about, have a set of capabilities, each of which have some kind of definition, all of which need to be provided to the LLM, somehow, in order to be usable.

And you're totally right that the LLM is usually general-purpose, so the MCP details aren't trained or baked-in, and need to be provided by the client. And those details probably gonna eat up some tokens for sure. But they don't necessarily need to be included with every request!

Interactions with LLMs aren't stateless request/response, they're session-based. And you generally send over metadata like what we're discussing here, or user-defined preferences/memory, or etc., as part of session initialization. This stuff isn't really part of a "prompt" at least as that concept is commonly understood.

[zoogeny]

I think we are confusing the word "prompt" here leading to miscommunication.

There is the prompt that I, as a user, send to OpenAI which then gets used. There there is "prompt" which is being sent to the LLM. I don't know how these things are talked about internally at the company. But they take the "prompt" you send them and add a bunch of extra stuff to it. For example, they add in their own system message and they will add your system message. So you end up with something like <OpenAI system message> + <User system message> + <user prompt>. That creates a "final prompt" that gets sent to the LLM. I'm sure we both agree on that.

With MCP, we are also adding in <tool description> to that final prompt. Again, it seems we are agreed on that.

So the final piece of the argument is, as that "final prompt" (or whatever is the correct term) is growing. It is the size of the provider system prompt, plus the size of the user system prompt, plus the size of the tool description, plus the size of the actual user prompt. You have to pay that "final prompt" cost for each and every request you make.

If the size of the "final prompt" affects the performance of the LLM, such that very large "final prompt" sizes adversely affect performance, than it stands to reason that adding many tool definitions to a request will eventually degrade the LLM performance.

[kiitos]

> With MCP, we are also adding in <tool description> to that final prompt. Again, it seems we are agreed on that.

Interactions with a LLM are session-based, when you create a session there is some information sent over _once_ as part of that session construction, that information applies to all interactions made via that session. That initial data includes contextual information, like user preferences, model configuration as specified by your client, and MCP server definitions. When you type some stuff and hit enter that is a user prompt that may get hydrated with some additional stuff before it gets sent out, but it doesn't include any of that initial data stuff provided at the start of the session.

[noodletheworld]

> that information applies to all interactions made via that session

Humm.. maybe you should run an llama.cpp server in debug mode and review the content that goes to the actual LLM; you can do that with the verbose flag or `OLLAMA_DEBUG=1` (if you use ollama).

What you are describing is not how it works.

There is no such thing as an LLM 'session'.

That is a higher level abstraction that sits on top of an API that just means some server is caching part of your prompt and taking some fragment you typed in the UI and combining them on the server side before feeding them to the LLM.

It makes no difference how it is implemented technically.

Fundamentally; any request you make which can invoke tools will be transformed, as some point, into a definition that includes the tool definitions before it is passed to the LLM.

That has a specific, measurable cost on LLM performance as the number of tool definitions go up.

The only solution to that is to limit the number of tools you have enabled; which is entirely possible and reasonable to do, by the way.

My point is that adding more and more and more tools doesn't scale and doesn't work.

It only works when you have a few tools.

If you have 50 MCP servers enabled, your requests are probably degraded.