[saurik]

This is like the simplest task you can give a software developer, as it is nigh-unto merely a document "translation" task, without much real thought required. If an LLM is failing to do this task, why do we hope whatever chain of reasoning it is about to embark on would work?

[frabjoused]

This is very naive. How many different APIs have you authenticated with and connected to? Just the big ones? What happens when the docs are wrong or incomplete?

[saurik]

> This is very naive. How many different APIs have you authenticated with and connected to? Just the big ones?

I mean, a lot? I have multiple times felt like that was my entire life for weeks or months on end during the past over three decades of doing software development...

(If we expand the scope a bit to network protocols, as opposed to just "APIs", I was even the person who first spiked nmap's protocol scanning and detection logic.)

To wit, I am one of those people who pretty much never use an SDK provided for an API: if I have to, I will rather reverse engineer the protocol using a disassembler.

(This then being part of why I've won millions of dollars in bug bounties, as part of my relentless drive to always operate at the lowest level presented to me.)

But, regardless, can we move past trying to attack my credibility on software, and shift back to more productive forms of analysis? (Why did this become so personal?!)

> What happens when the docs are wrong or incomplete?

If we posit that the documentation for the API is wrong, so we should this MCP description / wrapper, as both were written by the humans charged to enable this function.

And, of course, the real point is whether the task is easier than the thing we are trying to do... even writing a correct tree map is much harder than an API client.

^ Both of these arguments can be made by someone who doesn't even do software development, helping us try to understand why MCP is being hyped up as a new paradigm.

[frabjoused]

Congrats on the bug bounties!

I’m not hyping or defending MCP at all: I’m just saying AI can’t figure out APIs well enough to be something you can promise as a product.

I founded an integration platform so definitely a developer and I’ve been living these problems every day.

[nsonha]

https://news.ycombinator.com/item?id=43304457

Also > ...if an LLM is failing to do this task...

It CURRENTLY fails to do so, PREDITABLY and securely. What are you gonna do about that? Keep throwing more data into it and hope to start building stuff on top, one day?

[saurik]

I don't understand your response as it feels like it is also my point, and something I would say to you? :( If the LLM cannot do basic tasks such as this, what are you going to do? Keep throwing more "tools" at it and hope to start building stuff on top of it? People are trying to get these things to do complex multi-step reasoning tasks, including making changes to their codebase (?!), automating behaviors as "agents" that need to predictably and secure function... but, somehow, as part of the same explanation, it can't even translate API documentation, one of the most trivial code tasks a human does? There are reports that OpenAI is planning to sell "mid-tier agents for software development costing possibly $10,000 a month" (to quote The Information)... and yet, here we are, claiming it isn't up to writing API boilerplate? Help me make this make sense :(.

[nsonha]

> Keep throwing more "tools" at it and hope to start building stuff on top of it

Yes and it's working? People ARE CURRENTLY building things.

They are NOT currently whinning that LLM is not smart enough so they must sit and wait for the next model to be able to code any problem, (again RELIABLY) on demand.

> People are trying to get these things to do complex multi-step reasoning tasks, including making changes to their codebase (?!), automating behaviors as "agents" that need to predictably and secure function...

You understand that all these are powered by tools calling underneath? The planning and orchestration of tasks follows a structure, to be fed into tools. This is why a plain model cannot do shit, people have to make products with the right tools on top to make a llm behave the way they want, and not just chit chat endlessly.

The abitrary execution approach, if it ever works, is by building tools and MCP servers for code execution. Because obviously it's not the LLM server who executes code.

You clearly have never thought about how to actually build any of these things.

> ...writing API boilerplate

Tool/function calling can be anything, it's you who decided that you can only use it for API boilerplate. Does the word "function" always mean boilerplate in progranming?

[pests]

I feel like you are being overly hostile and you and saurik are both arguing the same point... maybe? Following this thread has not been easy, not sure which positions each of you are even defending.

[saurik]

I honestly clearly don't understand the downstream disagreement myself, am getting tired of being shouted at today, and feel like entering this part of the thread by replying to someone who was clearly already past their breaking point to defend what they had already claimed was an "idiotic take that many people seem to have", was a mistake ;P.

[imtringued]

How does the model retrieve documentation? Through an MCP server?