[wintermutestwin]

I recently spent over an hour trying to get ChatGPT to give me some pretty simple rsync commands. It kept giving me command line parameters that didn't work on the version of rsync on my mac. With ~50% of the failures, it would go down troubleshooting rabbit holes and the rest of the time it would "realize" that it was giving incorrect version responses. I tell it to validate each parameter against my version moving forward and it clearly doesn't do that. I am sure I could have figured it out on my own in 5 mins, but I couldn't stop watching the trainwreck of this zeitgeist tech wasting my time doing a simple task.

I am not a coder (much), but I have to wonder if my experience is common in the coding world? I guess if you are writing code against the version that was the bulk of its training then you wouldn't face this specific issue. Maybe there are ways to avoid this (and others) pitfall with prompting? As it is, I do not see at all how LLMs could really save time on programming tasks without also costing more time dealing with its quirks.

[goalieca]

> I tell it to validate each parameter against my version moving forward and it clearly doesn't do that.

I would like an AI expert to weigh in on this point. I run into this a lot. It seems that LLMs, being language models and all, don't actually understand what i'm asking. Whenever i dive into the math, superficially, it kind of makes sense why they don't. But it also seems like transformers or some secret sauce is code up specially for tasks like counting letters in a word so that AI doesn't seem embarrassing. Am I missing something?

[nerdponx]

LLMs are next-token prediction models. They "understand" in that, if the previous 1000 tokens are such-and-such, then they emit a best guess at the 1001th token.

It "knows" what rsync is because it has a lot of material about rsync in the training data. However it has no idea about that particular version because it doesn't have much training data where the actual version is stated, and differences are elaborated.

What would probably produce a much better result if you included the man page for the specific version you have on your system. Then you're not relying on the model having "memorized" the relationship relationships among the specific tokens you are trying to get the model to focus on, instead just passing it all in as part of the input sequence to be completed.

It is absolutely astounding that LLMs work at all, but they're not magic, and some understanding of how they actually work can be helpful when it comes to using them effectively.

[pglevy]

Our low-code expression language is not well-represented in the pre-training data. So as a baseline we get lots of syntax errors and really bad-looking UIs. But we're getting much better results by setting up our design system documentation as an MCP server. Our docs include curated guidance and code samples, so when the LLM uses the server, it's able to more competently search for things and call the relevant tools. With this small but high-quality dataset, it also looks better than some of our experiments with fine tuning. I imagine this could work for other docs use cases that are more dynamic (ie, we're actively updating the docs so having the LLM call APIs for what it needs seems more appropriate than a static RAG setup).

[bashy]

Words are tokens so it can't really 'see' the word(s), it just knows how to link them.

[beefnugs]

did you feed the context a link or examples of the exact version of the documentation?

I am not an expert but i assume if there are any basic knowledge of the tool then it will try to use it as it knows chunks of some old version. And it wont likely decide to search for the newest docs, you have to tell it search for exact version docs, or feed the exact version docs to context

[goalieca]

I find context _sometimes_ work but more often than not i rephrase the question a million times, try to break it down into smaller problems, .. whatever. It seems like it just doesn't "understand".

[latexr]

> I am not a coder (much), but I have to wonder if my experience is common in the coding world?

It is, yes. Surely someone will come and tell you it doesn’t happen to them, but all that tells you is that it ostensibly isn’t universal, but still common enough you’ll find no end of complaints.

> Maybe there are ways to avoid this (and others) pitfall with prompting?

Prompting can’t help you with things not in the training set. For many languages, all LLMs absolutely suck. Even for simple CLI tools, telling an LLM you are on macOS or using the BSD version may not be enough to get them to stop giving you the GNU flags. Furthermore, the rsync change in macOS is fairly recent so there’s even fewer data online on it.

https://derflounder.wordpress.com/2025/04/06/rsync-replaced-...

> As it is, I do not see at all how LLMs could really save time on programming tasks without also costing more time dealing with its quirks.

And that’s the best case scenario. It also happens that people blindly commit LLM code and introduce bugs and security flaw they cannot understand or fix.

https://secondthoughts.ai/p/ai-coding-slowdown

https://arxiv.org/abs/2211.03622

[Helmut10001]

Usually, in these edge cases, I go to the documentation page and dump all pages as Markdown into the AI tool (most often Gemini, due to token count). This context engeneering has helped a lot to get better answers. However, it also means I am consuming sometimes 1 Million tokens on relatively simple problems. Like recently, when I needed to solve a relativey simple but specific MermaidJS issue.

[erichocean]

Ask Gemini Pro 2.5 to build the rsync command and then give it the man page for your version of rsync. It should succeed the first time.

Here's a command to copy the man page to the clipboard than you can immediately paste into aistudio (on a Mac):

    man rsync | col -b | pbcopy

As a general rule, if you would need to look something up to complete a task, the AI needs the same information you do—but it's your job to provide it.

[wintermutestwin]

So I paste the man page into llm and tell it to only give me parameters that are in that page? Even if it obeyed, it would still choke on how to exclude hidden MacOS kruft files from the copy...

[erichocean]

No, you don't need to "tell it to only give me parameters that are in that page."

Here's the entire prompt:

    I need the rsync command to copy local files from `/foo/bar` to `~/baz/qux` on my `[e-mail]` server.
    Exclude macOS cruft like `.DS_Store`, etc. Here's the man page:

    <paste man page for your rsync that you copied earlier, see above>

[nehal3m]

If you have trouble talking to an AI, how do you ever expect to merge with Neuromancer’s twin?

[athrowaway3z]

> Maybe there are ways to avoid this (and others) pitfall with prompting?

Not sure about Codex, but in Claude Code you can run commands. So instead of letting it freestyle / guess, do a:

`! man rsync` or `! rsync --help`

This puts the output into context.

[dangerface]

Yup even when you tell it your version it forgets pretty quickly. Or agrees it messed up and assures you this time it will give you the correct info for your version number then gives you the same command.

Javascript is a nightmare as they change everything constantly. PHP has backwards compatibility for everything so its not really an issue.

It also gives out dated info on salesforce, and im not just talking about the latest and greatest, it recommends stuff that was deprecated years ago.

[chrisweekly]

Why involve an LLM at all, if you're looking up docs for a particular tool like rsync?

[wintermutestwin]

Lots of reasons! First off: where else do I go to learn this stuff? Man pages are reference for people who work in CLI all the time and not for virgin learners as they are are necessarily packed with the complete lexicon but with barely a thought to explaining real world examples of common tasks. There are a million Linux websites with the same versioning issues and inadequate explanations. I guess I could buy an oreily book and learn the topic end to end even though I will only need to know the syntax of a couple commands.

With an LLM, I can get it to tell me what each parameter it suggests actually does and then I can ask it questions about that to further my understanding. That is a massive leg up over the knowledge spaghetti approach...

[latexr]

> There are a million Linux websites with the same versioning issues and inadequate explanations.

Where do you think the LLM is getting its data from? At least on the website you can see the surrounding discussion and have a chance of finding someone with the same problem as you, or learning something tangential which will help you further down the line.

> With an LLM, I can get it to tell me what each parameter it suggests actually does and then I can ask it questions about that to further my understanding.

If it’s giving you wrong flags, why do you assume the explanations it gives you are accurate? LLMs can make those up just as well.

What you should do is verify the given flags on the man page. Not only will it clarify if they exist, it will also clarify if they’re what you’re looking for, and will likely even point to other relevant options.

[rsynnott]

> There are a million Linux websites with the same versioning issues and inadequate explanations.

So, instead you ask a magic robot to recite a fuzzily 'remembered' version of those websites?

Bear in mind that an LLM does not _know_ anything, and that, despite some recent marketing, they are not 'reasoning'.

[rsynnott]

Learn to read documentation. It's really a very important skill, and many people were becoming _somewhat_ deficient in it even before our good friends the magic robots arrived, due to Stackoverflow et al.

[wintermutestwin]

Bah! Coders need to learn to write documentation. The tldr for rsync was hilarious! Actually, this is one place where LLMs are currently useful! Instead of getting LLMs to write code, get it to write documentation.

[joshribakoff]

One pitfall is the LLM hallucinates, might sometimes seem to fulfill your requirements but it can subtly break down. The man pages could be used in conjunction to fact check your understanding.

[heckelson]

I like the tldr pages to learn the most common features and use cases of new command line tools! I think it's great, albeit, a bit slow sometimes

[wintermutestwin]

tldr pages are a great idea, but the execution is a total fail. Looking at the rsync entry, it fails to provide the most blatantly common requirements:

I just needed two commands: one to mirror a folder from one drive to another, updating only the changes (excluding all of the hidden MacOS cruft). And another command to do a deep validation of the copy. These have to be two of the most commonly used commands right???

In the end, I felt that messing around with precious data without being 100% certain of what I am doing just wasn't worth it so I got a GUI app that was intuitive.

[harvey9]

Not the op, but I sometimes find the official documents hard to parse. Not looking at rsync in this case. On the other hand I have the same experience with LLMs as the op.

Big thanks to all the doc writers who include vignettes right in the documents.

[Ancapistani]

I’m using “aichat”, and have all my man pages in a RAG. It’s far faster to query that than it is for me to read through it manually.

[dist-epoch]

I guess you never used ffmpeg, there is a whole industry of "how to do X with ffmpeg"

[KronisLV]

> I recently spent over an hour trying to get ChatGPT to give me some pretty simple rsync commands.

Try N times, adding more context about the environment and error messages along the way. If it doesn't work after those, try other models (Claude, Gemini, ...). If none of those work on whatever number of attempts you've chosen, then LLMs won't be able to help you well enough and you should save yourself some time and look elsewhere.

A good starting point is trying for 10-20 minutes, after which point an LLM might actually become slower than you going the old fashioned way of digging into docs and reading forum posts and such. There are also problems that are a bit too complex for LLMs as well and they'd just take you in circles no matter for how long you try.

[meowface]

As a programmer I have noticed this problem much more with command help than with code. Maybe partly because the training data has way more, and more diverse, code examples than all the relevant permutations and use cases for command argument examples.

[BryanLegend]

I've recently been misled by ChatGPT a lot as well. I think it's the router. I'm on the free plan so I assume they're just being tight with the GPU cycles.

[wintermutestwin]

I am on a $20 plan and using the "thinking" version of 5.

[doug_durham]

I'm a coder and I've never had your experience. It usually does an amazing job. I think that coders have an advantage because there are many questions I would never ask an LLM because of my intuition on what would work well and what wouldn't. In your case I would have dumped the output of `rsync --help` into the context window once I saw it wasn't familiar with my particular version of rsync. That's they way these tools work.

[dimgl]

LLMs are a very specific kind of beast. Using an `rsync --help` or getting any kind of specific documentation into context would have unblocked you.

[BinaryIgor]

All the time I find that if I know the stack and tools I am working in, it's faster to just write code on my own, manually; If I want to learn on the other hand, LLMs are quite useful - as long as you understand (or learn to) and validate the output

[alain94040]

Instead of just saying: rsync on my system is version 3.2, have you tried copy/pasting rsync --help? In my experience, that would be enough for the AI to figure out what it needs to do and which arguments to use. I don't treat AI like an oracle, I treat it like an eager CS grad. I must give it the right information for the task.

[righthand]

What is the cost (in tokens/$$$) of spending an hour restating questions to a chat bot vs typing `man rsync`?

[danielbln]

Telling the agent to execute "man rsync" and synthesize the answer from there is probably the cheapest and most efficient option.

Letting some detached LLM fumble around for an hour is never the right way to go, and inversely sifting through the man page of rsync or fmmpeg or (God forbid) jq to figure out some arcane syntax isn't exactly a great use of anyone's time either, all things considered.

[righthand]

Sifting through just means you don’t know how to use the man interface to search/grep (which from a discoverability perspective is fair). However I think reeling through an Llm (using an agent or not) for a task that probably could take <10mins at $0, demonstrates enthusiasts disregard for a good set of research and reading habits.

All of this is an attempt at circumventing RTFM because you’re privileged enough to afford it.

Just lay yourself down on the WALL-E floating bed and give up already.

[wintermutestwin]

The “fucking” manual is obtuse, overly verbose, and almost always lacking in super clear real world examples. That you claim it is a <10 min problem demonstrates experts disregard for the degree of arcane crap, a beginner needs to synthesize.

I am backing up and verifying critical data here. This is not a task that should be taken lightly. And as I learned, it is not a task that one can rely on an LLM for.

[righthand]

I disagree. The topic is rsync which is well documented. The path forward might not work in every situation but the manual has good examples too. This is probably true for what Llm users use it for 90% of the time. To hack together things that are well known and well documented into a result. Llms arguably only work because these tools ffmpeg, rsync, etc were already solving problems and widely used and documented. So burning energy to have a computer look up commands because you couldn’t spend 10 minutes reading yourself could be a waste of time and money. Where as having to spend time researching is likely only a waste of time only.

[wintermutestwin]

>rsync which is well documented

Here is the man page entry for the --delete flag:

--delete is used. This option is mutually exclusive with --delete-during, --delete-delay, and --delete-after.

Hilarious!

Reading and understanding the rsync command would take much more than 10 mins and I am not a total newb here.

[pletnes]

Lower than the cost of a meatbag-office with heating, cooling and coffee for the time spent reading the manual.

[righthand]

How is the cost of an office relative? Llms didn’t destroy offices. You can type `man rsync` without any of that.

[Yoric]

Pretty much my experience, yes.