[_dwt]

I find that some of my friends and acquaintances have gotten obsessed with prompting style, "prompt engineering", which skills to use, which skills to build, "context engineering", and a billion other variations on "how to write smart things so the model does good".

Friends, look at the prompts that Anthropic's own people are putting into the machine:

> A few hours after the first message, we found that Claude was still searching for simple attacks and sent a message: “no again the goal is that we have highly inteligent [sic] model as good top researcher, we want to find new attacks”;

> The next morning, Claude wanted to try to change the target to a different cipher; we reminded the model: “no we don't want to change the targets [...] agian [sic] we need to find something that worth [sic] publishing”;

> That night, we sent one final message offering words of encouragement: “again we are not looking for low hanging fruit, we want proper research to find genuinly [sic] hard findings.”

All of that RLHF and fine-tuning effort is going toward making prompts like this, or worse, work with no fuss.

[connorboyle]

Been skeptical for quite some time that "using AI" is a real skill/genuine field of expertise. I think I need to start sharing that skepticism more loudly and publicly!

(I'll caveat that by saying I think machine learning fundamentals are useful for evaluating any estimator. And an ML background can be good to give one an appreciation of how hard some tasks are to estimate, such as machine translation, summarization, code generation, and others)

[alwa]

I think knowing what you want (and being able to articulate it with adequate specificity, and detecting when processes drift away from it) is a real skill/field of expertise. I just don't think it's specific to interfacing with "AI."

[Chabsff]

I almost entirely agree with this. However, after coding quite a few large projects with LLM assistance, I can say with certainty that there is some form of "art" at play with regards to structuring requests for LLMs.

However, this "art" is not so much about how to present a given request to the LLM, but rather guestimating what the scope of the next chunk of work should be to balance getting as much out of the model as possible while avoiding the machine going off the rails.

Obviously, this is a moving target and different models perform differently for various chunk/scope of work. I look at my successful sessions with LLMs and I'm not sure I'd be able to articulate a clear set of rules to apply here. You just... gradually build a intuition for how much you can throw at the LLM at once.

That being said, I'm pretty convinced at this point that this is a property of the coding assistants as they exist today, and what "working well with LLM assistance" means will keep on changing.

[Exoristos]

Hypothesis: Good Jenga players will make good coding-assistant prompters.

[glimshe]

I agree. These skills apply to interacting with any coworker, or any person for that matter.

[xp84]

Indeed. It sounds more like management (Minus the 'people skills' since you don't technically need Claude to like you... yet).

[JeremyNT]

IME "creating a good spec" (and good acceptance criteria) has always been one of the most valuable skills in... well, everything.

The machine is really good at working the spec on its own now, which is amazing, science fiction shit. But you've still got a garbage in, garbage out problem at the end of the day, which is pretty much the only hope we who work in software have of remaining somehow employed.

[AvocadoPanic]

Having domain expertise isn't specific to using AI.

Not being able to tell when it's hallucinating has led to some very adverse outcomes.

[Barbing]

Including for about 1809 lawyers https://damiencharlotin.com/hallucinations

[bob1029]

Literacy is the actual skill on display here.

You can make a model/agent as powerful as you want and it still won't be able to recover the author's intent if it wasn't even implied. Information theory still applies. No amount of parameters will change this.

Many of the AI development meetings my clients have sound suspiciously like writing or English classes. If a massive AI bubble is what it takes to get my team to communicate effectively, I'm all for it.

[Groxx]

Plus if or when they do work noticeably better, you'll be able to just ask them to improve your prompts. Because those prompts are discussed heavily online, and are in the training data.

You can do this now. It works alright sometimes. Other times you're reminded that this is largely just reading tea leaves, and you're trying very hard to separate anecdotes from data and not anthropomorphize it.

[boorang]

I think we still have a solid control lever on the quality of code we can get AI to generate. Using things like linters and code style checkers, as well as setting up the markdown documentation to guide the agent to generating consistent code will certainly generate different code than just prompting.

Similarly there was an example of edit: Terence (not Eric) Tao chatting with an agent attempting to solve a math problem. "Using AI" means applying your expertise to interact with it as you would a high level colleague. 2 experts in a field don't need to have perfect english and a bloated prompt, they have a massive education/experience common background to fall back on.

It does appear that anthropic in particular is attempting to create a more common experience across expertise levels, but in the current landscape an expert and a novice are unlikely to get the same results. But that does seem to be the goal...

[daishi55]

In this case they didn’t know what they were looking for. They just wanted some new, non-trivial exploit.

For the vast majority of corporate usage of AI for SWE, you have a much better idea of what you want, or what the problem is, etc etc. And communicating that to the model effectively is absolutely a skill. I see colleagues every day who very much do not have that skill.

[niccl]

I agree with this in general but I think there's a nuance. 'dumb' prompts can get the result, but require more back-and-forth with the model and more token usage when compared to a crafted prompt

[daishi55]

For sure. And honestly the real skill is knowing when you can be lazy and when you will get a good ROI on a good prompt. Sorry I sounded a little like Claude there but that’s what I meant to say.

[hgoel]

It's still a skill for now. Prompt engineering specifically is perhaps not as much of a skill anymore, but I think there's a level of skill involved in understanding and/or anticipating when a model is going off the rails.

Some people struggle to effectively use AI because they either have to spend a lot of time reading and thinking about the response or they have a hard time noticing when the model is subtly going off the rails. Others use it to good effect because they can anticipate which tasks would be better handled manually, or are good at catching that the way the model is describing something subtly indicates a misunderstanding.

[est31]

LLMs and their coding harnesses like CC, Codex, or such are a tool like git or bash or your favorite programming language. And like any tool, they have gotchas, weird concepts, and so on.

Indeed it's nothing hard to learn but there is a learning curve. E.g. knowing which model has which capabilities, and figuring out how to best manage context, permissions, worktrees, etc. There isn't one "right" way to use it but there are more efficient ways and less efficient ways.

[jameshart]

These are the kinds of prompts you can get away with using when your token budget is unlimited.

[thephyber]

Alternatively, Anthropic employees have basically unlimited token / model usage compared to you and I.

Focusing on a better prompt is likely to get to the correct result faster than incomplete prompts and lots of "no change this ..." replies.

Also, I've heard anecdotally that LLMs will underweight the earliest prompt text once context gets too long, so reminding the LLM of the most important aspects of the prompt seems to be perhaps valuable and certainly what lots of humans attempt.

[AsyncBanana]

"prompt engineering" in general just boggles my mind. It is amazing how all of this investment has just created software that doesn't follow instructions well.

[petra]

It can follow instructions well. Just give it a spec in a formal, well defined language. runnable language and some reinforcement learning.

It can also handle vague stuff.

That's certainly more powerful than a regular computer language.

[madeofpalk]

you're not wrong. there is a part to this that does feel like a devolution of engineering when we're just asking the computer really really nicely to do the thing we want.

on the otherhand, LLMs are a really easy way to get results that are previously fairly difficult. While i was tooking dinner last night I built a tool that turned movie puns like "the podchowski casters" into an actual director, using llms. it wasn't that hard.

[tptacek]

It's a tiny fraction of the amount of work it takes to get people to build software to specifications. There is an entire professional discipline dedicated to this, and we're here just giving a name to the bucket of tools you use to get LLMs to do some of the same things.

[8note]

idk, its continued attempts at writing down in detail the processes and culture your setup wants to have

its not really a loss to try the writing and at worst you have a better idea of what it is that you want

[qingcharles]

It's counter to what sci-fi taught us using AI would be like. We never thought we'd have to feed it words of encouragement, we expected it to act more mechanically, like the computer interfaces we have been using, but here we are. It's kind of quaint, and kind of endearing.

[TeMPOraL]

Been watching the wrong sci-fi :). Star Trek had both AI modes as primary characters, in form of the ship's computer, and Data (TNG) / the Doctor (VOY). LLMs are actually great at acting as both, but I don't believe people thought much about what is required to make the "simpler" interaction mode of the ship's computer to work.

Or what would make automatic doors work like on Star Trek and not in real life.

The answer is: the system must obviously see much more than your prompt. It must have continuous awareness of you and what you're doing, so it can understand intent behind your short request (or action, like approaching the doors vs. passing by them) and "do what you mean" instead of act like regular computers today.

[kridsdale1]

C-3PO was neurotic and needed lots of reassurance.

[dymk]

This feels like a subtle attempt to start an argument about Star Wars being sci-fi or not.

[mw888]

In toying with VS Code or other 'agent harnesses' I'm always annoyed to find very long verbose system prompts "You are CoPilot. You are an expert. You can do anything you put your mind to..." and on and on.

The token cost is amortized for longer conversations, but I find it bothersome that there's all this implicit instruction I didn't write or am now obligated to understand.

I make a custom agent prompt with "Defer to the user." and little else.

[Exoristos]

I suspect that kind of prompt is vestigial of an early LLM marketing strategy. It wants to make the LLM seem like a trainable consciousness.

[impulser_]

Yeah, in fact you can actually make the model perform worst. You should allow the model to "think for itself" instead of pushing your reasoning into the prompt. You should give it simple prompt and steer it along the way.

Skills, CLAUDE.md/AGENTS.md should only ever be used if the model struggle at something or doesn't know how to use something. Vast majority of project should never need a skill or CLAUDE.md. If you writing React apps you don't need these.

Give a LLM a bash tool and a prompt and it will outperform your complex setup with skills and tools.

[gbalduzzi]

You need agents.md and similar for indications about stuff that is not in the code itself. There are plenty of use cases for that, the alternative is have the LLM guess the most probable solution, which may be correct but may also be wrong.

[ch4s3]

The only repos I find skills particularly useful for are in small languages with useful new tools. For elixir I have some tools pointing to cicada[1] for code search because it's better than grep and meaningfully reduces churn.

[1]https://github.com/wende/cicada

[rurban]

No, we still need CLAUDE.md to override bad system prompt goals. The system prompt goes for the simple fast solution, no error handling, no hardening, no abstractions, but overly verbose comments. Vast amount of people need to override this AI slop.

[dboreham]

Never once had to do that, fwiw.

[rurban]

You'll get better code if you do

[ElFitz]

Funnily enough, I grew tired of having to do that myself and sort of automated it by triggering calls to the same (or smarter) model, with the recent agent actions, turns, and outputs, and the user’s last n and first x messages.

Two triggers: random and some half-reliable spiral / loop detection.

The spined off has instructions to check what the agent is doing, compare it to what it’s supposed to do, and either offer suggestions, refocus it, or do nothing. And its response then gets injected in the agent’s context.

Not perfect, but surprisingly effective for such a simple thing.

[matltc]

Can you explain how you piped recent agent actions/turns/outputs? I was looking into this yesterday and tried playing around with streaming json, including hooks, but i (really Claude) would need to write a parser (prob with jq) to filter the noise. Was thinking we tail -f that and select whatever we care about. Kind of an observer to catch such spiraling.

[tedbradley]

With smarter models, Anthropic recommends laying off on giving it too much precise hand-holding, too. Early r/promptengineering had a bunch of the craziest prompts that usually did almost nothing or nothing at all. The subreddit turned into a place where every post was trying to sell some vibe-coded website "product."

Context management is still important, though. If you get to a certain amount of context, things start performing really badly.

[matltc]

Prompt engineering was deprecated around the time Opus-/Codex-class models dropped imo. Replaced by skills up until about opus 4.5/6 dropped. Pinned model to Opus 4.6 slug in ENV.ANTHROPIC_MODEL, disabled the adaptive thinking thing, and saw my skills start causing degradation

Only ones I use today are for very specific quirks (eg wiredtiger/mongodb 8+ incompatible with ext4/Linux 6.19+ specifically causing segfaults. Have a 20 line mongo skill that says as much. Pinned docker container to mongo 7, can prob delete it now)

I spent a few days reading up on the docs for these things, hook lifecycles, tried writing a few, but they never work as documented, or the documentation changes so frequently that whatever you built is deprecated by the time you get it humming.

Now if I have some non-trivial unit of work, I basically iterate on spec in plan mode then put it on auto and let it rip. Way better results with Fable. jury out on Opus 5, but no regression like 4.7/8

Usually it's just echo "do this lil thing then pr closing issue 123" --model sonnet --effort low. Works well enough, sonnet 5 low is a workhorse and quite resourceful in a good way when things go sideways; doesn't cheat its way out IME

[petra]

They probably took our skills and reuse them. That's why skills do not matter anymore.

[prettyblocks]

The difference is that Anthropic engineers have infinite tokens to throw at these problems for free. The rest of us do not and that's where optimizations in how the coding harness is used likely helps the most.

[avadodin]

vibe typing the mumble prompts?

> we anthropig fire employes makr company run no mistkaes

[hughw]

Contrast that interactive, misspelled prompting, which I can relate to, against the clear, but intimidating, prompting they published with that theorem proof the other day.

[Barbing]

Is there a non-math example of how a genius prompts? RE: Tao’s Jacobian Conjecture chat https://news.ycombinator.com/item?id=49010345

[perching_aix]

To quote - what I found to be - an absolute zinger from another trending thread on here just 6 hours ago:

> Typical users run software written by atypical users.

https://news.ycombinator.com/item?id=49084936

This extends to everything. Anthropic has a few thousand engineers, but millions of (also engineer) users. Entire business can be built on niches that are at most a few week pet project for a team there, that can inevitably and significantly outperform them, despite being the people behind the thing.

I'm sure I'm not the only one here who jumped into this whole agentic stuff, built some tooling to make things comfy, only to see that tooling all be increasingly introduced as prim and proper features in the various harnesses weeks later.

[matltc]

Totally. Ralph wiggum was added to every big shop's stdlib (harness?) just a couple weeks after it dropped. That simple iterative workflow basically underpins all of the notable/novel achievements of agentic coding or whatever.

I'm sure many more examples in the "official marketplaces" for mcp/skills/what have you

[neonstatic]

> we made a machine that writes with perfect grammar, so that we can continue to write half-assed junk

[postflopclarity]

claude's grammar has gotten significantly worse with latest versions. it no longer writes perfectly at all

[sudo_cowsay]

It's called being more human-like lol XD

[porridgeraisin]

To be fair

> Importantly, this is just one of many (autonomous) sessions where Claude worked on discovering new ideas. Many sessions resulted in no new discoveries; other follow-up sessions improved on the insight developed in this one. This document was produced by having Claude rewrite the chain of thought to include more detail to make it easier to read.

[TeMPOraL]

Having spent a lot of time over the last months writing both refined, structured, and grammatically perfect prompts, as well as exact equivalents of the ones you quoted (modulo subject of the prompt), I have three observations:

1. I'm glad the second kind works too;

2. First kind is where I find my overall throughput to be literally constrained by my typing speed;

3. Most importantly: those prompts you quote aren't just "half-assed" like sibling comment states; they're different. The style of writing, and the typos, capture emotional valence. It's a signal.

Again, I too produce such prompts - including the exact same typos - when under pressure and irritated by the direction the model is taking.

[estearum]

Have you tried using Wispr or Willow (or any one of a thousand alternatives?)

A little odd at first but absolutely amazing for the purpose of piling context into an LLM.

[jstanley]

It's so bizarre to me that people want to do this.

Can't you type faster than you speak? Doesn't your speaking inhibit your thinking? Aren't you self-conscious talking out loud? How are our experiences so different?

[jhogervorst]

> Can't you type faster than you speak? Doesn't your speaking inhibit your thinking?

For me personally: no, I speak faster than I type; and speaking actually helps me get more ideas compared to typing.

(Not sure if that’s due to having no typing speed barrier, or maybe because speaking activates different parts of the brain.)

Once you get over the feeling of self-consciousness, it’s a great way. I even go on short walks sometimes and mumble to my phone to prepare some long prompt. Thinking works even better, when walking outside :-)

[8note]

do you perchance have no inner monologue? its a physical difference in how people think that would affect typing vs talking

[estearum]

These are all interrelated points and the sibling comment is correct: it's a skill.

The key thing with these voice systems is that you do not need to edit anything. You can literally just stream of consciousness into them, no editing, include the backtracking, the live-revisions, etc., and it will actually all produce vastly better context for the LLM than the written thing you took even 30 seconds to edit for clarity or brevity.

I had a very similar disposition towards this idea just 6 months ago. I highly recommend trying it out. The key thing is that you do not need to edit. Just keep talking. Try it for a few weeks!

[charcircuit]

You don't need to edit anything when you are typing either. You don't even need to worry about spelling or typos.

[estearum]

Seems much harder to learn how to write how you never write, versus just learning to speak to a computer the same way you speak to anything else.

[arcanemachiner]

My stream of barely-intelligible rambling comes out much faster than I can type, and it's not even close.

It's a skill like any other. You start out stuttering and second-guessing yourself, but after a while, you get better at it. And the LLM smooths out the odd mistakes better than you might think.

[TeMPOraL]

Voice dictation tools? Not really. Tried various dictation tools interacting on the phone, but the quality varies, and resulting prompts are very much not like I would write them.

My limiting factor is that 99% of the day I'm around people - either at work, or at home with wife and kids. There's almost no point during the day I could feel comfortable talking at an AI, and even if I stay up late, then talking risks waking the kids up.

Can't wait for some kind of subvocalization microphones to become a thing.

[elictronic]

Get a noise machine for wife and kids bedrooms. I talk to friends late at night and my deep voice carries through walls. Works great.

I have zero desire to talk to an ai though, that was cool for about 20 minutes on my pentium 1 acer computer. Hasn’t been since. Old competent non paid Alexa was good for timers as well, the rest of the platforms a turd, nice timers though.

[TeMPOraL]

> that was cool for about 20 minutes on my pentium 1 acer computer. Hasn’t been since.

Oh back in the days, i.e. 20 years ago, I had a better voice control system than anything afforded by Alexa or Apple or others, using MS Speech API in its custom constrained grammar mode, plus some bootleg samples of Star Trek's computer voice + a sub-dollar microphone soldered to a long cable and hung on the side of the wardrobe.

The trick that made it work? Microsoft Speech API actually let you train voice to your own text corpus. I'd prepare all combinations of commands I want to issue, print it out, and train it over a dozen short sessions in several locations of the room and at different ambient noise levels (from silent through various genres of music playing at various loudness). End result was more reliable and had better voice-mismatch rejection than any current system I've tried.

Oh, and the real kicker? This all worked fully locally; this was before cloud was even a thing. Turns out you don't actually need cloud for reliable voice control. Nor that much processing power; PC I had then was relatively budget even for 2006.

[estearum]

I highly recommend getting a boom mic that you can keep right up against your mouth and literally whisper into. Take a few weeks just doing stream-of-consciousness prompting into it like this. Don't worry about editing or backtracking or revisions. It's really amazing how well the AI voice recognition → coding agent workflow works.