[brunoluiz]

Isn’t engineering an exact science while prompt engineering is completely not?

Although, even software engineering being an exact science, it is a funny one: most of us don’t get certified as like, let’s say, mechanical engineers do. Would they say we are engineers?

So perhaps the “engineer” term got overloaded in recent years?

[FishInTheWater]

Prompt "engineering" is just writing prayers to forest faeries.

Whilst BASIC/JavaScript/etc are all magic incantations to a child, a child will soon figure out there's underlaying logic, and learn the ability to reason about what code does, and what certain changes will do.

With prompts, it's all faerie logic. There is nothing to learn, there are only magic incantations that change drastically if the model is updated.

Worse yet, the incantations cannot be composed. E.g. take the SQL statement "SELECT column FROM table WHERE column = [%s]". For any given string you insert here, the output is predictable. You can even know which characters would trigger an injection attack.

With prompts you cannot predict results. Any word, phrase, or sequence of characters may upset the faeries and cause the model to misbehave in who knows what way. No processing of user-input will stop injection attacks.

Whilst it's dubious to call current software development practices "engineering", it's utterly ridiculous to do so for prompt-writing.

[broast]

I don't get where this sentiment comes from. I build software specifically on the concept of predictable results from llm's being composable.

Sure, the results are not deterministic in that 100% of the time the exact prompt returns the exact same result, but you can tune your prompts so that 100% of the time they give you a valid result in the result category you were seeking, and with a specific probability distribution of available choices.

Prompts are functions that can take concrete input and create a probabilistic output that can be automated upon. Especially if you only need to output one token, i.e a number, boolean, word, object reference. And for obvious reasons - the further you forecast out in a sequence the less accurate you will be.

As long as you don't change the underlying model, in a massive model with billions of parameters, there are definitely mechanisms and behaviors to discover that you can reason about.

[FishInTheWater]

but you can tune your prompts so that 100% of the time they give you a valid result in the result

You can't though, that's the issue. Illustrative here are tokens like "SolidGoldMagikarp", but this does happen to "normal" sequences of tokens as well.

There is no filter you can build to keep out such mistakes, any set of otherwise normal tokens could trigger the model to produce wrong output.

Because of how large these models and most prompts are, even slight changes in things like attention can cascade into extremely different results.

there are definitely mechanisms and behaviors to discover that you can reason about.

It's faerie logic. The behaviours are mere trends and observations, not underlaying truth.

The faeries reward you for offering them fruit. But offer them apple which fell from the tree exactly 74 hours ago down to the second and they'll kill you. There is no way to know ahead of time which things will upset them.

The risk here is that you're fooled into believing these systems are understandable, that you know how they work, and that you'll mistakenly use them for something where the wrong results have consequences. You'll stop double-checking the output, all humans are lazy like that, and then you'll have disaster on your hands.

[l33t233372]

You can reasonably expect an LLM to respond appropriately often. Which percentage of the time depends on the details, but it’s not much more magic than expecting the bridge you built to hold up.

[penjelly]

you could do a sort of validation of output by prompting the llm repeatedly with the same prompt and then compare the responses to eliminate outliers. I do feel like this stuff is magic though, just wanted to provide a counterpoint.

[kordlessagain]

In "The Information," James Gleick discusses a concept related to our current discourse. In the days when computers were merely an array of switching circuits, luminaries such as Claude Shannon believed that "thinking" could be captured in a structured format of logical representation.

However, even with formally composable languages like JavaScript, a semblance of unpredictability — akin to the "faerie logic" metaphor — still persists. Languages evolve over time; Python, for instance, with its various imports that constantly disrupt my code, serves as a good example. This is perhaps the reason behind the emergence of containers to ensure code consistency.

While some elements may be more "composable" than others, it appears increasingly unrealistic in today's world to encapsulate thought processes or interactions with systems within a rigid logical framework. Large Language Models (LLMs) will keep evolving and improving, making continual interaction with them unavoidable. The notion that we can pass a set of code or words through them once and expect a flawless result is simply illogical.

I firmly believe that any effective system should incorporate a robust user interaction component, regardless of the specific task or problem at hand.

[FishInTheWater]

It's not so much about formal logic, but general predictability.

even with formally composable languages like JavaScript, a semblance of unpredictability — akin to the "faerie logic" metaphor — still persists

And they're ridiculed for it, and as you state, we design around them or replace such systems entirely.

making continual interaction with them unavoidable

Technology is never unavoidable or "inevitable". We can choose not to use it, or when to use it.

The notion that we can pass a set of code or words through them once and expect a flawless result is simply illogical.

Yet that is what we expect when we put these systems into production use, especially when many proposed use cases are user-facing and subject to injection attacks.

Whether it be the writing of adcopy, the processing of loan applications, or generating code, mistakes in these tasks have very real consequences.

[kordlessagain]

I don't disagree we can choose to use it or not, but my point was more meant to indicate that, if we want a good experience with LLMs, we have to continue to interact with them to achieve good results.

Reminds me of raising kids...

[jmoak3]

You're too right.

We need to move away from prompt-engineering - it's AI-Management. You pretend you're speaking to another (albeit confusing/confused) person when extracting work from a model. You're coaxing things out of it based on hearsay and mysticism that work most of the time. Sounds a lot like AGILE and free pizza to get a junior to stay late and deliver on time.

That's not engineering, that's management.

[ml-anon]

It’s so refreshing to see someone actually write this about prompt writing. It makes an extremely refreshing change from Twitter AI influencers posting their ridiculous prose as some marvel of harnessing LLMs.

[l33t233372]

You cannot predict results in _any_ domain with 100% accuracy, especially not in most engineering domains.

Why do you think rockets explode, bridges collapse, etc.

[zerodensity]

This was magical really made my day. Thanks for this.

[mellosouls]

Software engineer has always been a daft, grandiose term that seems aimed at prestige rather than reality in the majority of cases.

If coders can call themselves engineers, no reason why anybody else solving puzzles for a living can't.

[xmcqdpt2]

In my opinion, coding is a craft. As software has only existed for like 70 years, we are more like the guilds building cathedrals in the middle ages than like modern civil engineering.

One day I think there will be true software engineering. When that happens you won't be able to start software projects without certifications, and most people (or programs!) who actually do the coding will be following careful plans and instructions from the engineers who designed the project.

I for one am very happy software isn't fully professionalized yet!

[tough]

> One day I think there will be true software engineering. When that happens you won't be able to start software projects without certifications, and most people (or programs!) who actually do the coding will be following careful plans and instructions from the engineers who designed the project.

Sounds really bad.

Software can't fall on your head and kill you, not all of it at least.

Different software should require different professionals building it.

And it's usually not about the software but about the management telling the engineers to take shortcuts or whatever (Boeing comes to mind)

[jes]

You might enjoy this article on the Therac-25 [1]. It's kind of the standard example of how errors in software can wind up harming people. I have written medical device software for about 30 years. In my experience, delivering high quality software for Class B and Class C devices is both challenging and expensive.

https://en.wikipedia.org/wiki/Therac-25

[rmilejczz]

Every software developer should know this story, it is a humbling and important lesson. Yes luckily most of can’t ship code that accidentally kills people but we can absolutely empathize with the conditions which led to it happening.

I recommend reading the entire postmortem, http://sunnyday.mit.edu/papers/therac.pdf yes it is quite long but if you write code in any capacity it’s worth the read

[jes]

Thank you so much for the link to the postmortem. I will be sharing it and discussing it with my colleagues. We are currently working on the embedded software for an AED.

[groestl]

You will need a software engineer to argue in your favor before the court ;)

[groestl]

Don't want to argue against you, since the term is definitely misused for signaling a lot of things, but in a stricter sense, software engineering is not about the puzzle, it's about the whole pipeline surrounding it.

[skwirl]

I’m guessing you aren’t from the US because in the US there is not much prestige in the title of engineer, but it seems to be a title people in Europe get weirdly hung up on. If anything, software engineering positions are more prestigious than most traditional engineering positions here.

[hospitalJail]

I was a real engineer for a decade before switching to programming.

I cannot use the word software engineer, since its nothing like real engineering.

Real engineering was def harder, more math intense, and the stakes were sooo much higher. While many software problems can cause you to lose money, engineering problems can cause you to lose time. Yes it sucks your CAD designer had everything on a 0.05 degree angle and it costs 1M to redo the tool, but it also costs 16 weeks to redo the tool. We'd even offer to pay absurd money, prevent future business, etc... to get the tool done in 8 weeks, but its impossible to get it done faster. Now everything in the company is 8 weeks behind schedule.

Anyway, real engineering was harder, but programming pays soooo much more money. Its a demand thing, not a difficulty thing.

[philovivero]

> While many software problems can cause you to lose money, engineering problems can cause you to lose time.

I'm guessing you were trying to say something else here. I literally cannot think of a single software engineering problem I've ever encountered that didn't cost time. By your definition, then, software engineering is engineering. Your claim and your definitions are at odds with one-another.

Also, you don't directly claim it, but you seem to imply, that software engineering can't have real-world consequences... or something? As another reply points out, sometimes software is in the critical path for things like rockets and airplanes, where mistakes cost lives.

And some people making software for less life-altering systems take their craft just as seriously. Some people think that losing $10M every single second while their software is failing is a big deal.

Are you claiming people who write HFT code, ad arbitrage code, code that powers the front page of Apple, Amazon, Microsoft, and Google are just cowboying it through the day, doing nothing special?

Overall I just find this comment very confused. Maybe you could put some thought into what you're trying to say, and say it better?

[SirMaster]

So if you have your PE license and are writing software that controls a commercial airplane system or a spacex rocket, you are not an engineer doing software engineering?

[9659]

From Wikipedia: "A software engineer is a person who applies the engineering design process to design, develop, maintain, test, and evaluate computer software."

"The engineering design process, also known as the engineering method, is a common series of steps that engineers use in creating functional products and processes. The process is highly iterative - parts of the process often need to be repeated many times before another can be entered - though the part(s) that get iterated and the number of such cycles in any given project may vary.

It is a decision making process (often iterative) in which the basic sciences, mathematics, and engineering sciences are applied to convert resources optimally to meet a stated objective. Among the fundamental elements of the design process are the establishment of objectives and criteria, synthesis, analysis, construction, testing and evaluation.[1]"

It is not dependent on the problem domain, rather on how the work is performed.

[hospitalJail]

Ahh I forgot to put my usual caveat.

Assembly and safety critical C can be considered software engineering.

Anything with abstraction, no.

[SirMaster]

I agree that far too many people call themselves software engineers when they really aren't.

But I just mean to say that software engineering itself is definitely still a real thing and there are many people out there that can and should call themselves software engineers.

[edgyquant]

“Real engineer,” is such an insufferable way to make your point. I guess only civic engineering is “real.” If you don’t like the term software engineer you’d be real mad if you found out about sound stage engineers. Being an engineer isn’t about engineering solutions to business problems; nope it’s having to deal with bureaucratic hurdles that slow you down that makes one an engineer.

[tough]

it's just -soft- engineering bro, get over it.

It's always been a demand thing, not a difficulty thing.

[zaphirplane]

Don’t want to be that person but (tm) Engineer has to mean something other than person doing something.

Knot engineer, floor polishing engineer, water spillage cleanup engineer, tax minimization engineer, heart engineer, bus operating engineer, flower pruning engineer

[kordlessagain]

In essence, programming is an ephemeral art. At its core, our work involves utilizing the electromagnetic field to perform tasks akin to 'thinking.' Without continuous computational processes, our products might as well be inert objects like rocks that simply exist unchanged for millennia.

[groestl]

I am a Software Engineer, I am certified (since in my country, they do so, just as with Mechanical Engineers). Still think it's overloaded though.

[elzbardico]

In my country there used to be some 5 years (realistically 6 for most students) Computer Engineering programs that were basically electronic engineering with selected classes from computer science tackled on top of it. And as Electronic Engineering programs used to have a lot of classes from Civil Engineering tackled upon it, legally said Computer Engineers were legally licensed to build small 3 floor buildings.

[KnobbleMcKnees]

People who have not studied and been given that distinction in engineering going on to call themselves engineers is the reason the term is overloaded.

I've always assumed there's a near 100% overlap between people using the term wrongly to describe any programming activity, and people complaining that it has no meaning or is self-aggrandisement

[pavel_lishin]

Canada?

[groestl]

Austria

[wil421]

How does certification work? I’d imagine it would be in a less abstracted language like C or C++. The problem for me is most of my schooling was based on web technologies and 2 classes of Java.

I wound hate having to studying for a C based test when the area I work in is all web tech. Same could be said of Java. I learned it in school and haven’t used it in 10 years except for it being the backend on my first front end dev project.

[EGreg]

Anyway, the best engineer for ChatGPT prompt can be ChatGPT itself.

People seem to think that “automation will create new jobs”, but in the age or AI, those job opprtunities will be very temporary as the companies making the AI automate that thin layer.

Similarly, people think that humans will control AIs. That’s a bit quaint, a bit like humans controlling a corporation. The thin layer of “control” can be easily swapped out and present an improvement in the market, so that the number of totally autonomous (no human in the loop) workflows will grow.

That can include predictive policing with Palantir (thanks Peter Thiel!), autonomous killbots in war etc. Seeing how reckless companies have been in releasing the current AI in an arms race, I don’t see how they would be restrained in a literal arms race of slaughterbot swarms and panopticon camera meshes.

PS: I remember this exact phase when computers like Deep Blue beat Garry Kasparov. For a while he and others advocated “centaurs” — humans collaborating with computers. But the last decade hardly anyone will claim that a system with humans in the loop can beat a system that’s fully automated: https://en.m.wikipedia.org/wiki/Advanced_chess

[umanwizard]

“Engineer” originally just meant someone who builds engines (in a broad sense of that word). The formal titles requiring certification, etc. are the more recent development.

[ris58h]

https://en.wikipedia.org/wiki/Engineer

> The word engineer (Latin ingeniator) is derived from the Latin words ingeniare ("to contrive, devise") and ingenium ("cleverness").

[PartiallyTyped]

what does “exact” mean?

If we are being pedantic and literal, this is exact in the sense that for identical seeds you get identical results.

[larodi]

you make a valid point, and no - we are not engineers. we are people with printed labels at best, where the label says architect or engineer. but most of these people with these labels don't even have a degree, which is the prerequisite to have this designations. architects also typically need to comply for a local guild.

we, the IT crowd, are long over-due for this formalization of the professions.

[aiisjustanif]

I feel like many places have tried this and didn’t like it or molded it into a Frankenstein.

The idea of an engineer that researches, designs, tests, and measures and a programmer that implements seemed to cost too much (not just monetarily) for the industry that employees them and there isn’t the sufficient need to regulate all sun-groups of the industries that employ software programmers / engineers.