[Aperocky]

In tech and don't share your worries. If anything, since release it has proven that 1. it does not have ability to reason logically creatively. 2. it has strong ability to manipulate language to human readable form.

If anything, I would say regular office workers are the most threatened, particularly if the jobs revolves around digesting and passing information.

[brap]

Sure. But will that be the case 5-10 years from now?

As a programmer, I gotta say if you're not at all concerned, then either you haven't been paying attention or you're in denial. Sooner or later it's coming.

[lolinder]

Or you have been paying attention, understand the technology better than those who buy into the hype, and aren't worried because you know that incremental improvements to this tech cannot be a serious threat to your job security because the paradigm isn't capable of replacing you.

When my job is at risk it will be because we have AGI, not a better language model, and at that point everyone is at risk. Worrying about job security in the face of AGI is like worrying about what you'll do after your city gets nuked: it's unlikely to happen and there's nothing you can do if it does.

[versteegen]

Why think only about "incremental" improvement? People aren't just making slight tweaks, new papers are published at a remarkable rate where people try significantly different architectures, training methods, etc, and that steady progress leads to ever more impressive results. How can you assume this direction of research will lead nowhere?

OK, ignore everyone who doesn't understand the technology. Of those of who do, I'm utterly amazed how pessimistic many are that this "isn't capable" of leading to AGI. Probably not Transformers specially, but LLMs show that intelligence is remarkably easy. You don't even need to put anything in the neural architecture designed to perform reasoning tasks, but they can be learnt regardless, because Transformers are flexible enough to learn to emulate computation (Turing machines) with bounded space and time, going beyond the famous result that 2-layer MLPs are universal function approximators.

[LegionMammal978]

> Probably not Transformers specially, but LLMs show that intelligence is remarkably easy.

LLMs show that language is remarkably easy. Ever since GPT-3 was released, I've been convinced that language comprehension isn't nearly as big a component of general intelligence as people are making it out to be. This makes some intuitive sense: I recall a writer for a tabloid expressing that they simply turn off their brain and start spinning up paragraphs.

But so far, I haven't seen any of these models perform logical reasoning, beyond basic memorization and reasoning by analogy. They can tell you all day what their "reasoning process" is, but the actual content of any step is simply something that looks like it would fit in that step. Where do you derive this confidence that advanced logical reasoning is a natural capability of transformer models? (Being capable of emulating finite Turing machines is hardly impressive: any sufficiently large finite circuit can do that.)

[hackinthebochs]

>Ever since GPT-3 was released, I've been convinced that language comprehension isn't nearly as big a component of general intelligence as people are making it out to be

"X is the key to intelligence"

computers do X

"Well actually, X isn't that hard..."

rinse and repeat 100x

At some point you have to stop and reflect on whether your concept of intelligence is faulty. All the milestones that came and went (arithmetic, simulations, chess, image recognition, language, etc) are all facets of intelligence. It's not that we're discovering intelligence isn't this or that computational feat, but that intelligence is just made up of many computational feats. Eventually we will have them all covered, much sooner than the naysayers think.

[LegionMammal978]

> All the milestones that came and went (arithmetic, simulations, chess, image recognition, language, etc) are all facets of intelligence.

Why should I have to care about those weird milestones that some other randos came up with once upon a time? I've never espoused any of those myself, so how is this supposed to prove anything about my thought process?

> It's not that we're discovering intelligence isn't this or that computational feat, but that intelligence is just made up of many computational feats. Eventually we will have them all covered, much sooner than the naysayers think.

Well, it certainly appears to me like there's a big qualitative difference between the capabilities you mentioned (arithmetic and simulations are just applications of predefined algorithms; chess, image recognition, and language are memorization, association, and analogy on a massive scale) and the kind of ad-hoc multi-step logical reasoning that I'd expect from any AGI. You can argue that the difference is purely illusory, but I'll have a very hard time believing that until I see it with my own eyes.

[terminal_d]

I think that's more representative of tabloid writers than anything, haha. Understanding text is difficult, and scales with g. GPT-3 can make us believe that it can comprehend text that falls in the median of internet content, and I guess there would have to be some edge cases addressed by the devs, but it can't convince humans that is understands more difficult content, or even content that isn't in its db.

[versteegen]

I totally agree with your comments on language. I was stretching it to cover "intelligence" too, what I should have said is "many components of intelligence". It really isn't one thing. But I think analogical reasoning is one of the most important, maybe the most important component! I'm not alone. [1]

> Where do you derive this confidence that advanced logical reasoning is a natural capability of transformer models?

("Advanced logical reasoning" is asking a lot, more than I wanted to claim.) I was going off papers like [2] which showed very high accuracy for multi-hop reasoning by fine tuning RoBERTa-large on a synthetic dataset, including for more hops than seen in training (although experiments "suggests that our results are not specific to RoBERTa or transformers, although transformers learn the tasks more easily"). While [3] found "that current transformers, given sufficient training data, are surprisingly robust at solving the resulting NLSat problems of substantially increased difficulty" but "transformer models’ limited scale-invariance suggests they are far from learning robust deductive reasoning algorithms". I think that low scalability is to be expected, transformers don't have a working memory on which they can iterate learnt algorithmic steps, only a fixed number of steps can be learnt (as I was saying).

Unfortunately, looking for other papers, I found [4] which pours a lot of cold water on [2], saying "a deeper analysis reveals that they appear to overfit to superficial patterns in the data rather than acquiring the logical principles governing the reasoning in these fragments". I suppose you were more correct. I still think there's more than just memorisation happening here, and it isn't necessarily dissimilar to intuitive (rapid) 'reasoning' in humans, but as with everything in LLMs, everything is muddied because capability seems to be a continuum.

[1] Hofstadter, 2001, Analogy as the core of cognition, http://worrydream.com/refs/Hofstadter%20-%20Analogy%20as%20t...

[2] AI2, 2020, RuleTaker: Transformers as Soft Reasoners over Language, https://allenai.org/data/ruletaker

[3] Richardson &al. 2021, Pushing the Limits of Rule Reasoning in Transformers through Natural Language Satisfiability https://arxiv.org/abs/2112.09054

[4] Schlegel &al. 2022, Can Transformers Reason in Fragments of Natural Language? https://arxiv.org/abs/2211.05417

[lolinder]

I never said incremental improvements to LLMs won't lead anywhere, I said they won't replace me. A sibling has already commented on why that would be, and I agree with them.

I just wanted to chime in and remind about the other part of my argument: my job is not threatened until we have AGI, and AGI would be so earth-shattering to the entire premise of our economy that there's literally no point in worrying about it as an individual. We can and should talk about society-level changes like UBI, but having individual anxiety about your own personal job is a strange response to the end of the entire global economic system.

[greenyoda]

> You don't even need to put anything in the neural architecture designed to perform reasoning tasks, but they can be learnt...

That sounds interesting. Can you provide a reference to this research?

[versteegen]

See my reply to sibling: https://news.ycombinator.com/item?id=34672865

A more interesting example of transformers learning a process may be [1].

There's a large literature on applying language models to reasoning tasks, but not many on what's actually going on inside them. But see for example [2]. Also https://transformer-circuits.pub/ has a body of work on it, but still at a very early stage (see in particular "In-context Learning and Induction Heads").

[1] Extraction of organic chemistry grammar from unsupervised learning of chemical reactions https://www.science.org/doi/10.1126/sciadv.abe4166

[2] Analyzing the Structure of Attention in a Transformer Language Model https://arxiv.org/abs/1906.04284

[thom]

Yep, I subscribe to this viewpoint. I am not as smart as the creators of ChatGPT or whatever is to follow, so maybe we’ll get lucky that AGI is pretty smart but can’t improve itself. But I think in the general case, if we create AIs that can replace programmers, economic concerns aren’t going to matter.

[qup]

Yes. And we'll be in the true exponential times, because humans will be building wonderfully complex software in an instant, so we'll build huge, robust, open ecosystems in just a few moments.

The programming job market going away will be the last thing on my mind at that time, I think it would be one of the biggest shifts in human history.

[CuriousSkeptic]

While I share your take on LLMs. I do add on the worry of may take a couple of years for the people who pay software engineers to figure out that they aren’t replaceable yet.

[Aperocky]

As a programmer that had a year of data scientist under my belt, and some level of understanding of the current machine learning systems, I'm not worried.

The newer research papers coming out are where I would focus if I'm really worried. ChatGPT is really the industrial grade implementation of ideas that aren't exactly new. And the idea itself (LLM) does not contain ability to generate novel logic or solve unlearned problems. In fact, I would go further and say that it does not see your prompt input as a logical problem, but rather a collection of words, and its output involves a transformation of words with its extensively training set. Which would contain logic that was in the training material but itself did not add anything to it, with no guarantee of correctness, only training weights.

There are no guarantee that it didn't also train on the question part of stackoverflow... and you only get answers as good as its training material.

Where I work we attempt to automate away repeatable problems with deterministic frameworks, which are imo far better. I would agree that ChatGPT will probably improve my writing if I were to use it on email and documents, but only in language, not in ideas.

[terminal_d]

> The newer research papers coming out are where I would focus if I'm really worried. Great reply, could you expand on this?

[Aperocky]

I wouldn't know until they come out, I don't think there are anything that smell like artificial general intelligence right now, pattern matching is much more dominant today.

[throwawaycopter]

> Sure. But will that be the case 5-10 years from now?

Not with language models. A language model can parse natural language, and with enough training data, give out what it thinks the answer is based on the data it was trained with. It is not General AI.

It cannot reason a solution for a problem that had an unknown answer. It won't be able to reflect logically on a context to foresee problems within this context. It cannot have a meaningful conversation. It won't be able to understand that one of the things it "knows" was incomplete, untrue, or just plain wrong, and fix itself.

It's a powerful tool, a game-changing tool. Perhaps as game-changing as the advent of computers, internet, or wireless communication. But it still won't replace humans.

General AI for now is science fiction. Perhaps this is unfortunate. I wouldn't mind an AI that can replace humans, even if I too am made obsolete with it.

[oceanplexian]

> General AI for now is science fiction. Perhaps this is unfortunate. I wouldn't mind an AI that can replace humans, even if I too am made obsolete with it.

Maybe I’m optimistic, but I feel like we need AGI to reach the next level of development as a civilization. If software engineering jobs are the price to pay so be it. World hunger, medical science, energy, space travel, if we can get all of these to take a ride on something resembling Moore’s Law we are in for a one hell of a fantastic future in our lifetimes.

[throwawaycopter]

I actually think AGI will bring about the collapse of civilization, and perhaps the end of humanity. I'm okay with it.

I also think it will solve things such as energy and space travel. World hunger, medical science (among other human problems) will become meaningless.

"Rejoice glory is ours / Our young men have not died in vain / Their graves need no flowers / The tapes have recorded their names"

[roncesvalles]

When ChatGPT was first released, I read a memorable comment on HN (paraphrasing from memory): It looks like AI is poised to take over all the things that I enjoyed doing as a human such as art, music, storytelling, teaching, programming.

There only needs to be one intelligent species in any ecosystem. If AI becomes that species, humans will be relegated to the role of horses - only good for menial physical labor. That is, of course, only until the AI invents cars.

[nequo]

My guess is that this will become a problem roughly around the time when automated theorem proving becomes viable.

Until then, language models just regurgitate sentences. Anything that relies on the sentences being logically coherent cannot rely on these models. This includes engineering, finance, and journalism (ideally, although I am aware of CNET's experiment).

After then, we salaried humans might be in a pickle.

Edit: sentence.

[joe_the_user]

Another to consider is that increased programmer productivity for simple code may result in just more code.

If any given music synthesizer company could produce it's own DAW (digital audio workstation) by employing a single programmer or a contractor, the demand for programmers could increase.

Just speculating, it's hard to know.

[roncesvalles]

>Sure. But will that be the case 5-10 years from now?

A big part of the failings of cryptocurrency and self-driving cars was that predictions of mass disruption were contingent upon the tech improving linearly from the current state. That didn't happen.

[ResearchCode]

Shouldn't non-programming office workers be more concerned is what I'm thinking.

[minusf]

when a program will be smart enough to write at least as usable code as me, i'll happily hang up my keyboard and take up gardening.

until that happens what's the point of worrying? the year of the linux desktop joke got retired anyway so let's use self driving cars and "ai programmers" :)

[joe_the_user]

Education faces the biggest problems of all. And it's self-inflicted. A large portion of what's required of students is listening to, regurgitating and producing half-coherent bullshit. And so teachers will have a hard verifying homework now.

What should happen is teachers and students are required to up their skills to the point that student are required to produce things better than AI. But the likely result will be lots of futile surveillance.

[pvaldes]

I see several possible solutions. One could be the teacher typing the same question in the chatbot and finding some subtle but wrong results and then looking for the same errors in the students texts.

[juve1996]

It was impossible to verify homework before ChatGPT. People just cheated off of the smart kid and copied the work. Fundamentally the same problem as ChatGPT.

[echelon]

I'm using it to cut my engineering time down. If you're not simultaneously elated and threatened, you're sleeping.

The AI wave is going to be bigger than the internet in terms of societal impact.

We're watching humanity's swan song in real time.

[sublinear]

I don't understand this perspective at all. The hardest parts of software are nailing down requirements, scheduling work, and fitting square pegs into round holes for legacy systems at a conceptual level.

No offense, but if writing the actual code is the part slowing you down there's something very wrong.

[echelon]

I'll bet you that everything you just outlined as being "hard" will see startups launched and successful within 5-10 years.

> The hardest parts of software are nailing down requirements, scheduling work, and fitting square pegs into round holes for legacy systems at a conceptual level.

Then why do we hire PMs and have engineering ICs? The only reason any of this is hard is because 1) there are a lot of stakeholders and moving pieces and 2) humans are subject to context switching productivity losses. Machines will absolutely be able to inject themselves into these business processes and chip away at our inefficiencies.

I bet you you're wrong. I quit my $400k+/yr TC job to focus on AI because I believe in this so strongly.

Let's check back in five years.

[juve1996]

Don't bother betting other people. Your entire life savings should be invested in AI right now with your level of confidence.

[echelon]

That's more or less what I'm doing.

I'm hiring, btw.

[UncleEntity]

How do you expect to make a difference without having a mega-server farm to train the Next Big Thing?

From what I’ve seen so far the ‘impactful’ models take tons of cash to initially train, tons of cash to continue to train and tons of cash to provide “free” access to the public to get the hypetrain chugging along.

Not criticizing but this is what I think people should be worried about, the “future” locked behind a handful of super rich corporate firewalls. Not even mentioning they have real-time data feeds on virtually everything they can pump through their AI for whatever purposes they can dream up. And I’m not even paranoid…

[sublinear]

> Machines will absolutely be able to inject themselves into these business processes and chip away at our inefficiencies.

Actually, this I agree with. Chat bots in their current form would help bridge technical knowledge gaps between stakeholders.

Whether it would significantly lean up the less technical staff seems unlikely. Getting rid of engineering is even less likely.

> there are a lot of stakeholders and moving pieces

Yes. There are so many extending all the way out to internet discussions like this one and beyond to consumers, investors, etc... I think all this discussion is necessary to actually making anything of value. Will consulting chat bots instead of hacker news be the future, or is that where we admit there's something pathological going on?

> humans are subject to context switching productivity losses

I don't think letting people figure out what they want is a "productivity loss".

[mdp2021]

> The AI wave is going to be bigger

This (in context) is not the «AI wave». Artificial Intelligence is that which automates solution finding normally tasked to a professional. It has to be qualitatively adequate and competitive (which you could take as the real meaning of the "Turing test" - otherwise, "being scammed by con artists" was already a known factor and perspective at the time of the formulation). If any output fitted the definition, a random number generator (or a constant output) would fit as AI.

Go back to Popper (this name just for example): there has to be a metric for "success".

[mdp2021]

> swan song

Swan song that started as the singers started regarding IQs of 0 as acceptable actors.

[echelon]

Most of the actions you and I do are "0 IQ". There's a little bit of intelligence in the in between.

[mdp2021]

When I wrote about «act[or|ion]s», I of course meant the relevant ones,

and in general, actions are based - prescriptively - on concretions of layers of developed intelligence (wisdom), intelligence which is constantly applied and exercised (just with different occasional effort, for resource management). If somebody went sparingly, their own practice and our damage - if they are active.

So, again: a "sub-bohemian" idea that "anything goes" is the swan song. Wait till you actually needed intelligent action, to prevent critical losses.

--

I will rephrase coincisely: you are supposed to exercise intelligence at any action. If you do not do it, your vice.