[eliomattia]

> This is true for almost everything politicians promote as a "solution"

:-) In their defense, this time they may not even be aware.

> So for most people, they actually see g which is something like g(f(H)))

It is interesting to consider the entailment structure of g(f(H)). f(H), or A, is entailed from H by f. g(f(H)), let us call it A', is entailed from f(H) by g. Using |- as the symbol for entailment, or "entails":

  f |- f(H)
  g |- g(f(H))

All included plugins, embeddings, other API endpoints in g will definitely affect the results, turning answer A, or f(H), into a modified answer A', or g(f(H)). However, what entails f? Nothing in the context of applying g.

To a first approximation, design D and training T are the ones that entail f. T includes all human feedback (training pairs, not direct edits on the entailment in f), and also previous chat histories H for subsequent releases f', however this inclusion is behind decision hierarchies, and the effects of the inclusion on f are largely outside of users' or researchers' precise or accurate control. Training pairs can be added to fine tune, e.g., to filter out violent content, but the results can only be checked downstream of the new f', and fine tuning will retain previous entailments hence acting similarly to a g().

  (D,T) |- f

Let us consider programming, where software S entails output o. What [subject] entails software [direct object]? Programmers P, directly and reliably. Not only can programs be affected precisely and accurately. If we want to modify a program, we do not wrap it inside another program g, hoping that the functional composition will yield the desired result, which will invariably contain entailment from the one we are stuck with. Instead, we edit it.

  S |- o
  P |- S
  S |- P

Programming is interactive, between programmers and software, whereas with LLMs users are downstream when it comes to wanting to affect directly the entailment patterns in f. The human-like chat UX of LLMs leads us to think that chatting with them, giving them human feedback, prompts, or guidelines, or adding plugins and context will naturally flow into us being able to affect f, and that since humans can be manipulated, then so can LLMs. The core of the matter is that f, which entails issues, cannot be affected easily, precisely, or accurately, despite any g that we may want to wrap around it.

[verdverm]

RLHF entails f for ChatGPT because you cannot get ChatGPT without RLHF. This is before the feedback from users, and was part of the initial training process. Without RLFH, you only have GPT-3 which is very different from its successors that people are actually enthralled with and worried about.

RLHF is Reinforcement Learning from Human Feedback

Thus you cannot get ChatGPT without humans in the loop, making it quite sensitive and irreproducible.

[eliomattia]

By human feedback in T, I meant indeed RLHF. By chat histories H in T, I meant a later selection of user feedback. While plug-ins and added context can be visualized as g(f(H)), fine tuning could be thought of as g(f)(H). There are humans in the loop, however which entailment loop? Not the one affecting the hardware-like f. Unless we retrain de novo.

[verdverm]

No one wants to read your imprecise pseudo math in plain text, write in clear sentences

Your premise is that these are not editable models, but you restrict that to a situation that is not true in practice. These models are constantly changing based on human feedback, updated information, config changes, and weight manipulation.

If you follow the conversation around LLMs, one of the top issues people have is inconsistent responses and the models changing under their prompts and breaking them.

[verdverm]

https://arxiv.org/abs/2305.12907

> We find that meta-in-context learning adaptively modifies priors over latent variables, ultimately leading to priors that closely resemble the true statistics of the environment. Furthermore, our analysis reveals that meta-in-context learning can not only be used to change prior expectations but is also capable of reshaping an LLM’s learning strategies

This was done using OpenAI's public facing APIs

Even if it is not a permanent "edit" it still influences the model at the lowest levels

[eliomattia]

The article can be summarized as: context richer prompt history yields answers that are better aligned with expectations.

> the practical constraint of a finite context window coupled with meta-in-context learning’s rapid prompt length increase

The model is not influenced by the added context. The answers are. The abstractions, or, what the LLM "knows", are in the model itself, whereas the answers are just byproducts.

There is an ongoing related discussion on LLMs lacking world model. One could say LLMs do implement a computable model, albeit a dead, static, non-editable one.