[sureglymop]

I find it interesting that no one has created a better interface for these LLMs.

Two things that should be available by now in conversations are branching and history editing. Branching is somewhat trivial so let's focus on the history.

Now when I last used an LLM API, the whole context was still managed manually. Meaning I had to send the API the whole conversation history as one long text for every new query.

This means that I could technically change a part of the history directly. Manipulating the history though is not really a trivial problem. The LLM would need to re evaluate starting from that point.

But, the re-evaluation may result in something completely different... If there are branches, perhaps it would also be desirable to let it propagate into the branches.

Next, re-evaluate until where? We can assume a conversation happened until the present moment and the user may have changed their reality/state during the conversation before that point. For example, I may have changed some function based on a suggestion of the LLM. Now, for re-evaluation it would actually be nice if the LLM could also take that state change into consideration.

Here it would be nice if the LLM had the concept of certain logical facts and pieces of information and how they relate to each other but with an interface so that we could see that. If such a piece of information in the conversation is then changed, that would affect the information that is related to it. We could follow a sort of sequence of logical conclusions being made to verify what happened.

Just some thoughts with no conclusion. I think current LLM interfaces could be a lot better.

[jstanley]

I think you may be confused about how LLMs work. Editing the history you send to the API works perfectly fine. You don't need the LLM to specifically "re-evaluate" the history from the point you edited.

In a way it is "re-evaluating" the entire history for every token it generates.

[sureglymop]

What I'm saying is: If, earlier in the history, I change something, that thing may also still be used further down in the history which I specifically don't want.

So yes, I know that I can just do a simple text replacement somewhere in the history but that's not really useful. I want the conversation to be re-evaluated from that point because it might have gone differently with that changed message.

[quantadev]

Yeah, that's where the 'branching' comes in for sure. Ideally a chatbot would be 'tree-based' where you can just go back to a prior point where you wish you had said something different, and just pickup as if you were back at that point in time, while the other parallel branches are ignored.

The way this is done, technically of course, is you build the "Context" by walking back up the tree parent by parent, until you reach the root, building up 'context' as you go along, in reverse order. That will then always be the 'right' context, based on branching.

[haiku2077]

I can do this with my assistants in Zed and Kagi just fine? Did a quick test and it works exactly how you describe.

[sureglymop]

No. When you edit one of your queries in Kagi Assistant, any queries you made in the conversation after that are discarded. Again, I don't just mean simple re-evaluation of a single query.

[haiku2077]

My previous queries aren't discarded. I can flip between them like branches in a conversation tree.

[justinrubek]

Precisely. They are discarded from the new query.

[dmd]

I'm confused by what you mean by this, because most LLM interfaces other than the very very basic ones do already have branching and history editing? E.g. pretty much every third party LLM interface? Are you talking about something more than that?

[sureglymop]

Yes. Here is a scenario:

Let's say you ask an LLM to help you with a programming task. The LLM gives you a solution and you copy that into your code. While getting to the result of this solution, spanned over multiple messages, you give it more context about your code base. You ask it to help with multiple sub problems and it answers with suitable responses containing code snippets.

Now, you change your mind and want to actually write your program in a different language. You could at this point just tell that to the LLM and it would change course from here on out. Everytime it is queried the full history including the original problem is sent to it. The change in programming language is taken into account because it eventually gets to the point where you changed your mind.

But instead what you'd probably want is the solution it gave you initially and all the little solutions to all the sub problems but in the newly chosen programming language! You'd like to simulate or re-evaluate the conversation but with the newly chosen programming language.

I haven't seen that implemented in a usable way but as I said, this is not a trivial feature either.

[mattnewton]

Checkout cursor. It doesn't have the old branch "rebasing" you are describing, but going back into the history reverts changes made after that point by default. (And honestly I am not convinced automatically rebasing that way is better than rerunning the prompts with changes in most cases, it seems too likely to go off the rails to me)

[MangoToupe]

You'd also want to have some mechanism to ensure deterministic generation if you add rebasing, I think? Though I might be behind the times in my understanding of how LLMs work.

[hombre_fatal]

Neither Claude, OpenAI, nor Gemini let you do either, so it's not ubiquitous.

You can't delete messages, you can't branch, and you can't edit the LLM's messages. These are basic, obvious, day-1 features I implemented in my own LLM UI that I built over 3 years ago, yet I'm still waiting for flagship providers to implement them.

It's pathetic how rare good UX is in software.

Can you list some UIs with these features?

[dmd]

Right, but anyone who wants those features can use a 3rd party front end like LibreChat.

[squidbeak]

You can certainly branch with Gemini in AIStudio.

[quantadev]

The first chatbot I wrote back in early 2023, had exactly this (branching and history editing), and I was highly active on the OpenAI discord showing them it, and trying to influence them, to go in this direction. Tree-based editors can be hard to implement, and can be confusing to users, so that's one reason GUIs tend to not attempt it, and just go with linear (although often nowadays "editable") chats.

Related to this I've learned that a lot of the time it's best to just use a "Context File" which is a big file of instructions explaining goals, architecture, and constraints so I can just tell the AI to read it and then say "Do Step 21 Now" for example, if step 21 is laid out in the file. This way I sort of can micromanage the context, rather than trying to reuse old discussions which are usually 90% throw away content, no longer appliable.

[RossBencina]

> Two things that should be available by now in conversations are branching and history editing.

Prapti can do editable history no problem, the entire history is just the markdown file. We experimented with branching using git to automatically switch branches based on how you edit the history.

I still use it frequently. I prefer keeping my chats in local markdown files.

https://github.com/prapti-ai/prapti/

[amelius]

One simple thing I'm missing is a way to point at a word, and let the LLM know that that's where it went wrong (so it can generate an alternative output).

[blharr]

LM studio allows you to edit the context of an existing message like this, and you can even insert whatever writing to continue a thought, then have it continue generating.

[sureglymop]

Yup, that's exactly what I mean!