Hacker News new | ask | show | jobs
by ComplexSystems 1102 days ago
What is the chat context length? Or, to be precise:

1. How long can a single chat input be? 2. If I am using a chat notebook with several previous inputs, when entering some new input below all of those, is it just appending all previous inputs together and sending the entire thing to the LLM? If so, how long can this be? Will it summarize if it exceeds that length? 3. What model is this using? GPT-3.5 or GPT-4 or something else? 4. What plans do you have next to integrate AI into Wolfram products?
2 comments
toxicFork 1102 days ago
https://github.com/WolframResearch/Chatbook/blob/ba3e783602f... should help answer some of those - I am on a mobile device so hard to write much, but, OpenAI 3.5 at least
link
connordg 1102 days ago
1, 2. In the Chat Notebooks implementation, we aren't currently imposing any limits of our own. Currently we support OpenAI's API, and the only limits imposed are those inherent to the model being used (i.e. GPT-3.5 or GPT-4).

When you have a sequence of alternating 'ChatInput' and 'ChatOutput' cells, those are sent in-order as individual, respectively, "role" -> "user" and "role" -> "assistant" messages to the chat API. It doesn't, for example, concatenate them into a single message.

The one exception is Chatbook's implementation of Personas, which are defined (roughly speaking) as a "system" message that gets prepended as the first message in the chat conversation on each ChatInput evaluation, and which in some cases is concatenated with other ChatInput cells that have been set to "role" -> "system".

Chatbook's don't currently do automatic behind-the-scenes summarization, but we've experimented with some possible approaches. There are UI and and UX considerations around how we present that possibility of automatic summarization that I think require greater consideration.

3. Currently we support the OpenAI Chat API, and all models supported by that API. However, Chatbook currently only implements quick UI access to the GPT-3.5 and GPT-4 models. If you want to use a different OpenAI model in a particular notebook, you can set the relevant setting by evaluating:

    CurrentValue[
        EvaluationNotebook[],
        {TaggingRules, "ChatNotebookSettings", "Model"}
    ] = "gpt-3.5-turbo-0301"
Now all subsequent 'ChatInput' evaluations in that notebook will inherit that "Model" setting (which can itself be overridden on a per-cell basis).

Our focus has been on flexible and easily user-extensible support for Personas (for example, developing the Prompt Repository[1]), but we're also planning to add built-in support for additional foundation models, and greater customizability so that new models are easy to add and use from the Chat Notebooks interface.

[1]: https://resources.wolframcloud.com/PromptRepository/

4. We've got a wide range of exciting projects coming down our R&D pipeline, and things are changing rapidly.

In some sense more fundamental that Chat functionality in notebooks is our work integrating LLM-related functionality more broadly into the Wolfram language: e.g. LLMFunction[..], LLMSynthesize[..], LLMPrompt[..], LLMTool[..] (all part of the Wolfram/LLMFunctions paclet[2]), where we're putting the power of LLMs directly in the hands of Wolfram users.

Making LLMs "just" a feature area added to the many existing powerful capabilities of Wolfram is making it easy for ourselves and our users to maximally leverage the new capabilities LLMs provide. So stay tuned, more is definitely coming :)

[2]: https://paclets.com/Wolfram/LLMFunctions

link