[corobo]

How badly is bad? What sort of output are we talking?

I am asking as I once had a Markov-chain IRC bot* and while it often struggled to string together a sentence, it was quite hilarious sometimes. Absolutely pointless other than the occasional laugh.

Can it form sentences or are those small models completely unusable for anything?

I'm not thinking OpenAI level uses - sort of compare a Postgres cluster to a SQLite file (not literally, conceptually I guess). Can it be used for single tasks in any way?

Could it figure out how to map search terms to URLs for a knowledge base type thing?

Forgive me if these are silly questions. The extent of my knowledge in this field is asking ChatGPT questions and going "that's so cool" when it answers.

* Your phone's predictive text except it finishes the sentence itself based on a word someone in chat used so that it felt on topic.

In my case it also learned how to form sentences from other people talking in chat, in hindsight it's amazing I never had a Tay issue.

https://en.m.wikipedia.org/wiki/Tay_(bot)

[ggerganov]

I was recently playing with the GPT-2 and GPT-J models. Results are often non-sensical for any practical purposes, but I think can be used for making something fun - similar to your IRC bot idea.

If you are interested in running these models yourself without having a beefy GPU, you can try my custom inference implementation. It's in pure C/C++ without any 3rd party dependencies, runs straight on the CPU and builds very easily. I think it is relatively well optimised. For example, on a MacBook M1 Pro I can run GPT-2 XL (1.5B params) at 42ms/token and GPT-J / GPT-JT (6B params) at 125ms/token.

Here are a couple of generated examples using GPT-J:

https://github.com/ggerganov/ggml/tree/master/examples/gpt-j

These are examples using zero-shot prompt where the model auto-completes a text given a starting prompt. You can try to make a conversation bot with a few-shot prompt, but it's not great. Probably the model needs some fine-tuning for that to become feasible.

[corobo]

I'll have to have a look into those, there's an audience of about 12 people that would be thrilled to hear "eggbot" is back with a shiny modern brain on, haha.

Oddly enough any processing delay is good in an "AI" chat bot, within reason, makes it feel more natural rather than getting a response ping instantly. Chat version of uncanny valley or something, haha.

Something it also did in Markov form was pick randomly from the longest words in the sentence it had decided to reply to, build the rest of it from that, then run itself "backwards" from the picked word to a sentence starter word it knew.

Thank you for the reply! Looking forward to some tinkering.

[wokwokwok]

> or are those small models completely unusable for anything?

Sadly, they really offer almost no value.

For the effort, you’re better off with an NLP framework like spacy.

You can play with the small neo gpt models on hugging face, eg. https://huggingface.co/EleutherAI/gpt-neo-125M

…but, the tldr is they’re cute to play with, but practically, the content they can generate is short, inconsistent and full of errors.

[blagie]

.... which is actually not of "almost no value." The value of smaller models is different. For example, I have anonymized data, with fields removed. The smaller models do fine for filling those fields in with plausible values.

The smaller models do okay for zero-shot clustering of data in many cases (e.g. liberal versus conservative text), and if not, with minimal training. For generating statistics or probabilistic information about large numbers of text, they're great.

GPT-3, they're not, but I use them in my day-to-day work quite a bit more than I thought I would. I bought a GPU for one purpose, and I find I spin it up a lot these days.

I /really/ want to be able to use a large-scale language model locally, though. For the types of things I'd like it for, such as helping me draft emails, I don't trust OpenAI with my data.

[visarga]

FLAN T5 shows promising signs, but it doesn't get even to 50% of GPT-3 performance.