[Reedx]

Carmack posted (yesterday) an interesting thought on models like GPT-3:

"Big AI models like GPT-3 train on massive internet text dumps, but the data is assumed to be independent and identically distributed. Incorporating time information for a decade of data might allow them to start writing tomorrow's reddit or twitter trends."

https://twitter.com/ID_AA_Carmack/status/1278840413919551488

[jcims]

I saw that yesterday and it spawned a thought process for me. It seems the current approach is very effective in developing a language model, but not always effective developing an interaction model. I wonder if it would be possible to build a graph of interactions between users/personas on various social media platforms and forums, and use that to help develop a more effective communicator.

Of course you could add things like date, community (e.g. avforums, /r/blacksmithing, etc) to the graph to help with the contextual cues.

After you have all of that, I didn’t wondered if we could visualize the latent space of human personas and see what that looks like. Does it map to the four quadrants of that political spectrum survey, left and right, old/young, etc.

[ipsum2]

That's what CTRL does:

Given a URL (or other prompt), generate some language.

From [1]:

> With CTRL, we can test which domain best explains a sequence. Note that this procedure is sensitive to subtle nuances in the query prompt. In the example below, "Global warming is a lie" differs from "Global warming is a lie." The latter is a simple declarative sentence as opposed to an open start to a sentence which may continue. Source attribution cannot be considered a measure of veracity, but only a measure of how much each domain token explains a given sequence.

Query Prompt Attributed Sources Global warming is a lie. r/unpopularopinion, r/conspiracy, r/science Global warming is a lie r/eli5, r/science, r/unpopularopinion Global warming is a real phenomenon r/eli5, r/science, r/changemyview Global warming is a real phenomenon. OpenWebText, r/changemyview, r/science

https://blog.einstein.ai/introducing-a-conditional-transform...

[jcims]

Whoa, cool, will check this out. Thanks!

[infogulch]

In that same vein, but along the code generation axis, I wonder if something on the scale of GPT-3 would be capable of generating commits directly. I think generating commits would be much more useful than generating programs whole hog.

[jcims]

That's a great idea. Particularly with bugfixes.