[Azantys]

Do people really use 100B+ models for writing? I am no writer but to me it seems like writing is one of the easiest tasks with barely any logic or reasoning and as long as its not longer than a handful of pages I expect even 8B models to perform great.

[orbital-decay]

It's pretty clear you've never experimented with it. Creative writing demands everything the model can do and more, and most problems are still unsolved. It's extremely heavy reasoning-wise, more so than coding (check e.g. Engram paper for some insights), but also needs good scattered retrieval, careful subjective training for prose quality, character, and human likeness, a ton of facts baked in, and much much more. Mode collapse is not solved. No LLM does creative writing well but historically only the absolute largest models were able to do write anything complex more or less convincingly and were creative enough.

[BoorishBears]

The largest model I've post-trained in the last 2 years of working on this problem was Kimi 2.5 at 1T parameters.

The simplest way I'd put it is, teaching a model to write coherently (follow rules, patterns, etc.) is easy enough: just use teacher forcing. Teaching a model to write creatively is easy enough: just use RL and punish it for not being creative.

Teaching a model to write well and creatively takes learning two partially opposing objectives that spike the learning requirements in ways that smaller models really struggle with.

[fc417fc802]

> is easy enough: just use RL and punish it for not being creative.

How are you scoring creativity in an unsupervised manner? That seems anything but easy.

[BoorishBears]

Did you try reading the whole comment?

Once creativity is being measured in isolation, getting multiple responses from the model is enough to measure creativity a ton of different ways: wordfreq to identify overused phrases, getting multiple responses for the same prompt and promoting the least similar as preferred for policy optimization, etc.

But that's of limited use for stuff like getting diverse names and such. You want creativity and coherency, and if you just punish the model for using an overused phrase, the first thing it does is strongly learn a new overused phrase (or gibberish).

(Also I don't think you mean unsupervised. You probably mean without humans [since LLMs struggle to judge creativity], but that's not what unsupervised means.)

[fc417fc802]

I did read the the full comment and I did in fact mean exactly what I wrote when I used the term "unsupervised". I think the condescension does nothing but get in the way. Try extending the benefit of the doubt.

> enough to measure creativity a ton of different ways ...

The things you listed seem more like temperature than creativity to me. At this point it occurs to me that this is likely yet another case of highly misleading technical jargon. Suffice to say that truly creative writing requires something entirely different than unusual sentence structure - in fact it doesn't require unusual phrasing at all.

Re unsupervised, it seems the misunderstanding here follows naturally from the previous difference in word meaning. Hopefully you see the difficulty of scoring long form answers for the creativity of the underlying ideas, as well as the impossibility of using a labeled dataset to train on such a criteria.

[BoorishBears]

The first thing I read from you was a sardonic browbeating in response to the exact comment I gave an earnest response.

And even in domains that lean heavily on "usual phrasing", like technical writing, human writing has notably higher perplexity compared to another LLM's outputs: https://www.sciencedirect.com/science/article/abs/pii/S10766...

With such a low baseline for what's unusual, you do need to get the LLM writing unusual phrases relative to its baseline. Otherwise you get things like repeated n-grams and overused constructs ("it's not X it's Y"), and suddenly the output is predictably not perceived as creative by humans even if you were to insert some otherwise creative or novel premise.

Getting the model to break out of that baseline without disrupting the model's ability to follow technical rules, maintain logic and reasoning, etc. is the difficult part.

-

Also you're again saying unsupervised then following up with descriptions that sure sound like you're referring to RL and supervised learning respectively this time. (supervised learning can improve creativity by the way

[fc417fc802]

> Getting the model to break out of that baseline without disrupting the model's ability to follow technical rules, maintain logic and reasoning, etc. is the difficult part.

Sure, that is also somewhat challenging and is necessary to get human sounding prose. However doing so is not sufficient to produce "creative" literature by any reasonable metric.

> you're again saying unsupervised then following up with descriptions that sure sound like you're referring to RL and supervised learning respectively this time.

Are you sure it isn't you who is confused about the usage of those terms? I merely suggested that both preparing and making use of labeled data (ie supervised learning) seemed like it would prove quite difficult here. Quoting from wikipedia (https://en.wikipedia.org/wiki/Unsupervised_learning):

> Unsupervised learning is a framework in machine learning where, in contrast to supervised learning, algorithms learn patterns exclusively from unlabeled data.

[dragonwriter]

> I am no writer but to me it seems like writing is one of the easiest tasks with barely any logic or reasoning

Virtually all logic or reasoning is, in one way or another, part of the support for writing. It’s what separates actual writing from generating nonsense that happens to fit grammar rules.

The specific details depend on the domain, of course, but I can’t see how anyone familiar with the output of writing can think that there is little logic or reasoning in doing it well.

[Azantys]

Of course there is logic but its nowhere near the complexity of math or programming

[numpad0]

let me just say, you're not going to sound smart saying that

[Azantys]

Useless comment

[numpad0]

If you can't make yourself sound adequately smart, it could lead to people ignoring you, and/or acting in spite of your opinions/logic, and/or spending extra effort trying to decipher you. That is not an optimal situation, especially in cases where you would be right[1].

1: I don't think you're right in this instance, but that's beside the point.

[dragonwriter]

That’s trivially false in the case where “math” or “programming” is the domain of the writing, but its also false more generally.

[fc417fc802]

> with barely any logic or reasoning

I take it you enjoy works of literature with inconsistent world building?

Or do you mean professional as opposed to creative writing? Because the bar is even higher for that.

[Azantys]

The wording wasnt very good I ment compared to programming or math the amount of logic and reasoning is small (Research level math hardly compares to writing a book in raw reasoning and logic). And I thing the smaller models have enough "intelligence" to write coherent with logical world building, but only the big models can truly do hard math and programming work

[nozzlegear]

> The wording wasnt very good

Writing isn't so easy after all.