[plewd]

like OP originally said, the LLM doesn't have access to the actual process of the author, only the completed/refined output.

Not sure why you need a concrete example to "test", but just think about the fact that the LLM has no idea how a writer brainstorms, re-iterates on their work, or even comes up with the ideas in the first place.

[empath75]

> has no idea how a writer brainstorms

This isn't true in general, and not even true in many specific cases, because a great deal of writers have described the process of writing in detail and all of that is in their training data. Claude and chatgpt very much know how novels are written, and you can go into claude code and tell it you want to write a novel and it'll walk you through quite a lot of it -- worldbuilding, characters, plotting, timelines, etc.

It's very true that LLMs are not good at "ideas" to begin with, though.

[GCA10]

Professional writer here. On our longer work, we go through multiple iterations, with lots of teardowns and recalibrations based on feedback from early, private readers, professional editors, pop culture -- and who knows. You won't find very clear explanations of how this happens, even in writers' attempts to explain their craft. We don't systematize it, and unless we keep detailed in-process logs (doubtful), we can't even reconstruct it.

It's certainly possible to mimic many aspects of a notable writer's published style. ("Bad Hemingway" contests have been a jokey delight for decades.) But on the sliding scale of ingenious-to-obnoxious uses for AI, this Grammarly/Superhuman idea feels uniquely misguided.

[AlotOfReading]

The distinction being made is the difference between intellectual knowledge and experience, not originality.

Imagine a interviewing a particularly diligent new grad. They've memorized every textbook and best practices book they can find. Will that alone make them a senior+ developer, or do they need a few years learning all the ways reality is more complicated than the curriculum?

LLMs aren't even to that level yet.

[re-thc]

> because a great deal of writers have described the process of writing in detail

And that's often inaccurate - just as much as asking startup founders how they came to be.

Part of it is forgot, part of it is don't know how to describe it and part of it is don't want to tell you so.

[JCharante]

why not? datasets are not only finished works, there's datasets that go into the process they're just available in smaller quantities

[elmomle]

Let's take the work of Raymond Carver as just one example. He would type drafts which would go through repeated iteration with a massive amount of hand-written markup, revision and excision by his editor.

To really recreate his writing style, you would need the notes he started with for himself, the drafts that never even made it to his editor, the drafts that did make to the editor, all the edits made, and the final product, all properly sequenced and encoded as data.

In theory, one could munge this data and train an LLM and it would probably get significantly better at writing terse prose where there are actually coherent, deep things going on in the underlying story (more generally, this is complicated by the fact that many authors intentionally destroy notes so their work can stand on its own--and this gives them another reason to do so). But until that's done, you're going to get LLMs replicating style without the deep cohesion that makes such writing rewarding to read.

[mold_aid]

A good point. "Famous author" is a marketing term for Grammarly here; it's easy to conceive of an "author" as being an individual that we associate with a finite set of published works, all of which contain data.

But authors have not done this work alone. Grammarly is not going to sell "get advice from the editorial team at Vintage" or "Grammarly requires your wife to type the thing out first, though"

I'll also note that no human would probably want advice from the living versions of the author themselves.

[simianwords]

Can a human replicate style without understanding process? Yes we can. We do it all the time with Shakespeare. Why not LLMs?

I can do it at the moment with Shakespeare an LLMs.

[Planktonne]

Mimicking the style of Shakespeare does not produce anything close to work with the quality of Shakespeare.

[simianwords]

i don't buy this logic. if i have studied an author greatly i will be able to recognise patterns and be able to write like them.

ex: i read a lot of shakespeare, understand patterns, understand where he came from, his biography and i will be able to write like him. why is it different for an LLM?

i again don't get what the point is?

[wongarsu]

You will produce output that emulates the patters of Shakespeare's works, but you won't arrive at them by the same process Shakespeare did. You are subject to similar limitations as the llm in this case, just to a lesser degree (you share some 'human experience' with the author, and might be able to reason about their though process from biographies and such)

As another example, I can write a story about hobbits and elves in a LotR world with a style that approximates Tolkien. But it won't be colored by my first-hand WW1 experiences, and won't be written with the intention of creating a world that gives my conlangs cultural context, or the intention of making a bedtime story for my kids. I will never be able to write what Tolkien would have written because I'm not Tolkien, and do not see the world as Tolkien saw it. I don't even like designing languages

[simianwords]

that's fair and you have highlighted a good limitation. but we do this all the time - we try to understand the author, learn from them and mimic them and we succeed to good extent.

that's why we have really good fake van gogh's for which a person can't tell the difference.

of course you can't do the same as the original person but you get close enough many times and as humans we do this frequently.

in the context of this post i think it is for sure possible to mimic a dead author and give steps to achieve writing that would sound like them using an LLM - just like a human.

[Planktonne]

You're still confusing "has a result that looks the same" and "uses the same process"; these are different things.

[simianwords]

Why do you say it has a different process? When I ask it to do integrals it uses the same process as me

[Planktonne]

Not everything works like integrals. Some things don't have a standard process that everyone follows the same way.

Editing is one of these things. There can be lots of different processes, informed by lots of different things, and getting similar output is no guarantee of a similar process.

[arkadiytehgraet]

Even if the visualization of the integration process via steps typed out in the chat interface is the same as what you would have done on paper, the way the steps were obtained is likely very different for you and LLM. You recognized the integral's type and applied corresponding technique to solve it. LLM found the most likely continuation of tokens after your input among all the data it has been fed, and those tokens happen to be the typography for the integral steps. It is very unlikely are you doing the same, i.e. calculating probabilities of all the words you know and then choosing the one with the highest probability of being correct.

[mold_aid]

You are not able to write like Shakespeare. Shakespeare isn't really even a great example of an "author" per se. Like anybody else you could get away with: "well I read a lot of Bukowski and can do a passable imitation" or "I'm a Steinbeck scholar and here's a description of his style." But not Shakespeare.

I get that you're into AI products and ok, fine. But no you have not "studied [Shakespeare] greatly" nor are you "able to write like [Shakespeare]." That's the one historical entity that you should not have chosen for this conversation.

This bot is likely just regurgitating bits from the non-fiction writing of authors like an animatronic robot in the Hall of Presidents. Literally nobody would know if the LLM was doing even a passable job of Truman Capote-ing its way through their half-written attempt at NaNoWriMo

[mold_aid]

>Literally nobody would know if the LLM was doing even a passable job of Truman >Capote-ing its way through their half-written attempt at NaNoWriMo

As I look back on my day, I find myself quite pleased with this line.

[tovej]

You can understand his biography and analyses about how shakespeare might have written. You can apply this knowledge to modify your writing process.

The LLM does not model text at this meta-level. It can only use those texts as examples, it cannot apply what is written there to it's generation process.

[simianwords]

no it does and what you said is easily falsifiable.

can you provide a _single_ example where LLM might fail? lets test this now.

[tovej]

Yes, what I said should be falsifiable. The burden is on you to give me an example, but I can give you an idea.

You need to show me an LLM applying writing techniques do not have examples in its corpus.

You would have to use some relatively unknown author, I can suggest Iida Turpeinen. There will be interviews of her describing her writing technique, but no examples that aren't from Elolliset (Beasts of the sea).

Find an interview where Turpeinen describes her method for writing Beasts of the Sea, e.g.: https://suffolkcommunitylibraries.co.uk/meet-the-author-iida...

Now ask it to produce a short story about a topic unrelated to Beasts of the Sea, let's say a book about the moonlanding.

A human doing this exercise will produce a text with the same feel as Beasts of the Sea, but an LLM-produced text will have nothing in common with it.

[simianwords]

>You need to show me an LLM applying writing techniques do not have examples in its corpus.

why are you bringing this constraint?

[tovej]

Because the entire point is the LLM cannot understand text about text.

If someone has already done the work of giving an example of how to produce text according to a process, we have no way of knowing if the LLM has followed the process or copied the existing example.

And my point of course is that copying examples is the only way that LLMs can produce text. If you use an author who has been so analyzed to death that there are hundreds of examples of how to write like them, say, Hemingway, then that would not prove anything, because the LLM will just copy some existing "exercise in writing like Hemingway".

[inaros]

>> i again don't get what the point is?

The point is that you dont become Jimi Hendrix or Eric Clapton even if you spend 20 years playing on a cover band. You can play the style, sound like but you wont create their next album.

Not being Jimi Hendrix or Eric Clapton is the context you are missing. LLMs are Cover Bands...

[TimorousBestie]

This is the plot of a short story of Borges’ called “Pierre Menard, the Author of Don Quixote.”

[Planktonne]

There's a relatively common pattern of "new tech idea => Borges already explained why that approach is conceptually flawed".