[thaumasiotes]

https://bactra.org/notebooks/nn-attention-and-transformers.h...

Just because something is called an "attention head" doesn't mean the terminology makes sense.

If I'm reading the article right, hungryhobbit is accurately describing a single attention head.

And... what having multiple attention heads means is that you do the "single attention head" thing several times, and average the results. There is no part of hungryhobbit's comment that changes if you make this adjustment.

In particular, he is clearly responding to this clause in the comment above:

>> the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time

Those are the "attention heads" that hungryhobbit is saying don't exist. You'll notice that this definition of "attention head" does not correspond to any part of an LLM; I fail to see the problem with saying that "attention heads", as defined by this clause, don't exist.

[Catloafdev]

There quite literally are internals that are called attention heads in modern LLMs. Saying 'there are no attention heads' is objectively false, which definition you follow is irrelevant.

[thaumasiotes]

Imagine the following conversation:

How was your trip to Los Angeles?

There were no movie stars!

Is the second person's statement "objectively false"?

Or is it more likely that you, Catloafdev, enjoy intentionally misinterpreting simple statements?

You might notice that in the remainder of the sentence you quoted, hungryhobbit takes the trouble to define the attention heads that don't exist.