[TrackerFF]

Machine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation.

Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the very start...that is not always the case. Rarely, even.

EDIT: Didn't even notice the notation switch, much appreciated.

[olalonde]

I never understood people who preferred traditional math notation (e.g. single letter symbols, weird characters like ∣q⟩ instead of writing down an explicit type, etc.). I guess the main advantage is terseness? To me, the mathematical expressions would be so much easier to understand if they were just written in pseudo code or an actual programming language like Python.

[OkayPhysicist]

Terseness is a significant advantage in pattern recognition. If you write a long, detailed breakdown of every step, not only are you spending a bunch of time writing, you're also obscuring the natural symmetries of the statement.

It's like saying "I never understood people who prefer to use functions instead of inlining everything". Adding a bunch of visual noise to a statement doesn't improve comprehension.

[smallmancontrov]

Yep. Does anyone think Cauchy's Integral Formula looks worse like this:

https://www.mathjax.org/

than like this?

    z = a + R * sp.exp(sp.I * t)
    
    f_at_a = 1 / (2 * sp.pi * sp.I) * sp.Integral(
        f(z) / (z - a) * sp.diff(z, t),
        (t, 0, 2 * sp.pi),
    )

The heiroglyphs let me offload some of the work to my vision system. It feels like there's some kind of hierarchical attention organization mechanism that works better in visual space. If I'm not pushing to "maximum math," both representations clear the hurdle of "good enough" (CIF does) but if I'm trying to chew through something genuinely gnarly I want all the help I can get.

[aeternum]

Math notation ultimately is pseudo code just with mostly single letter variables and many operators that are encoded purely by position thus not even requiring a symbol.

Remember that the oft-used e^x is actually an infinite series, even writing it out in summation form would be quite verbose given its frequency in many equations.

[SkyBelow]

I feel the magic in math is that the notation is to a more universal sort of language, and the e^x captures that. e^x means that infinite series, but it means a few other things at the same time. That those different things are sometimes related, that e^2 means e*e, a simple enough operation most can understand, but that it also means that infinite series with 2 in place of x, is part of the complexity of math that makes it quite different than a classical programming language. With most programming languages, someone assigned those equivalent meaning, and while it might be important to understand why, it shows you the mind of the who designed the language. But, in math (and in true Computer Science), when you see an equivalence of this sort, whose mind are you now glimpsing into?

But, for someone just starting out (or even an expert who just wandered into a new area), this becomes a barrier to understanding. Sometimes we need the mathematical summary, what tells us the immediately answer we are interested in, perhaps even if it loses the deeper nuances that are beneficial for the experts more fluent in the language.

(And yes, all math notation is human created, so it makes my point quite a bit messier in practice.)

[crubier]

exp(x)

[kurthr]

Longer to write and generally a numerical rather than symbolic representation.

[crubier]

Longer to write: Don't care, that's kinda the point. I'd rather read a descriptive name in english than a greek letter with a tilda and a dot subscript.

Generally a numerical rather than symbolic representation: Meh https://docs.sympy.org/latest/modules/functions/elementary.h...

[aeternum]

At least one benefit is you can now leverage LLMs to basically translate entire math textbooks into your preferred notation.

I would be curious whether you find it easier. I kinda think it'll be too many parenthesis.

[crubier]

This 1,000%

Trying to read any math paper is basically like trying to read CodeGolf.

[dboreham]

Well, humanity struggled for 2000 years trying to do mathematics without notation, so its benefit is not to be sniffed at. But really it's just an APL-vs-Fortran type debate. They're not fundamentally different. Remember also that Ramanujan had to re-use paper it was so costly/hard to find.

[glaslong]

You really just get tired of writing/reading "AbstractJavaSerializerBeanFactoryFactoryAbstactMutatorFactoryAccessEnterpriseBeanFactory()" over and over again.

So you and all your peers agree to call that procedure "ẽ"

[TeMPOraL]

Yes. Specifically, that thing on top is the "AbstractJAvaSerializerFactoryFactory over AbstractMutatorFactoryAccess over Factory; "e" is just EnterpriseBean.

[olalonde]

Mathematicians probably spend less time writing than programmers.

[htrp]

math notation doesn't bias towards English language understanding like pseudocode

[scraptor]

Diversity win: this notation can't be easily understood by anyone!

[dfee]

but it biases towards the latin alphabet. so your point is diminished.

[tyre]

and Greek!

[whatsakandr]

I used to think this, then I realized that the amount of time you spend with equations is so much more than code, and the terseness makes them much easier to read once you know what the symbols are.

Also, letters avoid having to name them, naming being a hard problem and all.

[Asraelite]

> At least in this article the author explicitly explains the notation at the very start

They explain one particular aspect of the notation but never define the variables used. What is k? q? S?

It's obvious if you've studied machine learning before, and for some of them you can make an educated guess, but it makes the article mostly opaque if you don't already have some domain-specific background knowledge.

[loubbrad]

The author does define these terms implicitly at the beginning of §1, where they define standard MHA in this notation. From the formula, (q_t) is the query generated from the hidden state at the (t)-th position (e.g. the token at position (t)), (k_i) is the key for the token at the (i)-th position, (o_t) is the (vector) attention output for the (t)-th position, etc. (S) is then defined later as the sum of the outer products of (k_i) and (v_i) over all positions up to (t). However, I do agree with you that it would not hurt to make this more explicit.