[HighFreqAsuka]

I've seen quite a few of these books attempting to explain deep learning from a mathematical perspective and it always surprises me. Deep learning is clearly an empirical science for the time being, and very little theoretical work that has been so impactful that I would think to include it in a book. Of the such books I've seen, this one seems like actively the worst one. A significant amount of space is dedicated to proving lemmas that provide no additional understanding and are only loosely related to deep learning. And a significant chunk of the code I see is just the plotting code, which I don't even understand why you'd include. I'm confident that very few people will ever read significant chunks of this.

I think the best textbooks are still Deep Learning by Goodfellow etal and the more modern Understanding Deep Learning (https://udlbook.github.io/udlbook/).

[thehappyfellow]

This book is not aimed at practitioners but I don’t think that means it deserves to be called „actively the worst one”.

Even though the frontier of deep learning is very much empirical, there’s interesting work trying to understand why the techniques work, not only which ones do.

I’m sorry but saying proofs are not a good method for gaining understanding is ridiculous. Of course it’s not great for everyone but a book titled „Mathematical Introduction to x” is obviously for people with some mathematical training. For that kind of audience lemmas and their proof are natural way of building understanding.

[HighFreqAsuka]

Just read the section on ResNets (Section 1.5) and tell me if you think that's the best way to explain ResNets to literally anyone. Tell me if, from that description, you take away that the reason skip connections improve performance is that they improve gradient flow in very deep networks.

[p1esk]

the reason skip connections improve performance is that they improve gradient flow in very deep networks.

Can you prove this statement?

[c7b]

Neither do the authors in the book, and I'd argue that after (only) reading the book, the reader wouldn't be equipped to attempt this either (see my other post in this thread), so I think the parent poster has a point.

[HighFreqAsuka]

Yes, I have a very good point in fact. But the above comment purposely chooses not to argue with it, because it's easier to ignore it entirely and argue something else.

[p1esk]

The problem is you presented something as a fact while it’s just a guess. Some people guess it’s an improved gradient flow, others guess it’s a smoother loss surface, someone else guesses it’s a shortcut for early layer information to reach later layers, etc. We don’t actually know why resnets work so well.

[HighFreqAsuka]

Empirically yes, I can consider a very deep fully-connected network, measure the gradients in each layer with and without skip connections, and compare. I can do this across multiple seeds and run a statistical test on the deltas.

[psyklic]

Empirical studies are only useful until the system is mathematically understood. For example, I can construct transformer circuits where the skip connection (provably) purely adds noise.

I can also prove in particular cases the MLP's sole purpose is to remove the noise added from the skip connection.

[danielmarkbruce]

UDL has some dense math notation in it.

Math isn't just about proofs. It's a way to communicate. There are several different ways to communicate how a neural net functions. One is with pictures. One is with some code. One is with words. One is with some quite dense math notation.

[n3ur0n]

I would say UDL should be very accessible to any undergrad from a strong program.

I would not call the notation ‘dense’ rather it’s ‘abused’ notation. Once you have seen the abused notation enough times, it makes just makes sense. Aka “mathematical maturity” in the ML space.

My views on this have changed as a first year PhD in ML I got annoyed by the shorthand. Now as someone with a PhD, I get it — It’s just too cumbersome to write out what exactly you mean and you write like you’re writing for peers +\- a level.

[HighFreqAsuka]

I agree with that, I think UDL uses the necessary amount of math to communicate the ideas correctly. That is obviously a good thing. What it does not do is pretend to be presenting a mathematical theory of deep learning. Basically UDL is exactly how I think current textbooks should be presented.

[blauditore]

I think the mathematical background starts making sense once you get a good understanding of the topic, and then people make the wrong assumption that understanding the math will help learning the overall topic, but it that's usually pretty hard.

Rather than trying to form an ituition based on the theory, it's often easier to understand the technicalities after getting an intuition. This is generally true in exact sciences, especially mathematics. That's why examples are helpful.