[albertzeyer]

What does "from scratch" really mean? You don't reimplement Python itself, or invent a new GPU hardware, a new CUDA including compiler, etc. You don't reimplement the OS. Where do you draw the line?

Do you reimplement matmul or other basics?

Do you reimplement auto-diff?

Maybe PyTorch or TensorFlow using auto-diff is a good "from scratch" basepoint, without using predefined optimizers, or modules/layers, or anything. Just using the low-level math functions, and then auto-diff.

[rg111]

I don't understand why you are arguing with me.

Yes, in those books, you do implement matmul, auto-diff, etc.

[albertzeyer]

I don't understand. I don't argue with you? I also don't speak about these books. I just made a generic comment, to start a discussion.

I just wanted to point out that "from scratch" is not really well defined. There is always some arbitrary line. I just found it interesting to discuss and think about where to draw this line exactly. Obviously it's never really from scratch, i.e. you don't reinvent the hardware level, for example. Or you don't start with teaching quantum physics. So you start from somewhere.

And I was wondering whether auto-diff is maybe something which could also already be the starting point, or also matmul. Reimplementing an efficient matmul on CUDA is not easy, and might distract from the main deep learning content. But it depends also where you want to have the focus on.

[rg111]

Now it makes more sense.

Maybe I misread your comment.

Thanks for rewording.

Where do people draw the line? Where they want to.

Some people choose to go closer to the metal than the others. It's just people's choice.

Some just write stuff in Python, some write CUDA kernel for themselves (some among them had to), a friend even wrote his own compiler and programming language for Deep Learning.

So it depends on your choice. And how much deep you want to go also depends on what you want to do- i.e. you choice of career, direction of research, etc.