Fastai. Updated series of lectures and notebooks for 2022. High level as well as building neural nets from scratch. Doing it at the moment and enjoying. Good as a starting point for more in depth studies.
part two of the course covers building everything from scratch and covers almost all of what OP wants, so this is a very good choice. anyone interested in a study group?
Is there any benefit in learning how to code the algo from scratch? In most cases, We will just use standard libraries, unless the goal is to build up intution?
I think it's still helpful as it indeed will give you more understanding. You wouldn't want to go for optimised algo's etc unless that rocks your boat, but just basic implementations is good to learn the basics from imho. And pretty satisfying to boot, again, imho.