[andyjohnson0]

Back in 2018 I did Andrew Ng's course in Machine Learning on Coursera. It was pretty much "from first principles" in that you learned a bit of linear algebra and then you implemented algorithms in Octave, working up to MNIST etc. I felt like I came out of it with a good understanding of the basics, and that ML is maths not magic.

Looks like the course has turned into a multi-course "specialization" and I have no idea if any of it is the aame as the course I did. But it might be a place to start.

[jonnycoder]

I've taken part 1 of 3 in Andrew Ng's machine learning specialization which covers the math for supervised learning, linear regression, etc. As I started part 2 (neural networks), it built off the math from part 1 such as the sigmoid activation function. This is what I think of when the OP refers to learning ML from first principles. I highly recommend Andrew Ng's course and I feel like I need to take it again to really understand those basic building blocks.