I'm really interested in learning the math basics for deep learning. Is there an online guide that you can point us to if we want to learn the basics of calculus, linear algebra, probability theory? In other words is there a "fast.ai" version of "math for deep learning" out there?
Yes for linear algebra, though it's probably best suited to people with at least some exposure to matrix/vector math (for example you've used SVD but have no idea how or why it works): https://www.fast.ai/2017/07/17/num-lin-alg/
For probability, machine learning is more about statistics (the two are related, but courses explicitly about probability will cover different things), so I would lean towards that. An Introduction to Statistical Learning in R (ISLR) is a frequently recommended book. You can ignore the R and do the exercises in Python.