That's a good point. If you had to explain how everything works without the libraries, you'd probably end up writing a book on how pandas and numpy works, not how ML works.
Good ideas. I think I'll try to add an appendix, minimize the number of numpy functions used, and explain any of the weird ones that are real time savers. Thanks for all your thought.