Anaconda distributions aren't exactly small. I'd also assume there's some sort of environment isolation, so you're getting multiple copies. Maybe one per workbook?
The embeddable package from python.org [1] is 20 MB unpacked. To install different sets of packages per workbook, just have Excel automatically set the module path as appropriate.
[1] https://www.python.org/downloads/release/python-3114/