Is there a way to run notebooks automatically, so you could regenerate notebooks like this after some library code changes or dependency upgrades and check that everything stillw orks?
Yep, Jupyter notebooks have an execution API, you can find more of it here - https://nbconvert.readthedocs.io/en/latest/execute_api.html. Hosted notebooks as a service is a growing area of investment and those services presumably use this API.
I built ReviewNB [1] to see visual diff for Jupyter Notebook changes & do a code review on it (by writing comments on cell changes etc.).
One of the next feature for ReviewNB is a CI pipeline for Jupyter Notebooks on GitHub. The idea is to make it easy for users to specify notebook "tests"/"checks" that can then be run on every change.
Given the nature of Notebooks, it's a bit hard to design CI for it in a clean way, but I appreciate any inputs or use cases that you might want to see fulfilled.