[edublancas]

Papermill is great but has quite some limitations because it spins up a new process to run the notebook:

- You cannot extract live variables (needed for testing)

- Cannot use pdb for debugging

- Cannot profile memory usage

You can do all of that with ploomber-engine (https://github.com/ploomber/ploomber-engine).

Disclaimer: I'm the author of this package

[ziddoap]

Not disclosed in this comment is that edublancas is

>Ploomber (YC W22) co-founder.

[Kalanos]

who is a great technologist with a lot of hands on experience. if it made sense to leverage papermill, he would have done so and focused on something else.

[ziddoap]

What does any of this have to do with disclosure?

[Kalanos]

calling attention to disclosure suggests bias. i'm obviously saying that i trust him not to be biased.

[fastasucan]

Not obvious at all. And no. It doesn't have to suggest bias, just a lack of disclosure.

[throwpoaster]

iirc, a few years back I was able to do all of these things with the Papermill IPython runtime.

Papermill is great, but yes: lots of room to hack on it and make it better.

[edublancas]

has papermill deprecated the ipython runtime? I used papermill extensively in the past and I never saw that in their docs.

[throwpoaster]

It’s been a while but you do it with a custom kernel and maybe some entry point tweaks. IIRC.