[GIFtheory]

So, this doesn't apply to torch specifically, but to give an example from my own experience... I have a tensorflow model that takes 45 minutes just to build the graph. You can imagine how frustrating it is to wait most of an hour to run your code, only to see it crash due to some trivial error that could have been caught by a compiler.

Although this is somewhat of an extreme example, I'd say most of my tensorflow models take at least several minutes to build a graph. Anything that reduces the possibility of runtime errors would therefore speed up my development cycle dramatically. I'll also note that the OCaml compiler is lightning-fast compared to say, gcc.

[Cacti]

Are you sure you’re writing it in a reasonable way? I’ve made some awfully large nets (implementing current state of the art models) with TF and I’ve never ran into anything like that. I mean I might have 30 seconds to a minute before it gets moving but that’s the most I’ve seen, and that’s including the entire init process (reserving GPUs, preprocessing enough data to keep the cache moving, initializing variables for optimizers and such, and so on). Some of these models have an absolute ton of ops.

What are you doing that requires a 10, 30, 45 min graph build?

[GIFtheory]

I'm fairly sure the model is implemented in a reasonable way. It's an experimental deep generative model based on https://github.com/openai/glow, though more complex because the warp and its inverse are evaluated at training time, and the outputs fed to other things. The warp has around 200 layers, IIRC. The model requires keeping track of the evolution of the log-determinant of the warp after each operation, along with the derivatives of those things... so the graph can get pretty huge.

[Cacti]

Interesting, thanks.