Torch was never written as a static graph computation framework. Torch was/is more a tensor manipulation library where you are executing the individual operations step by step and the graph can be tracked and constructed incrementally from those operations. For this reason, much of PyTorch is about building a layer on top of the underlying components (which are focused on efficiently manipulating tensors and/or implementing low level NN components) rather than re-architecting Torch itself.
This won't be the same for TensorFlow as it was written with the concept of a static computation graph at its core. I'm certainly not saying it's impossible to re-architect - and many smart people in the community and at Google are devoting thinking and code to it - but simply that the process will be far more painful as it was not written with this as an intended purpose.
To note - there are many advantages to static computation graphs. Of particular interest to Google is that they distribute their computations very effectively over large amounts of hardware. Being able to do this with a dynamic computation graph would be far more problematic.
Does the upcoming XLA interact with this as well? I.e. compilation would be too costly for dynamic graphs, and so it would only make sense for static graphs?
I am not highly clued in to XLA as it's new, quite experimental, and most honestly I've just not looked at it in detail. Given XLA provides compilation, JIT or ahead of time, it doesn't really (yet) factor in to the dynamic graph discussion.
What would theoretically be interesting is a JIT for dynamic computation graphs. Frequent subgraphs could be optimized and cached and re-used when appropriate, similar to a JIT for Javascript. No doubt they're already pondering such things.
This won't be the same for TensorFlow as it was written with the concept of a static computation graph at its core. I'm certainly not saying it's impossible to re-architect - and many smart people in the community and at Google are devoting thinking and code to it - but simply that the process will be far more painful as it was not written with this as an intended purpose.
To note - there are many advantages to static computation graphs. Of particular interest to Google is that they distribute their computations very effectively over large amounts of hardware. Being able to do this with a dynamic computation graph would be far more problematic.