[smhx]

the biggest hindrance to do this are language constructs that cannot be or are inconveniently expressed in the symbolic graph, such as python's if vs tf.if and for vs theano.scan, or conditioning on some python-code (not tensor operations). So to build an eagerly evaluating symbolic graph framework that is allowed to do arbitrary things would mean that you would (to an extent) reimplement the language you are working with.

[liuliu]

Let's assume Tensorflow has basic symbolic computation graph expressiveness. What you would do is to build a symbolic representation while executing your graph inline, your symbolic representation doesn't need to have any control structure, it is simpler than that. You execute while loop in Python as usual, and your symbolic representation won't have TF.While at all, it will simply be the execution you performed so far (matrix mul 5 times).

Once you have a reasonable symbolic computation graph library, you don't need to explicitly build a "tape" because the symbolic representation will record the order of execution and reverse AD even graph optimization (applying CSE etc) come naturally as well.