The way the python interpreter works, there's no way for tensorflow to know you're using a for loop in order to optimize it. With method calls and operator overrides etc, tensorflow can hold off evaluating your graph, see what the final thing looks like, and then compile it efficiently.
You can think of tensorflow like a different language where the syntax tree is constructed of python objects. For loops end up being like a preprocessor macro on that syntax tree. They can add a predetermined set of extra nodes, but the "for loop" part isn't represented in the final syntax tree itself, so tensorflow doesn't know it has anything to optimize.
There are ways out of this: one is dynamic graph execution. But this is less optimizable. Tensorflow doesn't get to see the whole graph so it can't do as many optimizations, and it can't optimize for loops because it's actually executing eagerly on each loop iteration.
Another way is this article: write whatever python code you want to generate some transformation, and tensorflow can try to guess which efficient operations produce it. Then you can paste the efficient operations into your final program.
Your first two paragraphs describe TF 1 perfectly, and you also correctly point out that TF 2 has eager mode, which executes op by op.
The part you missed is that @tf.function lets users tell TensorFlow that TF should execute the entire function, inspect the graph that it produces, and optimize it. On top of this JIT functionality that tf.function provides, AutoGraph will inspect the source code of your program and see that there's a for loop. If the for loop executes over a Python object, then it acts as a preprocessor macro. If it executes over a tensor, then AutoGraph transforms it into a tf.while_loop. So you can have both Eager-style convenience and graph-style optimizations.
You can think of tensorflow like a different language where the syntax tree is constructed of python objects. For loops end up being like a preprocessor macro on that syntax tree. They can add a predetermined set of extra nodes, but the "for loop" part isn't represented in the final syntax tree itself, so tensorflow doesn't know it has anything to optimize.
There are ways out of this: one is dynamic graph execution. But this is less optimizable. Tensorflow doesn't get to see the whole graph so it can't do as many optimizations, and it can't optimize for loops because it's actually executing eagerly on each loop iteration.
Another way is this article: write whatever python code you want to generate some transformation, and tensorflow can try to guess which efficient operations produce it. Then you can paste the efficient operations into your final program.