This is greatly mistaken. Continuous refactoring is imperative. Not doing it is like owning a house and never doing basic maintenance. Refactoring is orthogonal to optimization.
As a general rule, and operus memorandi, I agree 100%.
But like any rule, if we're talking specifics, there are exceptions. Scripts you use occasionally do not need developer time committed to iterating on. They can technically be improved, and refactoring them would likely lead to benefits, but they don't require it. There is nuance.