Imagine that you've refactored code and reduced complexity, but also reduced performance (the caching example) - would you move forward with the refactor?
From my perspective there should always be a buy in - after refactoring the system is more understandable, but also more coupled. Is this fine? If no, can given refactor be merged now and result tackled in separate refactor. Caching refactor can have a buy in as well - ie. remove caching because given request shouldn't be cached, or this functionality should be decoupled and done elsewhere