Modularize by functionality, not by layer. Start with a simple end-to-end path and grow outwards, rather than trying to go top-down or bottom-up, and don't split into distinct layers until you're actually deriving value from doing so. Writing code when you don't have the use case yet is always a bad idea.
Yep. And then at the third sprint into this path you realize that the way the backend has been built during the first two sprints makes it impossible to deliver some essential or just very desirable feature that nobody took in consideration because it was outside the scope of the first two sprints.
Seen it happen multiple times.
That's the opposite of my experience. It's always the extra layer that was added for some planned feature that we never actually implemented in the end that gets in the way.
I disagree. This is how you end up with messy code that needs constant refactoring. Making time to come up with a composable design shouldn't be that difficult. It usually isn't.
All code should be constantly refactored. This way your code gets refactored as the design changes (which... it will) rather than having to shoehorn an architecture that worked great originally into a set or requirements that have moved on.
Not true. If you have design ready before the implementation starts (as you should unless it is very simple) then you are free to implementation things separately. Just have the interface well specified.
This is sad, but my point still holds. Perhaps being honest with oneself about real tasks that need to be done will help. Planning the architecture/feature, doing proof of concept etc. are all valid tasks even if they don't bring immediate business values. If you claim that you can only do business features without doing any exploratory work then you are effectively claiming to have an oracle giving you perfect solutions out of the blue. In which case you should stop whatever you are trying to do and start selling the services of your oracle.
We always do a brainstorming before implementing any serious feature / change. Once we have the initial design business gets involved (if we can get their attention...). The results usually works without needing any drastic changes, which is still better than no planning at all. It does require involvement from a number of parties though.
Design is better when it's informed by implementation, IME. The only complete specification is working code; if you accept something lower-fidelity then it's very easy to miss ambiguities.
Even code have bugs so it's not perfect. Your design can incorporate high level algorithm to be implemented, ins/outs, or whatever else high level constraints are most applicable to your domain.
Depends what you're building. Which leads us to the worst (meta-)aspect of Agile/Scrum these days, which is that it's the industry's current favourite hammer and so it gets used to bash every single problem. Now, hammers are quite versatile, but you have to know when to bash, when to use the claw, when to lever or just nudge things instead of swinging.
The moment you start getting dogmatic about your process is the moment your decisions start being driven by something other than your actual needs at hand. And that's the moment when you start to produce a bad product.