I don't think it's mainly about agile. There is plenty of good examples of fragile OSS out there that has nothing to do with agile. Entire npm ecosystem is very good example.
It's not only about Agile, but incremental development is always detrimental especially for complex projects. Also, most companies apply some ghetto version of Scrum, most don't even know what SOS (Scum of Scrums) is, pick and treat the Scrum Master incorrectly, don't even do Planning Poker, don't use Epics at all, don't define stories right, don't use burndown charts at all, etc. In most cases, I see Scrum as an eyewash for the business. It's a pseudo-scientific approach to prove to business we've done something this week, and a solid excuse for missing deadlines, etc.
It's not only about Agile, but incremental development is always detrimental especially for complex projects
I really have to dispute that.
First, it's not really clear what you mean by "detrimental" - is that to software quality, or development schedule, or suitability? Or all of them, or others? Agile is very clear about the trade-offs - working products being more important than documentation for example, or responding to change over following plans.
Second, I don't think it's true. I've directly observed, in multiple organisations, that agile development is perfectly capable of delivering good systems on-budget and on-schedule. If anything, it's especially good for complex systems, which are those most likely to require flexibility and have hidden requirements.
I also dispute the idea that "ghetto Scrum" is invalid. Development processes should be tailored to the requirements of the product, organisation and team. Not every team needs epics or planning poker or scrum of scrums, and it's totally valid to take some of the ideas of scrum and apply them to your own development process without having to buy the whole thing.
I consistently see agile development hand waved away with this argument that it doesn't work because there are companies which do it badly, and it's baffling to me.
You can't build a house from the roof, but that's what Agile indirectly makes you do. Although Waterfall is the other dangerous extreme, it makes you think, correctly estimate, and plan. You can't apply Scrum to every project - especially those with firm deadlines! Yeah, it's good for startups, where feedback and market shape your product, by can you build a spaceship using Scrum? Honestly, as an engineer, even when I use Scrum, I still use Gantt charts, and that's another thing that makes people make funny faces in Scrum.
You can't build a house from the roof, but that's what Agile indirectly makes you do.
And you can't compare software with a house, or even a spaceship :) For one thing, we have far fewer constraints.
When we build software we are (generally, though not in every case) building a solution to a problem rather than a strictly defined product - and in many cases, we are unsure about what the final product will look like! Spotify did a good illustration of this process in this slide: http://www.chrisyin.com/images/spotify-mvp.png - the point is to continuously deliver features to customers and using the feedback generated to inform ongoing development, which like you point out, is absolutely essential in some cases.
, but that's what Agile indirectly makes you do. Although Waterfall is the other dangerous extreme, it makes you think, correctly estimate, and plan.
That's perhaps the idealistic view, but there is a cost there too - change becomes much harder. I also suspect that there is no such thing as a correct estimate!
You can't apply Scrum to every project - especially those with firm deadlines!
Maybe not, and Scrum is not the ideal methodology in all cases - but it's also not the only way to be agile. It's also a fallacy that planning in advance means a project will meet a deadline - let's remember that Scrum doesn't typically mean you are doing different work, just that you are slicing it and delivering it differently. I'd argue that one of the main differences is that if a Scrum-developed project misses it's deadline, you still have a working product - it just doesn't necessarily have all the features you'd like. But if you are correctly building the most important and complex features first, then that's potentially much better than the "waterfall" outcome of "the product isn't complete and can't be used".
can you build a spaceship using Scrum?
I probably wouldn't want to try it, but this is an example of a project with incredibly tight constraints on what can be built - far more than software typically is. But I'd imagine it's also possible to build components of any product in an agile manner.
I still use Gantt charts, and that's another thing that makes people make funny faces in Scrum.
That's great, and it's something that should be encouraged - development methodologies should be determined by what works well for you as a developer, team, or company! A Gannt chart, assuming you are changing it as product requirements change, is basically the same as a Scrum burndown chart with additional planning thrown in. And we do the same sort of thing - "these are the epics, this is how long we estimate they will each take, and here is the order we will do them in." It's not so different.
Ultimately these methodologies are designed to help projects cope with change, which has been a long-term problem in correctly sizing and executing software projects. Maybe they won't always work for every project with particular constraints, but I reckon they are far more applicable than some people believe.
