I think ordinary income is < 5% of the burden (not just complexity). Think about where the time is spent on your own return: it’s not copying 10 numbers from your W-2 or counting your kids, but rather figuring your investment gains/losses, K-1s, Schedule C for your side work or app.
People with only earned income already have a very easy time of it here as well.
This isn't true. Lower income people are more likely to work more than one job simultaneously, or to do casual work or a string of jobs, or be 'contractors' than have a single W2 with accurate numbers. Only 20% of taxpayers who qualify for free direct filing use it - mostly because they don't even know it exists. Many Americans are challenged by even the process of completing the form.
The W-2 case with maybe a 1099 and standard deduction is pretty simple. And should probably be even simpler by autofilling a "Here's what we think you owe/are owed" return. But you get into complex returns pretty quickly for the reasons you describe. I don't even have that complex a return and what I got from my accountant was probably a 1/2 inch thick last year.
Assuming no submission is equivalent to a submission with earned income only (standard in Europe) would save an incredible amount of time.