I spent a lot of time digging through all the oft-mentioned search engines here, and the interfaces between G, B, DDG and the others (Runnaroo, Startpage, etc) are all surprisingly similar. And they're all pretty unappealing (to me).
Visualizations of results have improved so much in the last 10+ years, hopefully "visualization of search results" can also be advanced.
As the creator of the upstart search engine on that list (Runnaroo), I do try to differentiate it from the the search providers, but there is a risk of being too different from what people expect.
Take a look at the below search [0], it is distinctly different from the SERPs of the other search engines for the same query, but you are correct that we still focus on just listing information vs. visualizing it. Maybe something for us to explore.
From a search visualization perspective, check out Carrot2 [1] for something more innovative. Swisscows [2] also has some nice visualization elements.
I don't think the index is the challenge. Anyone with enough machines and gigabits can index a big enough portion of "the web". The real difficult part is ranking and understanding web pages themselves.
Why would they? If Apple makes it the default search engine on iOS they immediately dwarf the ddg user base. And the front end is the easy part of a search engine.
What we really need is a spider/database that's publicly-funded like PBS, with an open API that anyone can write search engines on top of. That's the way forward for search innovation.
And while DDG's front end was innovative 20 years ago when Google did it, it's pretty basic (which I love) and shouldn't be too difficult to copy.
The big challenge with search is the index.