[cosheaf]

It's already dead. Google mined all the links that were curated by the initial internet communities for all it was worth and turned them into profits for Google's earliest employees and shareholders. Now that no one is curating useful links anymore their search quality, unsurprisingly, is deteriorating. Without human curation there is no signal for Google to use anymore and whatever signal is there is just SEO spam that is optimized for serving ads. It's like an ouroboros eating its own tail.

[6gvONxR4sf7o]

It’s not just the links. After the links, google mined facts, like “how much does a german shepherd weigh,” so no on gets those clicks, and the incentive is gone there too. They’re even mining the snippets of the content, lowering the incentives for creating that too.

[cosheaf]

It's essentially a machine for printing money and people don't really understand what they're giving up in exchange for "free" search results. Google is beholden to market forces, it's no longer in the business of indexing useful information because the market doesn't value useful information, it values ad revenue.

This is a structural problem and anything that gets large enough will succumb to the same forces. If the incentives are for optimizing ad revenue then that's what all corporate machines will do at scale, regardless of their initial motives and incentive structure. It doesn't help that Google is also an ad network, hence the ouroboros aspect.

[Lascaille]

You say it's a machine for printing money in a topic about people complaining that it doesn't work any more. It may have been but it won't be forever if things keep going the way they're going. Quality content is already being locked away.

[cosheaf]

We're in agreement. I don't think quarterly earnings are the right way to design and build products. Maximizing profits is not correlated with value and is often inversely proportional to it. Google was so successful that they changed the incentive structure of all content on the web. Now they're like the yeast drowning in the byproducts of their own metabolic processes. They exploited whatever nutrients were available (hand curated links) to make them initially successful and now there is no more worthwhile content being generated that is not designed to rank highly on Google (which is not the same thing as quality content and is instead content optimized for generating ad revenue from the Google ad network).

It's the same with social networks, upvotes and likes skew the the type of content that is generated to be liked by the people liking and upvoting instead of being insightful. Popularity is not the same thing as insight so the social web is full of mostly useless but popular content.

[Firmwarrior]

Well, if it makes you lads feel any better, we're not the only ones who feel this way

Paul Graham has been pushing for someone to disrupt web search on and off for at least the last 10 years: http://paulgraham.com/ambitious.html

(he tweeted about it more recently, but I couldn't dig it up since I'm not much of a tweet man)

[AnimalMuppet]

This sounds almost like Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure."

Google made links on the web the measure of how good a page was. That became the target of everyone trying to do SEO. As a result, it stopped being a good measure of how good a page was.

But in the long run, nothing will work in that environment, because every measure will be gamed as soon as people figure out that Google is using it. Google's only choice is to try to stay ahead of the SEO crowd, and I'm not sure they can do that (well) for too much longer. In fact, if the article is to be believed, they're already starting to fail.

[cosheaf]

Yes, it's very similar with the added caveat that Google has an interest in serving results that have ads from their own network. This is why Google's metrics can be hacked. Anything that is barely above being classified as spam but serves ads from Google's ad network will be prioritized over other results simply because they have to hit their quarterly revenue targets. SEO hacking is not possible if a search engine is just a search engine but Google is also an ad network so they will always be susceptible to being gamed.

This is also the case for social media platforms. They're incentivized to surface content that generates engagement and ad revenue. Basically ads are at the root of all problems when it comes to the internet and the content on it.

[gundmc]

Here and further down this thread you repeatedly allege Google upranks pages if they participate in their ads program. Google explicitly says they do not do this [1]. Do you have evidence to contradict this?

[1] - https://support.google.com/adsense/answer/9717?hl=en

[cosheaf]

You should sit down and think about how Google makes money instead of reading whatever is on their support page.

[gundmc]

Okay, so it's an unsubstantiated conspiracy theory. Got it.

[cosheaf]

Maybe I'll make a support page and then you'll read it and believe it.

[Lascaille]

> SEO hacking is not possible if a search engine is just a search engine

Uh, yes it is. If the owner of the site being searched is generating profit from that site being searched then they will game the search engine's algorithm to get them the most clicks.

[cosheaf]

I'm not interested in a pedantic argument. If you didn't understand what I meant then you should have asked for clarification. A search engine designed to surface useful information is not gameable if it is not in the business of generating quarterly profits from its own ad network. A site designed to drive traffic to itself can still try to hack the system by generating spam but without Google's incentives for surfacing such content because it serves ads from its own network there will be fewer such sites and useless content to go along with it.

At the moment Google is incentivized to uprank spam because the spam comes with ads from its own ad network.

[AnimalMuppet]

Of course it's still hackable.

A search engine finds 50 pages that are exact matches for the search. Which one does it present as the top of the list? How does it decide? Unless it decides literally by a random number generator, however it decides, someone will try to discover the algorithm, and exploit it. This is true whether or not the search engine allows or displays ads.

> I'm not interested in a pedantic argument. If you didn't understand what I meant then you should have asked for clarification.

I don't think it was a pedantic argument or a misunderstanding. I think Lascaille understood your position and disagreed with you, not just pedantically but over the substance.

[cosheaf]

This is actually what google does. They find the 50 sites for your query and then do a multi-armed bandit test to see which one gets the most clicks but with a bias towards sites that serve ads from the Google ad network. A search engine without that bias is not gameable because it will converge on the results that is most popular for a given query and not because it also serves ads that affect the search engine's bottom line.

Popularity is gameable but not the same way as Google is currently gameable because as soon as a site becomes popular and starts exploiting its ranking it will be easy enough to add a decay factor to prevent such sites from dominating the top results during the multi-armed bandit stage of ranking.

In any case, the logic of why Google is going to shit is obvious. Arguing about fixes is not going to change their underlying business model and why spam is dominating their results. As long as they are a search engine, an ad network, and a corporation that must maximize profits their results will continue to deteriorate until the top results are all just spam.

> I don't think it was a pedantic argument or a misunderstanding. I think Lascaille understood your position and disagreed with you, not just pedantically but over the substance.

Then that wasn't clear and seemed like a pedantic point since it's obvious that any algorithm is gameable and I should have made it clear that I wasn't talking about a perfect search engine but one that was not susceptible to profit driven spam (which is currently the reason that Google results are going to shit).

[Lascaille]

>I don't think it was a pedantic argument or a misunderstanding. I think Lascaille understood your position and disagreed with you, not just pedantically but over the substance.

Yes, I was a bit disappointed with the reddity response. The sites admin has an incentive to have their site appear first. The quality of their content may not suffer but they will additionally try to identify how the algorithm works and 'game' it.

[marginalia_nu]

I honestly don't think this is the problem, like at all. There are human websites made by humans, still. There's more crap, sure, but the good stuff is largely still out there.

The problem begins and ends with the conflict of interest that Google both sells ads and selects search results. If they didn't have a vested interest in people visiting sites with their ads on them, they could decimate the number of spam results.

[_cs2017_]

Which websites are shown in the search results is not influenced by whether Google has ads on them.

The only thing that influences Google search results is Google's desire to keep as many people using Search as often as possible, since nearly all of Google's money comes from showing those text ads at the top of the Search results. This is all public information, you can read it in the 10K etc.

So if Search sucks, it's not because Google has the wrong incentives but because they can't solve the problems Search faces.

[ajmurmann]

Hard to proof this but anecdotally lots of energy that used to go into people's hobby or passion websites now is going into digital platforms like reddit, pinterest or Twitter, or if lucky substack or Medium. Some of those are walled gardens and Reddit as discussed here is where people now search. Much less out on the actual net

[marginalia_nu]

Well, since I actually run a search engine looking for this sort of stuff, I think I'm able to demonstrate that a lot of what was out on the net is still there, just exceptionally hard to find on Google.

Admittedly more in some areas than others.

[ajmurmann]

Hah, I didn't catch that I was responding to you. Definitely would defer to your experience. Love your search engine btw. It brings the fun back to the web!

[marginalia_nu]

If anything I think I'm only scraping the surface, since a lot of my methods involve ruthlessly discarding data that doesn't live up to a fairly blunt set of criteria. I think with something like a headless browser (and a lot more processing power), I could probably use laxer standards and find even more good stuff.

[cosheaf]

Show me a site made by people with manually instead of algorithmically curated links and content.

[marginalia_nu]

I'm not quite sure what you are asking, but this maybe one of these?

https://simplifier.neocities.org/

https://cheapskatesguide.org/

https://indieseek.xyz/

https://www.gameboomers.com/

https://sadgrl.online/

[xanaxagoras]

Hello, I have never heard of cheapskatesguide.org before and I am loving it so far. Thanks for sharing, I will check out the rest as well.

[cosheaf]

Great, now compare this to all the content generated on content farms for selling ads and gaming search engine rankings.

[marginalia_nu]

Right, but that's pretty trivially identified. I've had great success doing that with my search engine. Here's a thousand domains that are low in farmed contents in no particular order:

https://downloads.marginalia.nu/good-domains.txt

[cosheaf]

So why do you suppose Google doesn't surface the domains and results you presented? For example, I searched for "game reviews" and gameboomers was nowhere to be found.

[mark_mcnally_je]

https://goodwebsites.org

[foobarian]

Wonder how we could set up an alt-web without the incentives that cause this problem. Delist any for-profit site? How would the sites keep the lights on without ads?

[agumonkey]

To me it's more a sociological problem than technological. Also networks have changed.. somehow the decentralization idea is spreading fast. For ideological, technical, cost .. or other reasons. Some people start neighborhood wireless networks etc.

It also seems to me that internet has somehow became a middle man and is not providing human deep enough interactions, especially outside chat-like website (basically any exchange, business)..

I could envision a whatsapp like system with quality control for producers and transparent transaction/tracking/accounting management offered by the network so people spend less time on side-loads and just focus into helping each others and doing what they need to.

[foxfluff]

People could certainly run hand-curated indexes and search engines seeded by such. I think marginalia.nu search behaves somewhat like that.

I'm really interested in the idea of decentralized search where everyone has the power to choose for themselves who to trust.

> How would the sites keep the lights on without ads?

Making them turn off the lights is the goal. Good riddance I say, once we get there.

[foobarian]

> Making them turn off the lights is the goal. Good riddance I say, once we get there.

I wish there was some middle ground. Think of all the useful Youtube videos showing how to play an instrument, do woodworking projects, fix cars... there is a vast amount of knowledge there. Maybe YT should be nationalized :-)

[rileyphone]

Youtubes early success was due to being a free video hosting platform, the monetization just led to the rise of 10:04 long videos. In any case most larger creators will put sponsorships in band like the good ol days. I’m hoping decentralized alternatives can take over like Peertube or Odysee, but I do also appreciate the more traditional business model of Vimeo.

[Lascaille]

> People could certainly run hand-curated indexes

Do you think there's a lot of good content out there left to index?

[marginalia_nu]

I've had no problem finding sites to index with my search engine. Like it's tiny compared to Google today, but it's about the same size they were when they first started out. Leads me to think there's probably as much, or more now as there ever was. There's just more noise to go with it.

[foxfluff]

Yep! There's a massive amount of useful content on the web. A lot of it is just a pain to find right now, because the quality of search is bad and the amount of garbage is a thousandfold greater.

It's true that walled gardens have been eating up useful information and that is a real shame, but make no mistake: there's still a ridiculous amount of good stuff on the open web. They're not playing the SEO optimization game so they get buried.

[Lascaille]

That actually has made me feel a lot more positive than I would have imagined.

I have sort of got stuck in this bubble of reddit and not much else and... the way people behave has really sapped my desire to contribute anything to the world. I'm looking around me at the opinions a lot of people are expressing and I think... no, these people don't deserve nice things. It seems sometimes like the whole world is full of venal, bitter and divisive people that do nothing but snap bitterly at the heels of anyone that makes something of themselves or strays too far from safe consensus.

[cosheaf]

Federation is the only reasonable solution at this time but the technical overhead of federated search is high enough that most people won't use it so it won't benefit from network effects like Google did in the beginning. There might be a combination of blockchain juju that could make federation viable but all the thought leaders in that ecosystem are too high on their own supply to realize they could use blockchains for anything other than gambling.

[fsflover]

https://yacy.net

[ForHackernews]

You could check out https://gemini.circumlunar.space/

It's a new internet protocol (NOT www) designed to be minimalist and interesting to hobbyists.

> How would the sites keep the lights on without ads?

The same way they did in the web 1.0 days - somebody would maintain the server themselves, or pay to have it maintained.

Discussion: https://news.ycombinator.com/item?id=30072085

[endisneigh]

There’s no better solution - you either have ads or a paywall, which will result in few users.

If there were a better solution we’d all already be using it. Certainly you can rely on savvy people to produce free stuff, but the total amount of content will be drastically lower and therefore fewer consumers.

The best thing is to just use bookmarks and your favorite sites’ own search.

There’s just too much trash on the net

[notacoward]

So basically Google is a parasite that killed its host. I really really like that analysis. Thanks!

[cosheaf]

Thanks but it's not really my analysis. I learned it from an art project: https://googlewilleatitself.com/.