[DanielBMarkham]

I've been observing SEO for several years, and out of all the things I've learned as a technologist, SEO is the most frustrating. It's consistently murky and unpalatable to me as a programmer because of it's lack of logic. And as far as pointing out how it works, I'm not happy with the state of affairs. I've simply decided that I'm going to talk frankly about how I understand it. Maybe you guys can straighten me out. I see so much posturing on SEO -- even by the guys who claim to be "shooting straight" that it's discouraging.

The problem you have is that you are trying to describe a system that has no specification. That is, there is no testable definition of "best content" that is repeatable and applicable to all users. After all, this was google's entire schtick -- the reason everybody thought it was cool was because they managed to hack the problem better than anybody else had up until then. But there is no solution. It's not that kind of problem.

One of the reasons SEO drives me nuts is that the concepts we are so used to in programming, "black box", "answer", "best content", "user", etc -- don't really have firm meanings in the way we would like them to. Yes, it would be awesome if there was a little magic box that told me what to do (or gave me all the answers) but -- and this is important -- even if there was, it wouldn't be a black box. We live in a digital age. Anything that can be put into code is instantly commoditized.

There is another assumption here that is equally slippery (aside from the fuzzy nature of all the adjectives and the impossibility of making the system opaque), and that is the idea that somehow one can determine content quality mechanically. That's like saying you can pick the "best" painting at an art show by using some kind of hand-held scanner and an image-processing algorithm. Content is about people interacting with people -- it's not deterministic. We are not machines.

We keep wanting Search Engines to work like the library: go to the card catalog, pick a topic, find authoritative sources. But they keep working like the dance hall: show up with your best suit, make some friends, and work the crowd, become popular. That's frustrating. But as one SEO expert told me in a recent interview[1], if you don't use social signals (popularity), how else would you do it? This is the way we've been judging content since 3 guys sat around in a cave looking at mammoth drawings.

[1](If you have time, you might want to listen to the interview. I tried to touch on this exact subject because I know how touchy an issue SEO is: http://www.hn-books.com/Books/SEOMoz.htm#the_video )

[derefr]

> Content is about people interacting with people -- it's not deterministic. We are not machines.

Here's a thought experiment for you, then: imagine that someone builds a product called TheBestSearchForYou, where you step into an advanced MRI machine and think your query. Then:

1. Your brain is copied into a billion virtual emulations, each of which then sees a single page from the TBSFY's index.

2. TBSFY then measures (through whatever destructive, invasive process is necessary) which em had the most positive response according to their own utility functions (which are all the same as yours, mind.)

3. TBSFY returns to you the page that gave "you" the strongest positive response.

4. Then the ems are stopped and deleted, because—since you're going to be a slightly different you the next time you make a query—it's useless to keep them around ;)

I would say TBSFY adheres to the definition of an "optimal" search engine, however infeasable. The actual question it raises, though, is how closely TBSFY's results can be approximated by a company that knows increasing amounts of personal information about you.

[DanielBMarkham]

This is a fascinating line of discussion because such a simple question generates such unforeseen complexity Here we're only three or four comments in and we've already brought in an MRI and a supercomputer. :)

But once again, there are assumptions you are making that I'm not ready to go along with. The biggest one is that I know what is good for me -- that I have a set of criteria (even if un-describable) that can be replicated. The beauty of real-world interactions is that there is a mix of happy and sad, interesting and boring. Sometimes the guy you meet at the gas station says something off-hand that you think about the rest of the day. Human social interactions require an element of randomness. As a social animal, I need to interact with a machine that also has social needs, not a machine that is trying to make me perfectly happy.

That's a big deal, because if you're looking at this as some sort of optimization function, yes, we could somehow come up with a system that would optimize your happiness, or emotional reaction, or whatever. I'd argue that places like HN and most social sites are early attempts at optimization. We will continue to get better.

But let's say I'm angry. Some guy in a clown suit cut me off in traffic. You know what would make me happy? Seeing a bunch of clowns on fire, that's what. But do you know what I _need_ to consume? Something that reminds me that all people are human and deserve respect. I would absolutely _hate_ consuming that material, but a week from now, a month from now -- anytime but right now -- I would tell you that it is the right thing to do.

In fact, this optimization process is a grave danger: we risk becoming zoned out in little echo chambers where the only thing we consume is the rock candy of me-too, feel-good content.

It's a tough, probably intractable problem, at least until we come to terms with what the nature of the optimization needs to be. It would not surprise me to discover that the optimization that is best for the species as a whole and the optimization that makes me the happiest are two completely different things. Not good.

[derefr]

> It would not surprise me to discover that the optimization that is best for the species as a whole and the optimization that makes me the happiest are two completely different things.

That is absolutely so--thus the need for a complex http://singinst.org/upload/CEV.html, rather than a simple one which just averages everyone's utility functions together.

[hallmark]

That was a pretty dense read, at least for me. I particularly liked two parts, though they may not add up to the meat of the paper:

1. "If you find a genie bottle that gives you three wishes, it's probably a good idea to seal the genie bottle in a locked safety box under your bed, unless the genie pays attention to your volition, not just your decision." Nice. Grown up version of some cartoon plots.

2. The description of spread, muddle, and distance. Very relevant to political theory.

If you are able to compose a tl;dr of the entire paper, that'd be awesome.