[tiredofcareer]

    set beresp.ttl = 30s;

Varnish gives you your own throttle for how often you want invalidation. It's a tool specifically designed to make misbehaving apps -- i.e., that widget -- misbehave. You just have to slap the dog on the nose when it behaves. I'm just saying you could have made this work on Linode (and I have), but I do see your post-purchase rationalization at work, so I know anything I say will be fruitless anyway.

There's also the possibility that you had shitty neighbors.

[jacques_chester]

Right, and when I hosted on WPEngine and then on Pagely (and both of them choked), my users immediately piped up that the recent comments were inaccurate. In fact there were a number of page freshness anomalies which I believe were down to whole-page caching that I was frequently quizzed about by users.

If your page is basically static, then yes, whole-page caching will fly. But several of the sites under my supervision are, to quote Pagely's founder, used "like a chat room".

Edit, per your edit:

> I'm just saying you could have made this work on Linode (and I have), but I do see your post-purchase rationalization at work, so I know anything I say will be fruitless anyway.

Basically, I was there, I saw the numbers and I know why they wound up looking the way they did. I suspect that anyone in my particular situation would have evolved their approach in the same way that I have. I've been running Wordpress blogs since 2004. I feel that I've picked up some ideas on how to make it fast, but sometimes the general solutions don't work because you have a specific problem.

[tiredofcareer]

Got it. So you have users that expect to have real-time conversations in the comments on a blog, meaning you can't optimize a blog application like a blog is designed to be used, meaning you have to pay for SSDs in order to make your blog function at all because apparently MySQL can't handle N inserts/second and however many people have these conversations refreshing every ten seconds, generating a few SELECTs that are rapidly in query cache.

I completely understand how this could be a problem and how switching providers would fix it.

[jacques_chester]

Sarcasm aside, you clearly don't understand what my problem is.

1. Recent Comments invalidates every page it appears on whenever a comment is posted in any thread. In practice that means that the entire site cache is invalid. That breaks whole-page caching models.

2. This means that Wordpress will regenerate from scratch.

3. This means first of all generating the page, which joins multiple tables including TEXT fields. Because of the brilliant design of MySQL, these joins ignore indices on the joining fields and frequently the join will occur on disk.

4. The Recent Comments plugin also causes joins on disk because it too refers to tables with TEXT fields.

5. The query cache helps a lot, but the site on Linode was still observably jammed on I/O, even when MySQL was given an entire server to itself.

However, if you feel you can do it better, I am happy to engage your services as a fulltime replacement. WPEngine said they could do it for $250/month (they couldn't). Pagely said they could do it for $149/month (they couldn't). I invite your bid.

[tiredofcareer]

> Sarcasm aside, you clearly don't understand what my problem is.

Yeah, I spent this entire thread clueless about the issue you're running into, even though you spelled it out a few different times because you think I don't get it. Wordpress falls over under normal site load, film at eleven.

Since you want to switch to condescension, I'm assuming wise sir moved MySQL's tmpdir to a RAM disk and found that unsatisfactory for his mystical, MySQL-breaking SELECT/INSERT workload? Also, I'm far more expensive, and I know that WPEngine is multitenancy Wordpress on Linode in the backend. (That one's free.)

[jacques_chester]

We're not being very productive here, are we? I could nitpick your comment just now but it wouldn't change your mind either.

You think I'm an idiot. Possibly you think I'm a liar.

I don't think you're an idiot. All I can do is point out that I looked at the numbers, I've tested various strategies or tools (and adopted most of them), I referred the problem to the experts, and this is where I've had to go.

So let's just ignore each other from now on.

[tiredofcareer]

I think you're unnecessarily combative in the face of advice and sitting comfortably atop your pillar of experience, ready to shoot down anyone that dare take time to offer you advice. You've appealed to your authority on this matter more times than I can count. Look at how you've approached the conversation from the very first reply, which set the tone for the rest:

    - I've been using WP Supercache since it was released
    - [I've been using] Nginx for I think 4 years at this point
    - Basically, I was there
    - I saw the numbers and I know why they wound up looking the way they did
    - I've been running Wordpress blogs since 2004
    - You clearly don't understand what my problem is

Now I've asked you something specific. You've lamented that you identified the issue as on-disk joins, when MySQL has to resort to an on-disk temporary table due to a TEXT column. That's discussed here[1]. I'm assuming, because I didn't assume you are stupid (unlike in the inverse), that you deduced this was the case by inspecting created_tmp_disk_tables. I then asked if you tried removing the disk from the picture by creating a RAM disk, mounting it somewhere, then instructing MySQL to use it for its temporary disk table area by setting tmpdir. I also assume you know that tmpdir defaults to the system /tmp, which might not be on a filesystem that you prefer[2]. Again, I assumed you knew these things, and just asked if you tried them.

How do you respond? "Let's ignore each other." So now I'm left wondering if you genuinely don't know how to scale MySQL, and you've tired yourself of appealing to your own authority in order to prove me wrong. What I'm telling you, is the notion that your blog network creating a workload for MySQL that it is incapable of operating on commodity disk is completely ridiculous, and I'd laugh you out of an interview if I pressed you like this. I think you gave up, but I wasn't saying it, but now that you've gone at me like this, I will. You're basically saying you couldn't make MySQL work with a <50QPS write load (I refuse to believe you're writing more than 50QPS to MySQL) because of some TEXT columns.

I'd have far more respect for you if you'd just say, yeah, I probably could make MySQL keep up with my blog workload, I just didn't put much effort into it and bought SSDs on a provider I don't prefer instead.

(But wait: I don't understand. Username oddly appropriate.)

[1]: http://dev.mysql.com/doc/refman/5.0/en/internal-temporary-ta...

[2]: http://dev.mysql.com/doc/refman/5.0/en/temporary-files.html