[sailormoon]

I really do not dig this tone. The guy is obviously not a system admin. He paid top dollar for rackspace managed hosting precisely so he wouldn't have to do the kinds of things you mention.

"You can't outsource responsibility" is utter nonsense. It is completely impossible to "own" responsibility for everything important in a complex society. Meaningless platitudes should not distract from the fact - Rackspace did not do their job.

Yes, he messed up. He messed up by making assumptions and not checking Rackspace's work more closely. That's not the same as messing up in your own work. His post is a reminder to be more careful checking on the work of your "upstream". There's no need to pile on with the "if you didn't know 'top 1' you shouldn't be running a startup!" etc.

[thaumaturgy]

I'm not a big fan of the tone, either, however, jacquesm is spot-on in his assessment.

For one thing, my understanding of Rackspace's business practices -- and I've only dealt with them peripherally, so I might be a bit wrong here -- is that they "manage" things like their network, and the actual server hardware, and stuff like that. So, if you want a CPU upgrade, sure, they'll do that. If you need your server rebooted, they'll do that too. But, they don't have anyone sitting there monitoring your system's performance metrics and doing your sysadmin duties for you.

The way I read it, Rackspace did do their job: they upgraded the hardware. It was up to the server admin -- not Rackspace -- to check that the software was then configured correctly.

And finally, I don't generally agree with statements of the form, "If you don't know X, you shouldn't be doing Y", but ... looking at dmesg and top are both really, really, really standard sysadmin operations. Entry level stuff, really. Sysadmin work doesn't just mean messing around with Apache's configuration; there are many more nuances, and it's likely that their system is vulnerable to problems that they don't even know about.

[jacquesm]

The tone is probably in large part because the OP does not take any responsibility for his own part in this and instead is pointing his finger at a third party that may have been partially at fault. But that is by no means sure.

This is typical with what I think is a real problem in society, the 'externalization of blame'.

Inability to see your own responsibility is a serious issue, and it is really pervasive. If I were in the OPs position I would be headbutting a piece of concrete for 20 minutes to make sure I never ever make a mistake like that again, and I would thank rackspace for finally finding the fault that I could have noticed in 5 minutes two years ago.

That's why you have post-delivery checklists, burn in tools and inventory management, staples of everybody that has # on machines that do customer work.

I'll try to keep my 'tone' better under control, apologies for that.

At least it wasn't in Dutch ;)

[inaka]

probably not a good idea to comment authoritatively on a company you haven't worked with, but no, kernel management is part of rackspace's job. performance and monitoring is part of their job. they have an SLA and this is absolutely part of it...

[jacquesm]

I didn't say he shouldn't be running a startup, I said he should not be managing the servers their customers stuff runs on.

As for the tone, you may disagree with that but that does not distract from the fact that if you operate a business, that you should know your stuff.

And if you outsource something you should at least know how to check up on the bits that you've outsourced.

Outsourcing does not mean that your responsibility disappears, it simply changes from 'doing' to 'monitoring'.

Maybe rackspace did not do their job, I have no insight in the communications that went on between the party involved and rackspace.

All we get here is a pointing finger without any responsibility taken, that is not a realistic picture.

It could be the difference in the wording of the upgrade request ("please install another CPU in our machine" vs "please install and configure another CPU in our machine").

Even then, rackspace probably should get part of the blame, but really not all of it. The fact that the situation persisted for two years is completely on the OPs account, in two years you have many more opportunities than your hosting provider to find this out, after all they will leave your machine alone unless it malfunctions and there is no indication that they ever were requested to look in to this, and when they were they actually found the problem.

I quote from the article "In investigating an unrelated issue, we followed up with Rackspace on a Kernel patch that couldn’t be applied to our server. One of the technicians immediately realized why – we were not running the SMP kernel."

How come someone is trying to patch a kernel, can't apply the patch and then still doesn't clue in to the situation ?

Also, we do not know if the SMP kernel was installed or not, it might have been, and then on the final reboot the wrong kernel was brought up. And that's a very easy mistake to make.

But dmesg would tell you in a heartbeat, as would 'top '1'', which you would be using plenty of times while debugging performance issues to make sure all your cores are doing the right amount of work.

[sailormoon]

"I said he should not be managing the servers their customers stuff runs on."

And what if it's only him? No go then huh?

You've been saying a lot of this kind of thing lately. That guy before with the App Store payment problem? You came down on him like a ton of bricks. And now this. Just because people haven't dotted every i and crossed every t. It's not exactly the hacker mentality is it?

[jacquesm]

Then he could hire a part-time sysadmin, there are plenty of those looking for work. I figure for $200 / month he can switch to a similar powered dedicated server with a competitor and pay a guy for 4 hours worth of real hands on sysadmin time every month. That way he pays roughly the same and comes out ahead in every way.

Managing UNIX systems that have to perform well under load takes quite a bit of knowledge. Sure, everybody can install 'ubuntu', 'redhat', 'gentoo' or whatever flavor is popular this week. But that does not make you a system administrator. I wouldn't trust myself with my customers machines either, simply because to stay up-to-date on all the holes in all the packages that you may have installed and keeping them patched is real work.

I don't think I came down on the app-store guy 'like a ton of bricks', in fact I gave what I thought was pretty sensible advice and offered (after Sam Odio did) to help him out.

But it's essentially the same problem as what is happening here, blame company X because of something that you caused yourself.

The app store guy:

  - quit job before having money in the bank
  - set up overly complicated corporate structure to avoid non-existent liability

This guy:

  - take responsibility for a part of the operation that he's not qualified to do
  - keep on messing for two years without calling in outside help (sure, it will cost)

And both of them point the finger at another party.

So maybe that's why it seems to you that this is a 'lot of this kind of thing'.

As for whether or not it is the hacker mentality is not my thing, I call it as I see it.

I've had people here rip me to bits for making a stupid remark (and rightly so), if you can dish it 'Rackspace is at fault because they don't know how to upgrade a cpu' or 'Apple is at fault because they don't pay me' then you should be able to take it.

[sailormoon]

Since it's basically personal I'll take it to e-mail if that's OK with you.

[jacquesm]

Anytime!

[dagw]

It all comes down to what I paid for. Check your contracts carefully.

Some places basically just give you a computer and make sure that it always has power and network and that the hard drive is backed up, and everything after that is up to you. Other places give you 24 hour sys-admins that continuously monitor everything and basically manage every aspect of your server. The former obviously costs a lot less than the latter.

It's perfectly OK to outsource responsibility, but you've got to pay top dollar for someone to take on that responsibility. You cannot go for the cheap option and expect all the services offered by the expensive option.