[mechanical_fish]

I'm not counting sysadmin salary in that figure though, because I am the sysadmin.

I think your accounting technique needs to be refined.

The time which you routinely spend each month, setting up and updating these systems, is a cost.

When a system goes down and you must drop whatever you're doing to fix it, that context switch is an additional cost. Especially if it costs you a night's sleep. You might lose days of productivity on your regular job.

When something happens that you didn't foresee, and that you have no experience with -- you are, after all, only a part-time sysadmin, not a professional cloud administrator -- and you have to spend days reading manuals and nagging Dell vendors on the phone, that's a cost. One that is easy to overlook in advance (who likes to sit around, dreaming up the improbable but scary ways that your systems might fail?) but which is very real when averaged over time.

And when you get tired of playing sysadmin in your spare time and decide to stop -- perhaps by quitting your job -- replacing you is going to cost your company a fortune, especially if they end up hiring the wrong person, who accidentally corrupts two years' worth of backups and then drops the main database. The thing about paying (e.g.) Amazon is that it isn't just a way of hiring a competent part-time sysadmin. It's also an algorithm for hiring a series of future part-time sysadmins: As long as you keep paying Amazon, they will probably keep finding and hiring the right people. (They are, after all, more experienced at finding good sysadmins than you are.) Or, if they fail at that task, perhaps one of their thousands of other customers will discover that fact before you have to, and you'll hear the news early enough to make a clean and event-free migration to another provider.

[aaronblohowiak]

Paying Amazon does not negate the need for a sysadmin!! Someone still needs to manage all of the software, os, networking and upgrade stuff. Only the hardware part is taken care of for you (which is a relatively small amount.)

[mechanical_fish]

Paying Amazon does not negate the need for a sysadmin!!

I didn't mean to suggest that it did. Administering Amazon is a big part of my job at the moment. It's a lot of work.

Bare-bones Amazon is a poor example. I should have picked, e.g., Engine Yard: someone at least one step farther up the food chain.

And, even then, I'm not suggesting that hiring a cloud host will make these concerns magically go away. I'm just suggesting that any argument which assumes that these concerns have magically gone away -- rather than specifically addressing them -- is leaving something out.

Meanwhile... it's a mistake to say that Amazon takes care of "only the hardware part". That's like claiming that a car company only takes care of "the hardware part" because you still have to learn how to drive in order to use their product.

Among other things, Amazon provides an abstract interface for provisioning hardware. They provide a standard platform that lots of people know how to use, so that you can install pre-configured AMIs, follow canned recipes, run third-party utilities for managing EC2 hardware, ask questions on Stack Overflow, or hire people who are guaranteed to have experience with your exact platform. And they provide the ability to rent variable amounts of hardware by the minute.

One thing which I expect to see on the scene any minute now are third-party hosting providers who provide similar services to Amazon with a compatible API, so that tools used for managing EC2 instances can be run transparently against the new host.

[aaronblohowiak]

You're right in that they take care of the hardware part and the hardware provisioning part.

Even with EY you have to administer anything non-standard yourself (at least that was my experience with a former employer.)

I am saying that Amazon takes care of "only the hardware part" in the same way that a trucking company might lease its trucks. Sure, maintenance, part fulfillment, repair and truck acquisition are taken care of by the leasing company, but the bulk of the business is in the logistics and operations.

I am pro-EC2 for a large class of problems, and just recommended cloudfront/s3 to a friend last night. But some people have this fantasy scenario where the cloud is a magical place where applications can live and scale horizonatlly, and the reality is that Amazon offers a very good starting ground on which to build and administer your systems. (As you know.)

I think we're largely in agreement. I apologize for minimizing the amount of legwork that Amazon takes care of for you in an attempt to stress the amount of work that still remains.

[rjurney]

This interchange is the model of informative civil discourse.

[Periodic]

I've seen a few people hand wave around the costs of sys admins and claim that since they have a bunch of cloud-based systems they don't have to worry about it. Then a few days later they complain about getting the APIs to work and their custom software they had to write to store things to S3 instead of local disk.

It's a hard trade off to make and I would suggest the following: Go where your strengths are and outsource where they aren't. You can generally save money by doing something yourself, all other things being equal, but you'll generally lose money if you don't know what you're doing and could have hired a professional.

[antonovka]

In the ideal model, you:

1) Have an automated tool to build OS images.

2) Configure the OS image to automatically fetch and install the software/configuration required for your deployment.

3) Simply reboot instances to upgrade them.

At that point, your need for a sysadmin is very low -- especially if you use software suites that are standalone, rather than requiring extensive OS integration.

Instead of hiring operations staff, you've invested in the software necessary to pull off the above.

[nkassis]

I would consider those task something a good sysadmin would be great at doing. Those scripts will need modification and I think that is what a good sysadmin should be able to handle. Sysadmin are automation engineers and infrastructure engineers. Maybe this person is more the one who integrates the pieces together in your model.

Sysadmins deal with more than just hardware and pressing go on the upgrade script.

[delano]

True, hardware support generally doesn't require ongoing effort like software does but when there are problems than can take a HUGE amount of time (as mechanical_fish highlighted).

The other advantage Amazon provides is the API which allows you to automate most deployment and configuration processes. You're then left with the stuff you can't automate and the occasional software bugs and other issues that crop up from time to time.