[_sgo4]

It's even cheaper if you build your servers yourself and colocate them at a datacenter. Of course, not every company has a sysadmin, but if you do, you will save a bundle. This also means supporting them yourself, though, which at times can suck a fat one.

At Clicky, we have almost 30 servers, all of which we built ourselves. Up front cost is about $1500 per server, and monthly cost is $75 each. If we were leasing servers of similar capacity (RAID, 8GB+ RAM each) they would cost us a minimum of $400 per server per month, which would total close to $12,000 per month. By doing stuff ourselves we instead pay around $3500 per month (~$2000 hosting, and approx. 1 new server installed per month). That's serious money in the bank. I'm not counting sysadmin salary in that figure though, because I am the sysadmin. Still, if you have the capability, it's worth it.

[mechanical_fish]

I'm not counting sysadmin salary in that figure though, because I am the sysadmin.

I think your accounting technique needs to be refined.

The time which you routinely spend each month, setting up and updating these systems, is a cost.

When a system goes down and you must drop whatever you're doing to fix it, that context switch is an additional cost. Especially if it costs you a night's sleep. You might lose days of productivity on your regular job.

When something happens that you didn't foresee, and that you have no experience with -- you are, after all, only a part-time sysadmin, not a professional cloud administrator -- and you have to spend days reading manuals and nagging Dell vendors on the phone, that's a cost. One that is easy to overlook in advance (who likes to sit around, dreaming up the improbable but scary ways that your systems might fail?) but which is very real when averaged over time.

And when you get tired of playing sysadmin in your spare time and decide to stop -- perhaps by quitting your job -- replacing you is going to cost your company a fortune, especially if they end up hiring the wrong person, who accidentally corrupts two years' worth of backups and then drops the main database. The thing about paying (e.g.) Amazon is that it isn't just a way of hiring a competent part-time sysadmin. It's also an algorithm for hiring a series of future part-time sysadmins: As long as you keep paying Amazon, they will probably keep finding and hiring the right people. (They are, after all, more experienced at finding good sysadmins than you are.) Or, if they fail at that task, perhaps one of their thousands of other customers will discover that fact before you have to, and you'll hear the news early enough to make a clean and event-free migration to another provider.

[aaronblohowiak]

Paying Amazon does not negate the need for a sysadmin!! Someone still needs to manage all of the software, os, networking and upgrade stuff. Only the hardware part is taken care of for you (which is a relatively small amount.)

[mechanical_fish]

Paying Amazon does not negate the need for a sysadmin!!

I didn't mean to suggest that it did. Administering Amazon is a big part of my job at the moment. It's a lot of work.

Bare-bones Amazon is a poor example. I should have picked, e.g., Engine Yard: someone at least one step farther up the food chain.

And, even then, I'm not suggesting that hiring a cloud host will make these concerns magically go away. I'm just suggesting that any argument which assumes that these concerns have magically gone away -- rather than specifically addressing them -- is leaving something out.

Meanwhile... it's a mistake to say that Amazon takes care of "only the hardware part". That's like claiming that a car company only takes care of "the hardware part" because you still have to learn how to drive in order to use their product.

Among other things, Amazon provides an abstract interface for provisioning hardware. They provide a standard platform that lots of people know how to use, so that you can install pre-configured AMIs, follow canned recipes, run third-party utilities for managing EC2 hardware, ask questions on Stack Overflow, or hire people who are guaranteed to have experience with your exact platform. And they provide the ability to rent variable amounts of hardware by the minute.

One thing which I expect to see on the scene any minute now are third-party hosting providers who provide similar services to Amazon with a compatible API, so that tools used for managing EC2 instances can be run transparently against the new host.

[aaronblohowiak]

You're right in that they take care of the hardware part and the hardware provisioning part.

Even with EY you have to administer anything non-standard yourself (at least that was my experience with a former employer.)

I am saying that Amazon takes care of "only the hardware part" in the same way that a trucking company might lease its trucks. Sure, maintenance, part fulfillment, repair and truck acquisition are taken care of by the leasing company, but the bulk of the business is in the logistics and operations.

I am pro-EC2 for a large class of problems, and just recommended cloudfront/s3 to a friend last night. But some people have this fantasy scenario where the cloud is a magical place where applications can live and scale horizonatlly, and the reality is that Amazon offers a very good starting ground on which to build and administer your systems. (As you know.)

I think we're largely in agreement. I apologize for minimizing the amount of legwork that Amazon takes care of for you in an attempt to stress the amount of work that still remains.

[rjurney]

This interchange is the model of informative civil discourse.

[Periodic]

I've seen a few people hand wave around the costs of sys admins and claim that since they have a bunch of cloud-based systems they don't have to worry about it. Then a few days later they complain about getting the APIs to work and their custom software they had to write to store things to S3 instead of local disk.

It's a hard trade off to make and I would suggest the following: Go where your strengths are and outsource where they aren't. You can generally save money by doing something yourself, all other things being equal, but you'll generally lose money if you don't know what you're doing and could have hired a professional.

[antonovka]

In the ideal model, you:

1) Have an automated tool to build OS images.

2) Configure the OS image to automatically fetch and install the software/configuration required for your deployment.

3) Simply reboot instances to upgrade them.

At that point, your need for a sysadmin is very low -- especially if you use software suites that are standalone, rather than requiring extensive OS integration.

Instead of hiring operations staff, you've invested in the software necessary to pull off the above.

[nkassis]

I would consider those task something a good sysadmin would be great at doing. Those scripts will need modification and I think that is what a good sysadmin should be able to handle. Sysadmin are automation engineers and infrastructure engineers. Maybe this person is more the one who integrates the pieces together in your model.

Sysadmins deal with more than just hardware and pressing go on the upgrade script.

[delano]

True, hardware support generally doesn't require ongoing effort like software does but when there are problems than can take a HUGE amount of time (as mechanical_fish highlighted).

The other advantage Amazon provides is the API which allows you to automate most deployment and configuration processes. You're then left with the stuff you can't automate and the occasional software bugs and other issues that crop up from time to time.

[lsc]

are you sure that renting a 8GiB single socket server would cost $400 a month? I wasn't able to rent any at half that price. Hell, I haven't been able to rent 8 core 32GiB ram servers for that much. http://book.xen.prgmr.com/mediawiki/index.php/Servers_of_opp... - not one bite at that price. (I've rented a bunch at high setup/low monthly deals... but I'm not doing that anymore. It makes more sense to split them up and sell them as virtual private servers)

[dejb]

It might be more marketing and product related in your case. Also when people get to the $400 level they probably want to know that there is someone they can call you can physically access the server 24/7. It isn't clear that this is the case from your material.

I'm a VPS customer by the way. Reliable service from what I've seen but I found it a little too 'bared bones'. Wasn't expecting to have to install and configure Apache, Mysql etc. Maybe I chose the wrong image or something. Great for an existing or aspiring sysadmin but too tough for me. But I'm tempted to hold onto the VPS 'just in case' at those prices.

[lsc]

ah. thanks.

Yeah. My setup is very bare bones, and you do need a SysAdmin.

I am focusing on creating a flexible and inexpensive system where it is possible to do everything you want to do, before I'm going to spend time making a system that makes it easier to do specific things. For instance, I let you run any paravirtualized kernel you like, and you can change kernels without my help. (well, any kernel that works with pvgrub. I've not gotten OpenSolaris working with PVGRUB, though it seems to work on my systems with PyGRUB. )

Yeah, there are many customers who want an easy web-based control panel, and they are better off with linode or slicehost. There are many of us who prefer the command line, though. Scaling a support organization is very difficult, and it's much cheaper, easier, less stressful and more fun to support people who are willing to figure things out for themselves, so why not target those people, and pass on the savings? I'm ramen profitable now at 600 customers; if I double in size again, I'll be edging into the 'more money than I could hope to make from a salary job' range. So I don't need to target everyone. I'm happy in my niche.

I'm trying to get something together so that you can build images and share them, like you can for ec2. (only I'm trying to do it with kickstart-like systems, rather than with images, which I think mitigates most of the trust issues. Usually kickstart-like systems result in much 'cleaner' systems; systems that are easier to upgrade.)

Personally, I think the biggest weakness of prgmr.com right now is provisioning delay. I need to automate that (and really have no excuse for not doing so before now) as part of that, I need to setup a system where users can do automated 'network installs' of many systems.

As for support, yea most of my competitors have better staff hours and better response times than I do. But then, I think I can say that I'm a little bit more experienced than most front-line tech support people. This is the usual small company/large company tradeoff. (and yes, I sleep about 30 minutes from the data center, but hey, you have a serial console, so I expect you to be able to figure most things out yourself.)

[dejb]

> There are many of us who prefer the command line, though.

I prefer it too. But I never realised how much work setting up the LAMP stack could be. I must admit I was trying to set it up with php4 for a legacy system so that probable made it harder than it needed to be. I recon a default LAMP install would be a good option. But I can understand the motivations for wanting to scare off the cpanel crowd who will most likely demand 'managed server' type support and then badmouth you for not providing it.

Apart from automating provisioning you should have an automatic payment option as well. I'm sure most hosts would have a large number of 'sleeper' customers who can't be bothered to get around to cancelling their account even though they don't really need it. So apart from the time saved with chasing payments, you'd have more customers who use little or no resources. Kind of like I might end up being. Now I've got to go and pay your most recent invoice.

[lsc]

supporting legacy systems can be a huge amount of work. If you use whatever all the kids are using, setting up a LAMP system is trivial... seriously, apt-get install php5 should get you something that works in 90% of the cases, and it will be done before you have time to get a cup of coffee.

But yeah. once you want something weird, well, then you've spent the afternoon recompiling source packages and tracking down security patches for obsolete libraries, or dicking around with some half-broken poorly-maintained third party repos. I mean, it's not that hard, if you have experience with autoconf and compiling stuff, but even so it's a few hours of dicking around, and after you are done, you feel like a moron 'cause you just spent three hours installing php. (granted, there may be a good repo with a well-maintained php4 package out there somewhere, but I'm paranoid, and generally am slow to trust 3rd party repos.)

but then, php4 doesn't seem like it should be that weird. In fact, it should be just like apt-get install php4, but I just tried it and it doesn't work.

E: Package php4 has no installation candidate

so yeah. this is part of why I hate languages like python that aggressively break things that used to work when you upgrade. From what I remember of janitoring php, running php4 code on php5 usually worked just fine, though.

[lsc]

heh. Yeah. well, my billing system is in no shape to do any sort of 'pull' based billing. using paypal helps keeps me out of a dreamhost-style billing disaster where I bill everyone for the next year all at once.

I will set something up with paypal recurring payments, though, for those who want it. But I like to think that the people using my service are getting something out of it, you know?

[mtw]

also the good thing about getting your own servers is having hardware options (like getting SSD drives instead of the regular ones, having a video card (if you do media transcoding), etc.)

[jawngee]

You don't need a video card for media transcoding.

[listic]

But video transcoding can be faster by a factor of several, if done on a GPU. That's a big deal for those who do video transcoding all the time.

The irony is that GPU is a more powerful processor than CPU, just not as versatile and easy to use (only a small subset of programming tools can use it yet) but it changes.

[jawngee]

Show me an open source encoder with GPU support. From my research, when you are doing high quality encodings the speed difference is negligible at best.

Also, at web scale encoding time isn't nearly as important as the ability to go parallel which is significantly cheaper and easier to scale using a bunch of EC2 instances.

We process 2-300 videos a month, though during competitions we process 10x that in a month. All of the bandwidth, storage, processing is less than $10K a year with an average encoding FPS of 33fps.

I would love pointers to sources if this isn't the case and I can do this for massify cheaper. Would be appreciated.

EDIT: We also encode 2-3 different versions of the video at varying sizes and bitrates.

[adnam]

> Show me an open source encoder with GPU support

Have you tried Fluendo? I think it has GPU support to a certain extent. Not sure if this is a help, but here you go:

https://code.fluendo.com/pigment/trac/wiki/GPUSupport

[mey]

Lookup CUDA

For example http://www.nvidia.com/object/cuda_home.html#state=detailsOpe...

[jawngee]

There aren't any open source CUDA apps that do transcoding, so far they are all Windows apps. And non-scriptable ones at that, which rules out web scale.

[Periodic]

But it can make it much faster. Video cards optimized for a lot of those video-related operations.

[pyre]

Well, as long as you're making less than $102k/year as a sysadmin for them, they are still saving money on the monthly bill.

[jhancock]

sysadmin duties is not always a full-time job. I have a single server in production (4 cores, 8GB, RAID-1). I own it and pay a reasonably-low monthly fee to run it at a good data center through a small provider that has solid skills to back me up if I run into something out of my league.

I spent some time installing and configuring it. I documented my install very well. I rarely touch it now. Backups are automated. It runs smooth. If I do have trouble, I have my docs to go back to. Most likely any trouble will be hardware failure. That's the biggest risk of running your own server and only having one. So far, its been 4 years in operation and has more than paid for itself. I have horsepower, memory, and disk space to spare.

At this point, if I had a hardware failure expensive enough that caused me to want to toss the server and start over, I could be back up on something new in a day thanks to my trusty docs. So far, 4 years, zero downtime!! The last year, I spend about 2 hours a month doing sysadmin stuff.

[lsc]

where are you hosted?

Also, at 8GiB each, I assume you are using single-socket servers. How come you aren't using dual-socket servers? my experience has been that it's cheaper per gigabyte ram, per watt power, and per core.