[mrmondo]

Still slow on the scale of things these days. I just provisioned a few servers with over 500,000 IOP/s / 2000MB/s read and write each, 100% SSD with 3-10 year warranties and they use bugger all power. Very low running cost and maintenance overhead and cost less than 8k a unit (1u chassis, redundant power, 32GB RAM, 2x 6 core Xeon v3) and I can guarantee the performance is consistent and there when We need it.

I'm all for outsourcing hardware hosting ('cloud') to save costs and to allow for quick provisioning of new instances - but went you need raw power and in cases where it's inefficient to horizontally scale - the latest generation of PCIe NVMe SSDs are really very impressive and in a recent evaluation we performed of our storage - it was actually going to work out significantly cheaper to A) host our high speed storage ourselves and B) buy SSDs and do away with rotational drives.

[tgeek]

Can you completely snapshot those volumes at any time, recreate them and attach them to new servers? Could you take these snapshots and easily copy them around the world?(again assuming you could snapshot). Are those SSD's automatically replicated to 2 different storage devices behind the scenes to give you near-instant failover? When they go boom are you then driving out to the datacenter to replace them (assuming you have replacements and don't need to wait for them to arrive). Can you do all of this without any upfront cost or excess in capacity??

Probably not. At all.

EBS is NOT harddisks inside a server. Comparing them to such is missing out on all the things that makes it a SERVICE and not disks you buy from Newegg/PCmall/<insert vendor here>. Yes there are disks you can buy to physically put in a server and they are super blazing fast. In fact AWS has those in their i2 instances and they get hundreds of thousands of IOPs as well.

This isn't even comparing apples to oranges, its apples to space monkeys.

[mrmondo]

Yes we can and do snapshot them, at several levels actually - I don't think that's a particularly hard thing to do so I'm not sure why that's relevant.

Yes there is replication both to separate disk arrays AND seperate physical servers with live failover and load balancing - again nothing new here?

No we don't send out storage to other countries - in fact that would be illegal, and if we were to do so our clients would suffer as Australia's international peering is pretty woeful.

We also gain on-disk compression and encryption on a LUN by LUN basis as we require it, storage is automatically provisioned to new application instances, all the software is 100% open source and mature, we don't have to phone a large corporate that doesn't really care about us, we pass security audits because we can prove where things are and how they're configured.

By the way, none of this is your 'new egg' gear you referenced, we use Intel DC P3600/P3700 PCIe storage. Oh and as a bonus - there's no licensing or monthly invoices that need attention.

Is shared hosting / hardware outsourcing / cloud computing amazing - yes! Of course it is!

But you must remember it is their intentions to sell their product as the only right answer and to tell you what you should care about. In some cases it applies and in some it doesn't. The danger in jumping on the bandwagon and becoming an Amazon 'fanboy' (I'm really sorry for using that term - I hate it) is that you quickly become silod from external opperuntities and security / high vertical performance solutions.

If I was in a small team of devs working on launching a web app that's going to be targeted at an international audience, my growth is highly unpredictable, our future uncertain and our skill set focused on developing great software - I wouldn't think twice about using AWS/Rackspace etc...

But when you understand your environment well, when you have a limited budget, when you have a predicable customer base with strick security requirements and when you're pushing databases pretty hard - would I use AWS? No, it's not cost effective for us, nor is it legally (and perhaps morally) viable. Do we waste lots of time looking after our hardware? No! It's 2015 - hardware is easy.

[tgeek]

You say LUN, are these SAN devices, or is it direct attached storage? Was the replication, load balancing, and snapshotting all something that you set up and manage yourselves?

--edit-- Ahh you've been editing your comments so the thread is a bit out of wack! (no problemo)

Newegg: http://www.newegg.com/Product/Product.aspx?Item=N82E16820167... ;) (yes its not the same as some of the much higher end stuff).

Fair enough, but again, your comment is about hardware that you are managing, that you've built, thats glued together from a lot of different components, both software and hardware, and this post is about a cloud service that doesn't even compare. So your initial post comes off a bit as trolling for the sake of trolling.

I've done my fair share of rack-n-stack, and I've now spent the past few years "in the clouds" as it were. Wouldn't go back for anything, but I dont think this makes me a fanboy. Sure there is kit that you'd only ever be able to build/buy yourself (for now at least), but most ppl will never need more than 100k IOPs, let alone 500k+.

--edit again-- In regards to security, if you think you are a capable of running an infrastructure more secure in a datacenter yourself, than on one of the major 3 cloud provider's infrastructure ( AWS, GOOG, MSFT ) where they have some of the best sec teams in the world, then you are probably not as deeply aware of whats possible in cloud from a security standpoint. Banks, Medical institutions, government agencies, and so forth are all trusting their infrastructure on the cloud, across many countries in the world.

[mrmondo]

Yeah sorry I didn't want it to end up sounding like a threaded argument - and I was sort of brain dumping as I go.

Hardware wise - We use standard servers (super micro), packed with several tiers of SSDs (Intel for the high end, SanDisk for the lower end).

Software wise, again all off the shelf, well understood tools: Debian Linux, DRBD, iSCSI, LACP, LVM, Puppet.

Our compute servers are blades with Debian VMs running Docker containers Of our applications.

Edit: something we've gained greatly from that isn't off the shelf is that we moved to running very modern Linux Kernels - we have CI builds triggered as new stable versions are released and they are stock standard except that we do patch them with GRSecurity and ensure SELinux is enforcing.

All this doesn't cost much time to manage at all - we don't even have a storage admin and to be honest - if we needed one we'd be doing something wrong - apart from physical failure (which is very rare these days) there really isn't anything to do with storage - it's almost boring!

[mrmondo]

I actually have to get some sleep now - it's after 1AM here in Aussie, I wanted to stress that I'm absolutely not against using cloud hosted services - just that they're not the answer to all situations and there's a lot to be gained from ensuring you don't get sucked in to too much of the 'Spin' that vendors provide.

[tgeek]

np! at the end of the day we all hate on call.

cheers.

[AlfonsoP]

> most ppl will never need more than 100k IOPs, let alone 500k+

These types of statements are always false. If there is anything the computing industry has taught us is that people always need more resources. Always.

[tgeek]

I sincerely hope you are joking here.

[AlfonsoP]

I sincerely hope YOU are joking, right?

I can think of many examples why even small businesses need more than 100k IOPS. Case in point: 5 years ago I did consulting work for an email marketing company that was generating a daily report on a database of about 1TB. The report took 10+ hours to generate due to the SQL queries aggregating data from joined tables in more or less random patterns. I upgraded their DB server from a 2-way RAID0 on 15kRPM HDD (about 500 IOPS) to a single SSD (20k IOPS), and it cut down report generation time to 15 minutes. 4 years later their database has continued growing and generation took 1 hour. They called me up again, I upgraded them to a 4-way SSD-based RAID5 (I benchmarked 250k IOPS) and again it cut down report generation to 6-8 minutes. This was a small company: a dozen marketers, 1 software guy.

[mrmondo]

FYI - there are laws around where data is allowed to be hosted and what country that parent company is located in. Even if there weren't laws on this - I think it would be pretty irresponsible to trust all your data and servers to one large off shore corporation, especially one that has a fragile political climate (mind you, want countries don't!)

[istvan__]

Well your security is definitely worse than Amazon's. Please prove me wrong.

http://aws.amazon.com/compliance/

I think the fallacy in you comment is that you think about storage as it was not using energy and network. If add those costs to the bill, are you sure that you are still cheaper? On the other hand, I much rather pay a monthly fee that I can turn off if things go sideways, than buy extremely expensive gear that I cannot get rid off at all.

Just to summarize:

- monthly cost is almost all the time better for small businesses - your security is way worse than Amazon's - the overall cost of your operation has to include electricity and network for the complete comparison

My experience is that companies rarely need expensive network storage gear and most of the time it is better for everybody to split up the problem and make it horizontally scalable. There are also other solutions, using distributed storage engines running on commodity HW. Having said that, there are quite few companies out there with SAN/NAS solutions, because this is what traditional computer vendors were selling for a long time. I think by time we are going to see more horizontally scalable storage solutions going forward.

[vidarh]

Amazon's bandwidth rates are more than 10 times what we can get locally, and power is included in our colo rental fees. I'm assuming that will pretty much be the situation for mrmondo too.

In terms of monthly costs, all the gear I deal with is lease to own: We pay less per month when the servers are new, and 3 years down the line our bills drop. There's no reason to have large capital expenditures just because you want your own gear.

As for security, it's not really that simple. Amazon's physical security may be top notch, and their patching for Xen and network security may be just fine, but beyond that you're pretty much on your own with Amazon just as you are with your own gear. You still need to understand how to configure firewall settings, and understand how to keep your VMs secure. Amazon's security needs to be top notch because it adds an additional layer that you don't have direct control over, but that does not provide any additional security that you would not have in most reasonable colo facilities where the physical network devices past the service providers network drop is totally in your control, in a locked environment.

> My experience is that companies rarely need expensive network storage gear

The thing is, this gear isn't expensive. For about $250/month I can lease to own a 2TB PCIe SSD delivering 2.8GB/s read, 1.9GB/s write, 450k read and 150k write IOPS. That's in the UK, with 20% VAT, and without shopping around. Or if I want something with the performance profile of Amazons new offering, I can pay $25/month. Amazons cheapest EBS offerings, which are nowhere near what this article about, costs $200/month for 2TB space. Go for provisioned IOPS and the EBS cost skyrockets.

AWS is the expensive network storage option, not leasing your own.

I can lease servers to put it in to get me "free" compute capacity for the difference in cost of the raw storage and still have money left over after spares and hosting/power.

Factor in bandwidth and it gets downright comical - Amazons bandwidth prices are so totally out of whack that where for managed/colo setups CDN is an expense, for AWS setups a good CDN can save you vast amounts of money by cutting your bandwidth charges. And that's without discounted rates. Start putting decent volumes through and host at a carrier neutral facility and paying even 1/20'th for bandwidth vs. AWS is well within reach with peering arrangements and a good mix of transit providers.

[mrmondo]

I was going to reply to the parent comment but you've summarised exactly what I would have said - sounds like you've done your research and come to similar conclusions to us as well.

[vidarh]

> Can you do all of this without any upfront cost or excess in capacity??

The price premium of using AWS is high enough that it's trivial to afford leasing tons of excess capacity to handle failures and still save tons of money.

But by and large it's not really necessary - most hosting providers can provide rapidly provisioned managed servers or VPS's in the same data centres as their colo offerings these days, which provides an excellent fallback if we get into capacity issues, meaning that thanks to the existence of cloud services, the cost of running your own base load can be pushed down significantly (everything I deploy is deployed in VMs or containers, and sometimes containers in VMs (don't ask...), and whether they run on our hardware or on a cloud providers hardware is merely a configuration issue.

In fact I have a couple of Xen based VPSs we rented in New Zealand to serve a customer that's tied seamlessly into our UK based infrastructure because it's not somewhere we can justify operating our own setup.

AWS certainly is convenient, but it's also so expensive I'm charging my highest day rates ever for projects to help clients move off AWS these days. It's easy to justify high fees when people see how much they can save.

[tgeek]

"The price premium of using AWS is high enough that it's trivial to afford leasing tons of excess capacity to handle failures and still save tons of money."

"AWS certainly is convenient, but it's also so expensive I'm charging my highest day rates ever for projects to help clients move off AWS these days."

This is FUD and nothing more than FUD. As bad as the article the other day that said a company saved 50% moving off AWS. Show me a company that can save 50% by moving off AWS, and I'll show you a company that isn't using AWS properly at all.

When you compare a single piece of hardware that you can buy and run yourself to an instance in EC2, you are leaving a lot off the equation.

Such as:

Deployment tools (CloudFormation, CodeDeploy, EB, OpsWorks)

Monitoring tools (CloudWatch (sure it leaves some to be desired)

Machine Image building/tracking tools

Hardware provisioning tools

Security tools ( Security Groups, NACL's in VPC, plus the stuff you don't see in the infrastructure)

More varied hardware than you'd have in house.(different amounts of ram/cpu/storage).

More hardware than you'd have in house. (no need for spare parts cabinets, waiting on vendors for replacements).

Storage (EBS, S3, local).

When you pay for EC2, you get all of this. Doing this yourself isn't free in any way shape or form, even with opensource tools. A company with a good ops team that is cloud savy is going to be several times more effective at a smaller size than a team that has to manage a datacenter, all the hardware, and all these other bits. Folks discredit all of these when they do apples to apples comparison of hardware you can buy to a service that you use.

Let alone that most people don't understand per-server network, space, and power cost over its lifecycle. (I've spent months with companies doing datacenter ROI analysis and having no idea what things cost).

And going to a managed hosting provider, you're either locking yourself into the frozen tech world they are in, small development resources, and typically constrained resources. You don't hear of managed hosting providers building any of the above themselves, so you continue to have a higher management overhead than a true cloud provider like GOOG, AWS, MSFT.

[moe]

Show me a company that can save 50% by moving off AWS, and I'll show you a company that isn't using AWS properly at all.

Using AWS "properly" usually means moving most of your stuff off AWS.

AWS has a few very narrow sweet-spots. Your average web-stack is not one of them.

[marktangotango]

Guys this is good information freely given. Please don't down vote frivolously, it discourages this type of sharing.