This is a strategic attack on Google. A proliferation of scalable data centers hurts Google a lot more than Facebook by enabling Google's competitors. Cheap computation matters much more to search engines than social networking sites.
Because Google could never benefit from Facebook's designs too? Google does a lot of things bespoke, but now they have a model to compare with that is using more off-the-shelf tech. That can only help them in the buy-vs-build decision process.
I think you're going overboard in thinking that everything the big players do is about killing the other guy. Sometimes they just want to reduce costs. If you decide that part of your infrastructure isn't strategic to own, it always makes sense to be open.
A proliferation of scalable data centers could also enable the next generation of social networking sites ... I don't see why it's limited to helping Google's competitors and not Facebook's.
Becuase Search is ultimately about technology, but Social is about the users.
If I was the only user of Google (and they had the same technology stack) then +1 aside - my experience would be the same as it is now. If I was the only user of facebook - what would be the point.
I'm sorry, but it seems like you're implying that search doesn't need users. Some of Google's ranking depends on analyzing search histories, so if you were the only user, the experience would be worse. That means that search is also about the users.
Data center technology can benefit everybody, that's all I'm trying to say.
>Google's ranking depends on analyzing search histories, so if you were the only user, the experience would be worse.
I disagree. At this point, I believe that because of how big google is and because of how profitable gaming google is, getting good results, I believe that more than half of the work of finding relevant search results is filtering out the people attempting to game the search engine.
I mean, it is /possible/ that the help they get from user's search histories outweighs the massive amount of effort spammers put in to making google suck, but I doubt it.
I bet that things are much easier for a new search engine while they are small enough that it's not worth much time and effort to game the results.
>Data center technology can benefit everybody, that's all I'm trying to say.
This is true, but I believe what other users are likely saying is that google probably alrealdy has something as good or better than this.
Personally, I won't get excited until someone starts building out co-location centers that have efficent cooling. Nearly all co-location centers use mixed air, and most end up keeping the place much cooler than required, and they pass on those costs to me.
The thing is, for anyone who is big enough to take advantage of their data center design, this isn't particularly new. I mean, it's great they are sharing the documentation of how they did it... but that mostly helps the little guys, and the little guys can't really take advantage of it... we're stuck in co-location centers, like I said, with antiquated cooling designs.
The chassis designs, on the other hand, might be great for the little guys. Hell, I am not that far away from enough scale to job out the chassis to a metal shop, and the design would be a pretty big part of that cost, that facebook just reduced to zero. Thanks!
But the power savings on the chassis are very small compared to the power savings on the cooling/data center design.
One of the most important fact form this article which is likely to be overlooked is : "the Prineville facility runs at 85°F with a 65 percent relative humidity". Running at such a high humidity + Temp combination is impressive esp when you look at the money saved in environmental costs.
Any idea on how much would it cost to produce a small quantity of those boards? At which scale does it starts to make sense to use a custom made motherboard instead of a off-the-shelf PC?
One thing to keep in mind is that one rack holds 90 servers and each battery rack feeds 180 servers, so (at least in Facebook's case) that's the basic unit. If this stuff takes off I would expect someone to produce a smaller 30-slot rack, though.
A list of Facebook's open-sourced applications and their major contributions to other open-source projects can be found here: http://developers.facebook.com/opensource/
Indeed. This is how I want to see society. Sharing knowledge for the greater good. Probably the reason I'm not a fan of detriments of this such as software patents and Apple in general.
Have ARM actually done more than announce that they will be moving into servers? If they have then I missed the announcement. Either way, seems like a fairly stupid dig at ARM, can't really expect companies like Facebook to have moved onto ARM servers this quickly, even if it is the direction they intend to go in.
This "sorry ARM" stuff is mostly just the media's need to turn every story into a horse race. But there is some technical detail here: Facebook said they're not ever willing to use 32-bit and ARM won't have 64-bit for years. Also notice that Facebook's servers are "conventional" 2-socket with lots of DIMM slots, not 1-socket "microservers".
Fwiw, Freescale has a 64bit ppc processor now... Either way though, until you're able to just drop an ARM processor into a "commodity" motherboard, it won't really see widespread use, aside from cell phones, and other mobile devices. There is also the fact that ARM processors all seem to have the graphics card embedded in the chip.
Current ARM SoCs have all been built for embedded applications, which is why they integrate things like GPUs and radios.
The next-generation high end ARM core, the Cortex A15, will be the first one that's intended to be suitable for server use: they've added virtualization support, PAE for up to 1TB of RAM, and cache coherency to their bus to support SMP. The server-oriented implementations will be quad-core 2.5Ghz chips, of which you would be able to put at least 4 on a motherboard like this. When they start hitting the market next year, they'll probably all but kill the market for Intel Atom-based servers.
QorIQ is pretty cool, but for whatever reason there's massive hype about ARM servers and zero about PowerPC. Perhaps this is evidence of an information cascade.
If Facebook is already getting custom Intel and AMD motherboards, ARM motherboards should be no problem. (Note that a major undercurrent of Facebook's announcement is that "custom" is actually cheaper than "commodity" at "Web scale".) ARM server chips (Armada XP and Calxeda) don't have GPUs.
Why would you want to saddle an ARM processor with all the IBM PC 5150 compatibility that lives in every "commodity" motherboard? What would be gained from obliging ARM processors to have the same external signals as x86's? How would they guess how to properly initialize the various components (something processors depend on processor-specific BIOSes and EFI to do)?
No they haven't. The closest you would come would be a tegra2 which is dual core and I believe the omap4 is dual core as well. Though iirc, you can't currently use both cores in Linux. Freescale have announced the imx6 which should have quad, dual and single cores but availability won't be until q4 (though they are pushing hard to get it before then I think.) You still are currently limited with ram on an ARM machine which I think is the biggest hinderance to going in to servers. Presently almost all server software works on an ARM machine with a few exceptions like spidermonkey (so no running Launchpad) but plenty of people use them as personal servers.
I'd be very interested in hearing from someone with more experience in running their own hardware what portions (if any) of what Facebook has announced today is applicable at the small scale of say having a couple of co-located racks in some datacenter. Maybe the base server designs?
I'd say it's more of a refinement than something new. I haven't looked at much more than what they're saying in the video, but it boils down to:
- new featureset mobos: stripped down, no parallel ports among other things I'm sure. there looked to be some SATA/Mini-USB type sockets.
- fancy new power supplies
- the triple racks don't seem to be anything special at face value, convenient maybe, but with the power features above two of these triple racks are serviced by one battery rack. So, you'd need, oh, 400+ servers to take full advantage at this level.
NOW HOW MUCH WOULD YOU PAY?
- Fancy power distribution and cabling and stuff, mostly part of the racks, possibly involving wacky connectors on the compute side. For smaller operations this would be like having a 60-outlet power strip.
- 95% efficient from plug to party-time. This helps with the bills.
- Using the room/building as the cooling mechanism. While this has likely been done before in some manner (at least an intentional use of stack-effect in HVAC), that weird slot that one of the boxes gets pushed into makes me think they have some kind of sealed thingy from floor to ceiling that interfaces with the racks, basically using the building cooling to push in and suck out air forced through the racks. at a basic level it's all about CFMs, after all. This could also be done with forced current from front to back, or vice versa.
whatever is going on here, you first have to get to the "your own server room" part of business. this can be had at smaller companies, too, but for the room-cool ducting you'd need to cut into walls and stuff in the server room to pipe that stuff in. Spendy.
So, this leaves the smalltimer to save $15 on their monthly power bill by using new rack servers that don't need or have an NVidia GeForce 9000+ and 10-drive RAID. Stripped down BIOS and maybe no more IDE support, that kind of thing. Hot Rod rack servers of the skeleton/pure-compute variety, not the AlienWare one.
I can give you an interesting data point: our power bill is only about $600/mo (2.5KWh), our bandwidth bill is about $3,500/mo (1.2Gbps). So power is a smaller consideration.
I don't get the AC PSUs on the servers (http://opencompute.org/servers/). Any reason why those are being used when each 6-rack group has a UPS (and batteries) connected to them? Going DC would get rid of the inverter on the UPS out and the 200 or so PSUs on each server.
Amir Michael who worked on the hardware team said, "The utility provides us with AC voltage which we then convert to DC very close to the motherboard in the power supply. Our goal was to bring the high voltage as close as possible to the load to minimize IR losses. We could do an efficient AC –DC conversion in the rack and then distribute DC to the individual servers but that would mean several feet of low voltage conductors which would be made from large copper bars and a would have higher IR losses."
I guess in their instance the lifetime of the server was never likely to be eclipsed by that of the battery. In telco, 15-20yrs of operation is not uncommon, so a single set of replaceable/easily service batteries is important.
> In August 2008, Intel conducted a 10-month study to assess the effectiveness of using only outside air to cool a data center. The temperature range was 64°F to 92°F. Humidity varied from 4% to over 90% and changed rapidly at times. No increase in server failure was observed.
A certain amount of humidity helps to prevent issues with static electricity. And unless you're pouring water directly across electrical components, there's really not a problem.
I don't know whether 65% humidity is going to hurt a server, but evaporative cooling doesn't work well if the relative humidity is high. So maybe it's a good idea in dry climates, but I can't see it being very effective in the midwest of the US in July/August.
What are the chances of these being commercially available anytime soon? Is a single consumer (even if it's facebook-sized) enough to jumpstart a B2C supply chain? It's hard to see this catching on with anyone not building green field unless it can come in at least on-par with a Dell/HP/Supermicro quote.
I found that interesting too, thought it was a joke at first.
The only details I was able to dig up are in the Data Center specs [1], and they're pretty brief:
> Energy-efficient LED lighting is used throughout the data center interior. / Innovative power over Ethernet LED lighting system. / Each fixture has an occupancy sensor with manual override. / Programmable alerts via flashing LEDs.
I wondered what the justification for PoE lighting could possibly be, sounds like all the lighting is also functional as instrumentation.
All true. Right now 14/3 Romex costs about 3x more than Cat5. Copper is expensive!
LED lamps use DC power. The LED lamps that you can buy that screw into a standard Edison socket contain electronics to convert the AC line voltage into DC. This is inefficient -- generates heat and wastes power (though still not nearly as much as a traditional incandescent bulb).
By using power over Ethernet for their lighting, the datacenter can use cheaper, cooler, more efficient bulbs, and save a lot of money on the wiring too.
I can definitely foresee a future where new construction includes wiring for both line voltage AC and also low voltage DC. It could eliminate all the bulky transformers scattered around a typical house and save energy and money.
Not sure how many startups built their own servers anymore, this event seems like a response to the Greenpeace accusations of Facebook's environmental responsibility or wherever.
Edit: I agree it's a good thing, it's just that hosting a press event rather than just making the announcement through a blog post suggests other motives as well.
This isn't a response to Greenpeace, this is a response to the real costs that it takes to run compute infrastructure at the scale of Facebook, Amazon, Google, etc.
If a datacenter can run at 1.07 PUE (as Facebook's new datacenter does), then it directly translates to massive power cost savings. Environmental efficiency just comes along with it.
Okay, but why does it benefit Facebook's bottom line (and Google's!) to open-source that stuff? Wouldn't they be better off if their competitors had to pay those higher costs? It's not as if they have substantial computation-service providers they're buying from, or data-center-oriented companies they're selling to.
The upside is that FB can tweak Google's nose (who has a very high opinion internally of how good they are at operational efficiency) and interest some of Google's operations folks. They get to dispell some of the mystery and disbelief at Google's efficiency claims. If Microsoft, Apple, or Amazon hasn't yet figured out 21st century data centers they give them a big eye opener so that those guys are more competitive against Google as well. (there is a reason Google abandoned shipping containers :-)
The more things they can throw at Google, especially now as the organization is digesting the CEO transition is nothing but win for Facebook.
They are hoping that the same philosophies which apply to open source software will benefit Facebook providing open source hardware. They mention a few goals they have during the talk. Mainly that they hope people will offer criticism and help them improve the technology. Facebook's core competency is not hardware so any improvements to the hardware sector directly benefit Facebook's bottom-line.
Facebook's core advantage is the social graph, not the infrastructure operation, which is Google's thing. Consequently, Facebook has tight grip on the access of its graph whereas Google keeps the secrets about its search algorithm and server setup. By open-source it, Facebook is not only getting good PR but also undercutting the competitor.
This is a nice benefit of two companies competing asynchronously (each has a different core competency). The open source community gets the non-core competency of both.
Now if only there was a company whose non-core competency was Google-style spelling check ("Did you mean to search for..")
To get custom computing gear built, you have to show your designs and plans and requirements to various potential vendors, so (and NDA or not) the vendors now know what you're doing.
If FB can cause these servers to be offered as standard configurations and produced in (much) higher volumes than when produced as custom solutions built for for FB (or Google), then the costs to FB drop, and FB can potentially also get the vendors to compete with each other.
It's a clever move.
(If you're just running big DCs, and not looking to move into the cloud hosting business.)
The only thing I could come up with is that, anyone implementing these at scale (fb's competitors) has already committed substantial dollar amounts to existing infrastructure.
To leverage the minor improvements in design that will come of this, they would have to start by rolling out new facebook style racks/data centers.
Whereas, for facebook any design improvements would be easier to implement on a larger scale.
Either that or they are just do-gooders, which I highly doubt.
And if competitors do adopt FB's infrastructure, they'll essentially be training their engineers to work at Facebook. They'd better have counteroffer packages ready.
Releasing information about this helps others better compete with Facebook's competitors while not diminishing the value of Facebook's core asset at all.
Should the design proliferate into other DCs, it would provide a ready disaster recovery base (perhaps even sharing agreements between different operators).
Was anything mentioned about the OS running the servers ? I presume it is an optimized linux, but who knows.
It's not about the fact that open sourcing this info affects their bottom line as much as it is about responsibly contributing to the progress and advancement of computing... however it should be noted that for years, google has been very secretive about all aspects of its internal DC designs.
The fact they were making their own equipment, incorporating UPS functions in the power supplies of their servers, how they were cooling - the fact they get the cheapest cabling possible etc etc all played to their competitive advantage.
Highly efficient low cost commodity gear results directly in lower TCO / OPEX thus allowing you to do more where it counts.
With the mass defection of high level folks from google to facebook, and with the very agile innovative team that facebook has, it is just expected that strides in these areas would occur there as well.
This is one area where I have a lot of respect for what facebook is doing.
The impact may not be direct to tiny web companies - but organizations that have massive energy costs, specifically hospitals, can greatly benefit from this information.
The problem is though that hospitals are not IT companies, and thus they dont focus on the physical characteristics of their equipment or directly look at the designs of their datacenters as they buy gear from the big names we all know.
If companies like google and facebook work to get the suppliers to incorporate these design elements - it will result in organizations like hospitals benefiting in the long term. This is a good thing.
El Camino Hospital in Mountain View pays over $500K PER MONTH in power. Imagine if they can reduce that by say 20%
The biggest take away I have in looking at this information is the POE LED Lighting. I am really interested in this because of the impact it has on the overall electrical infrastructure in a large building.
Currently, you design your infrastructure so that you have emergency power backing your MPOE, DC and IDFs. This means that the more POE devices you have off the IDF, when utility power goes out, your supporting those devices via the UPS/generator infrastructure.
If you add lighting to this, its going to redistribute costs from the electrical/infrastructure expense to install and power the lighting, to adding a port on the switch and the requisite load on the IT side. I assume that the wattage per lumins could be less - and the overall cost of emergency lighting could be reduced in a very large facility.
This is right in line with the idea of "Technology is a utility" -- the number of devices and range of services we now hang off the network is amazing - lighting is just another example, and this will ultimately simplify and reduce the wiring infrastructure in your building.
This is true. However, the services that startups use to host their servers (Amazon Web Services, Rackspace, etc.) could adopt these practices in new datacenters because the specs are open. This could lower costs for the startups that use these services, and be good for the environment.
Any large scale service already adopted such practices, may be not to the same degree. All large players buy custom hardware with specific tweaks to be more energy (read cost) efficient.
"Graham Weston, the chairman of Rackspace, said that his company would use the new Open Compute servers in its own designs, and Zynga's chief technical officer said that his company would take a serious look at adding the new technology to its own cloud."
http://www.pcmag.com/article2/0,2817,2383257,00.asp
The question comes up a lot when you're building very large scale applications. When does it make sense to have your own infrastructure vs using someone else's (like Amazon's).
If you're running 'one' thing (like say a hadoop farm) and you can optimize out the things you don't need, there can be a pretty durable benefit in building your own machines.
People like Rackable, HP, Dell, or IBM who sell servers need to build them able to do 'anything' you might want, in order to do that cost effectively they often put things on the mother board (lowest marginal cost) which are perhaps not useful in all cases. However, when you're using lots of machines you have to power and cool those unused sound chips and USB hub chips, and may firewire ports that aren't really all that useful to a web app.
I talk about it as 'rack level' blades, basically motherboards on a cookie sheet that only have network and storage interfaces. Taking away a size constraint makes building them a lot easier (you don't need a custom backplane for example, you just plug cables in)
The only other motive it suggests is garnering as much publicity for their innovation/company as they can. Facebook is notorious for being able to magnify a small announcement into a big event and reap the dividends -- remember the announcement of a redesign via 60 minutes? If you can get media and influences to come pay attention to cool stuff you're doing, why wouldn't you?