[larrys]

"As a respected hosting provider, I hope they do the correct thing and refund me for this liability due to their error. Many people trust Linode, and they have proven themselves as a serious contender for hosting critical sensitive operations on the internet. I would hate to not see them live up to that reputation."

"hosting critical sensitive operations" in particular. If you are doing "critical sensitive operations" you need a more secure solution and process which will cost you more money.

Under no circumstances can a hosting provider assume the liability for something like this.

The tradeoff you make for the low cost you pay is that you might have an issue like this because someone screws up.

You pay more for a safe to store your money (and for a safe deposit box to store your valuables) because it's important and you understand the risk involved in not doing that. If you have valuable jewelry many times the insurance company will only insure if you keep it in the safe when you are not wearing it and even the amount of days is specified when it can be out of the safe.

It's unreasonable to expect (and linode's contract clearly states as other's have mentioned) a hosting provider to have a liability over what you are paying them. Edit Add: Unless you specifically have an agreement in advance or that is what they promised or charged you for.

Before anyone reacts to this with any harsh criticism please think for a second what liability you would want for any mistakes that you make with your web startup or idea. You could either be charging zero or charging a small $5 to $20 per month charge. You might make a mistake. Are you willing to accept and even be able to insure for thousands or even millions in liability for those mistakes?

[bigiain]

"please think for a second what liability you would want for any mistakes that you make with your web startup or idea"

It seems to me that bitcoin wallets are a relatively new and not well enough understood risk. There are very few other "files" like them, in that an attacker copying them can deprive you of their value in a way that you cant protect with backups. I feel a big part of current "internet security best practices" are about minimising the risk of getting exploited - but with a pragmatic limit to how much effort you invest mediated by the excuse of "if we _do_ get rooted, we can always reinstall and recover from backups". It'll only cost you time, and perhaps some reputation, and may put assumed-private-to-you information in someone else's hands, but it hasn't deprived you of access to any of your data. That doesn't apply to bitcoin wallets, and example like this are pointing out flaws in assumptions people are making about appropriate ways to manage them.

It'd suck to be "that guy" who provides the object lesson in why we need to think differently about bitcoin wallets to just about any other file type we might put on an internet accessible machine, but we _do_, and I don't know whether we have an answer to the question "Is there a way to secure a bitcoin wallet on a machine someone else has root access to (either your datacenter's staff with physical access, or the people with hypervisor access to the hardware your vm is running on)?"

I _think_ the answer is "if you can't trust those people, you can't risk storing your bitcoins there". There's a reason people keep their money in banks, and not in train station luggage lockers. I'm guessing inexpensive commodity VPS's should be considered closer to storage lockers than bank vaults. I suspect the finance sector and/or fortune500 companies have hosting arrangements with companies offering bank-vault grade protection and reserve bank style insurance - but sure as hell not at $24.95/month.

[Tuna-Fish]

An interesting point is that for receiving coins and for long-term storage, bitcoin wallets do not need to be online, and in fact do not need to ever have touched a machine that has been online. While slush "just" lost his "hot wallet", another user lost 200k. When will it become common best practice to store high-value accounts entirely off computers? You can print out a bitcoin wallet and put it in a safe deposit box for storage, and still add money to it.

[baddox]

I don't see what's so conceptually new about bitcoin wallets. They're just plain text that you don't want people getting access to. It's no different than storing passwords in plain text: if someone copies them, they're completely compromised (until the user changes them). The solution is pretty simple: encrypt your own bitcoins with your own password (or more ideally, your private key). Then, if someone hacks your server, they don't get anything.

[kamaal]

Sorry for asking a noob question.

But what really is a bit coin? I mean in physical existence. Is it just a file(plain text) with some data/metadata?

And stealing it means copying those files, and then deleting the source?

Which in case how is this any different than traditional bank account. My money in the bank is basically DB record. And that can be stolen.

The bank can then just say to every one 'look this transaction from db such and such is no longer valid'.

Can't bit coin do the same? I guess I'm missing something fundamental.

Can somebody explain this?

[coopdog]

A 'bit coin' is a space on a block chain that everyone has a copy of. You lock the coin with a cryptographpic key, which you need to store. Whoever has a copy of the key can unlock the bitcoin and re lock it with a different key, such that now only the new owner has a copy of the key. Everyone can still see the entire block chain, but only one account (that no one knows the owner of) has the ability to move that coin.

So they got access to these peoples keys and transferred ownership of the coins.

Most money supplies are regulated, but bitcoin isn't regulated. No one has the ability to say 'reverse that transaction', but it also makes the currency safe from inflation and interference by money printing governments and privacy snoops.

[chalst]

makes the currency safe from inflation

There's an economic myth that inflexible supply of a commodity gives that commodity when treated as money, stability. It does not, as looking at this graph of US inflation/deflation time shows (1944 is when the dollar stopped being gold convertible):

http://en.wikipedia.org/wiki/File:US_Historical_Inflation_An...

A few points:

1. The money supply around a currency not only contains the mined/minted instances of that currency, but also liquid currency-denominated assets, like customer bank balances. So money supply is not necessarily bounded.

2. Inflation/deflation can be considered measures of the change in demand for the currency. In times of deflation, holding money is valuable because it becomes more valuable.

3. From the above graph, you see that during the gold standard, inflation tended to be mostly balanced out by deflation in the long-term, so long-term inflation was low. But in the short term, prices were very unstable as inflation jumped all over the place, and far more unstable even than fiat money in the past three turbulent years that we've seen.

4. From the point of view of an economy, inflation and deflation are not symmetric; because of the value of sitting on money during periods of deflation, savers do not tend to invest their money but move money from investments to cash savings. This undermines economic activity. But in an economy with a rich range of investment opportunities, moderate inflation does not penalise acquisition of money and does encourage investment.

If you want a non-performing store of value and don't mind big fluctuations in value, gold is there and we know how to secure gold rather well. Bitcoins are another non-performing store of value with far more drastic fluctuations in value, and securing it involves the double vulnerability: physical security of storage media, information security of computations involving bitcoins. And it is much easier to accidentally lose bitcoins than gold, pirate tales notwithstanding.

[stuhood]

> physical security of storage media, information security of computations involving bitcoins

Only funds that you have daily access to need be vulnerable to the latter point, as physical security (air-gapping) is sufficient when you do not need to -send- funds.

> And it is much easier to accidentally lose bitcoins than gold, pirate tales notwithstanding.

Strongly disagree. Can you keep N redundant copies of your gold? Combined with secret splitting, you could require that at least K of N secure locations be accessed.

[vrotaru]

Not exactly. In order to spend bitcoins (transfer them to other wallets) you need them not encrypted.

So if you are a bitcoin business making bitcoin payments, at moment you cannot avoid the risk of having the wallet stolen if someone gains unauthorized access to your operating computer.

[sans-serif]

Keys normally grant access so worst case once they're compromised you can take the server offline physically. Even private signing keys can be revoked though some damage may have been done.

With Bitcoin, once it's copied, you can consider it gone forever, irrevocably, in totality.

[repsilat]

I thought the idea was if someone copied your bitcoins it was essentially a race to be the first to spend them. I guess if you're stealing bitcoins, though, you're going to be quick to run them through a couple of transactions to claim ownership of them.

(Of course, my understanding might be completely off-base.)

[gizzlon]

"Is there a way to secure a bitcoin wallet on a machine someone else has root access to (either your datacenter's staff with physical access, or the people with hypervisor access to the hardware your vm is running on)?"

Probably not, _maybe_ trusted computing could help[1]. But it might secure enough if you had a special piece of hardware that stored the bitcoin-key and did all the signing operations. I believe things like these exists for Certifications and other signing-keys. I doubt VeriSign stores their root certificates on just-another-box :)

[1] In theory, TC could "anchor" your data to the hardware chip on the motherboard, and breaking it would require physical tampering.

[JoachimSchipper]

You're thinking of a hardware security module (HSM). These are standard for e.g. certificate authorities. To the best of my knowledge, they have never been used by bitcoin outfits, but that is certainly possible, in principle.

Of course, sane people don't mix HSMs and VMs.

[etcet]

Don't store your money on servers you don't own. Don't have a web interface. Don't allow anyone to connect. Don't use passwords, use public key encryption. These are the basics.

[tomp]

What if they steal the key?

[jrockway]

The key is encrypted with a long and key-strengthened passphrase, so the Universe will run out of energy long before their computers get the key.

[tomp]

But then, you're back where you started. Why not just use a password in the first place?

[noduerme]

Still, the amount stolen in Bitcoins here is probably small potatoes next to the value of all the credit cards and personal details stored in other Linode instances. If someone can hack their admin panel and get root access to 8 accounts, they can get access to all the accounts. This time it happened to be done by someone who was going after the Bitcoins. But how can they claim anything in the way of security?

People who host on a VPS typically do so because they want to run complex applications that scale up, and do so without the cost or headaches of adding more physical hardware. Most people who use Linode probably do not do so to host static webpages, which can be done just fine on a shared server for a quarter the price. While Linode might be cheap, they do claim to be secure, and if they didn't it's doubtful people would host their apps there. And most apps do take some personal information; and a large number of them take financial details.

No one would use Linode to host their apps if they thought all accounts were rootable from a master login panel. And no one would use them to host static pages. So who would their customers be if people didn't expect their instances to be secure?

[dave_sullivan]

It's certainly a grey area, but at what point is it safe to assume that if you get hacked, it's not going to be because your ISP got hacked? Say this happened to Amazon and it affected a company like Heroku or dropbox, both users of AWS? Regardless of what terms of service says, I'll bet there's some liability somewhere. And if there's a cut off, maybe linode should advertise that? "Hey, we're cheap, but you get what you pay for!" rather than "You're getting ripped off if you go with amazon over linode!"

If a bank gets robbed, I'm not liable for the cash they steal. But how about if I've got cash in a safe deposit box and someone uses a fake id to get into it, and the bank doesn't recognize the fraud? That's trickier. And if someone robs my house and I've got a bunch of cash under my mattress, that's another story too. I know the analogy doesn't quite hold up because it's kind of like a bank and a customer engineering a safe together (eg both could be at fault for a break in), but there's got to be some responsibility on Linode's part.

[tadfisher]

I'd say this is more akin to stashing a bunch of money in a self-storage unit instead of a bank account. One explicitly insures against theft, the other does not. The onus is on you as a customer to decide what to go with.

[tylermenezes]

> One explicitly insures against theft, the other does not.

I'd take issue with that summary. If I put things in a self-storage unit, and it gets robbed because some employee left his master key under the door mat, regardless of how bad of an idea it was to store my money there, that's still their issue.

Think about it this way. I could store my money under a table at McDonald's, in a self-storage unit, or in a bank. Clearly the self-storage unit should provide me some more security than McDonald's. So when an attacker gets access through some really trivial method that they really should be protected against, that's their fault; it doesn't matter that there was a better security option, because it's still below what I was paying for.

[larrys]

"So when an attacker gets access through some really trivial method that they really should be protected against, that's their fault"

A good point but keep in mind that courts don't have the level of expertise to judge what in terms of security is trivial and what is not.

The person trying the case and/or the jury may very well be someone who uses "football" as a password.

Also there are multiple cases of the very best companies with supposedly the highest levels of security getting hacked on a regular basis (might be a small percentage but it always makes the news). Consequently any company defending could make an argument that "this stuff happens even with the best and brightest" and it might be believed. (Well anyway that's what I would argue if I was a lawyer..)

So the public could easily be convinced in the case of a technology company something that would never fly as far as a screwup at the self storage - something physical that they can relate to (like leaving a door unlocked which is easy to understand).

[tripzilch]

A bank or safe deposit box business will most definitely have clear terms about how liable they are for if something gets stolen.

For a hosting company this is different. Especially because--it's hard to draw a line but I feel it's there--there's a difference between storing sensitive data and storing (what are practically) valuables/money.

I wonder though, I'm not clear on his set up (or business, even), but basically he was literally storing his bitcoin money on a Linode server? Since they're not a bank, nor a safe-deposit service, or are in the business of storing valuables (as opposed to sensitive data). I can't come up with a realworld analogy (they usually break down anyway), but wouldn't you want to wrap this data in an extra layer of encryption or something? It's not that hard to come up with some scheme so that people with root access to the Linode server can't do anything with it either. Since this is about (almost) real money, that's what I'd do.

[larrys]

"And if there's a cut off, maybe linode should advertise that? "Hey, we're cheap, but you get what you pay for!" rather than "You're getting ripped off if you go with amazon over linode!""

Well of course that's never going to happen and the truth is the contracts of the more expensive provider no doubt also limit liability. (And all of this is in the TOS/contract etc). So what we are really talking about is who will do a better job protecting what you have AND more importantly who has more to loose if they screw up. (Small linode has more to loose but they also don't have deep pockets to pay. Amazon has deep pockets but access to magnitudes greater legal help to prevent having to pay.)

[RLG_RLG]

> but at what point is it safe to assume that if you get hacked, it's not going to be because your ISP got hacked

If you co-locate your hardware at a data center and your staff competently secures your systems.

[spindritf]

Secure with what? A booby trap? There's always a level of trust you need to have for your provider. Even when you get a cage in their DC.

[RLG_RLG]

Set low level passwords and use filesystem encryption. eg: passwords on all networking devices, boot loaders, and BIOS.

Three letter agencies and foreign governments could attack your data if they took it off line, but you monitoring should detect that.

[eli]

I can put a "not responsible for stolen items" sign in my restaurant, but if the coat check employee bolts out the door when you hand them your coat, I'm buying you a new one.

[bigiain]

<devil's advocate>Yeah, but could the (ex) coat owner hold the restaurant's landlord liable? Isn't it the restaurant _managers_ problem?

I think there's a _lot_ of "grey areas" here, and while I feel sympathy for the guy who's out ~$13k worth of bitcoins, I can't help but think he was "doing the wrong thing" relying on the security of an inexpensive vps to keep them safe…

[chc]

What would you recommend? Would you say the same thing if he'd been colocating and a data center employee had stolen his bitcoins? Because that seems far more analogous than any restaurant analogy, and I don't see any reasonable way for somebody who's not a huge corporation to avoid this kind of risk.

You have to trust somebody at some point unless you're keeping the server locked in your own closet. It seems really bizarre to me to say that a hosting provider doesn't have a responsibility not to steal your stuff.

[bigiain]

What do I recommend? I'm really not sure…

Firstly, I'd start asking whether a $19.95/month shared hosting* account is a "reasonable" place to store $13k worth of (effectively) cash. I'd be _very_ careful if I had that sort of folding-money-type-cash on hand, and would under normal circumstances automatically deposit in a bank account to mitigate the risks involved with carrying it around. And I'd usually take steps to not ever have that sort of value of cash build up or be required - the only transaction I've ever done of that sort of value in cash is selling or buying a car from an individual - and that's always been a direct from transaction to the bank type of arrangement.

If I had enough bitcoin value that it'd hurt to lose it, I would not (at least now in hindsight) store that on a machine that other people I don't know/trust have root access to. Maybe I'd keep my wallet on a usb stick in my pocket or in a safe at home? I think though that at somewhere near the $13k value the "right" thing to do is convert it to cash and take advantage of the existing banking system and its time-tested security and insurability.

(* Which is fundamentally what a linode VPS is, at least from anyone with access to the hypervisors point of view.)

[larrys]

Would have to meet several tests. 1) Hosting company knew and agreed to the value of the things they had control over. 2) Employee did it and they were negligent in hiring that individual. 3) Reasonable and customary for that type of stuff to be in that situation given pricing and the practices of others.

With respect for #3 it would be reasonable for a bank safe deposit box to contain a $100,000 ring maybe but not to contain a 10,000,000 ring.

[showerst]

Then he should Colo with a hosting provider with a contract provision that specifically holds them liable for any losses related to problems caused by the host, and enumerates those possible losses beforehand.

[larrys]

Simply not going to happen at any reasonable rate.

There's a saying in many businesses with different variations:

"Price, quality, speed" pick any two.

So this would be:

"Price, security, bandwidth" pick any two.

People pick on price and bandwidth security is taken for granted to be commercially acceptable. The colo can easily figure out price and bandwidth they are clearly defined. Losses from a security breach not as easy - to many variables. Same reason insurance companies love to write life insurance but hate to write disability insurance. Life insurance is absolute (you know when someone is dead and tables exist to compute probabilities on when they will die). Disability is open to interpretation, fraud and other things. It's not black and white.

[eli]

Does such a thing even exist?

[larrys]

Sure but there is legal precedent for that as well as it's an easy concept for the everyday man to understand. (Employee and seeing them makes it a open and shut case.)

As an aside, signs like that are the same as ones that appear in auto shops that say "insurance regulations don't allow you in the shop". I owned a company that did manufacturing and said a similar thing because I didn't want anyone in the machine area. I've dealt with insurance companies for many years they don't dictate things like that at least from my experience.

Your example is correct though the restaurant would be liable in the situation you cited because you actually saw an employee steal the coat. If you saw someone else NOT connected to the restaurant AND the coat was expensive you are probably out of luck. It's your property and there is no assumption that a restaurant protects you from acts of god.

The key is was their negligence on the part of the restaurant.

On the other hand if there was a large party at the restaurant and normally a coat check girl that's a different story (devils in the details with this stuff).

[Drbble]

What if the coat was full of diamonds?

[nknight]

In an actual court case, the reasonableness of everyone's actions would be evaluated, but it's hard to imagine a court finding it unreasonable that someone placed data worth $13,000 to them on a respected VPS provider.

That doesn't mean Linode has any legal liability in this case, just that your analogy is off the mark.

[bigiain]

"it's hard to imagine a court finding it unreasonable that someone placed data worth $13,000 to them on a respected VPS provider."

Really? (I'm reading that as saying you think it _is_ a reasonable thing to store $13k worth of effectively-cash-value in a $19.95/month vps account?)

Does anyone know what regulations like HIPPA or PCI have to say about the security of data stored on managed-by-3rd-party servers like VPSs?

[orofino]

IANAL (or QSA)

PCI doesn't specifically say anything about usage of a VPS. It does however speak about access to data.

If you have encrypted credit card information, you'll be asked to list those that have access to the encrypted information, they encryption key, and the key encrypting key. Then you'll be asked to justify their access.

I'm sure I could come up with with several other major violations, but this alone is severe enough that I can't envision a way that you'd pass a PCI audit.

[nknight]

First of all, yes, I think it's reasonable.

Second, where are you getting $19.95/month from, anyway? I haven't seen the plan in question mentioned, and even if this particular VPS happened to be Linode's lowest-end, the last time I looked (a while back, granted), slush had multiple large VPSs with Linode.

Third, really, what does the price of the VPS have to do with it? You think as the cost of the VPS goes down, we're entitled to less assurance that an employee isn't going to bolt with our data?

Finally, HIPAA and PCI regulations are ginormously complex, but violations of them almost inevitably cost a hell of a lot more than $13k.

[bigiain]

Interesting.

I'll freely admit I've only been thinking about this since reading this article, so I'm both not-fully-informed and I'm thinking about it as a response to some guy losing ~$13k, but to me it's _not_ reasonable.

(And my $19.95 number is perhaps hyperbolically chosen from their least expensive vps offering - but my assumption would be that the management/hypervisor back end would be shared across their entire infrastructure, so I think my argument holds, in that I'd expect the higher priced offerings to "only" have the employee-reliability-assurance of the cheapest vps…)

[larrys]

Linode plans (like those of other hosting/vps providers) are differentiated on storage/memory/bandwidth. Security isn't a factor. Compare that to an auto where "security" is definitely mentioned and part of the selling proposition and what you pay (they mention theft devices, crash stuff, airbags etc.)

[jhdevos]

What if the person had a $100,000 worth of cash in his coat pocket -- would you reimburse that, too?

[finnw]

He could never prove that he had the cash in his pocket. But you can prove that the bitcoins existed and had not been spent.