Hacker News new | ask | show | jobs
by setq 3291 days ago
If there's anything I've learned about object storage over the last year, it is: don't use object storage unless you need it for a particularly narrow use case of silly data volumes or CDN storage for a massively distributed content network. Anything else, forget it.

It rarely if ever works properly with standard Linux or Windows tools (s3), it has a rat's nest of arbitrary restrictions which require a language lawyer to decypher (s3/iam/vpc/roles), the APIs are vendor specific and sometimes even region specific (s3), the APIs are obtuse (s3 multipart), the clients are buggy (boto/boto3), suddenly you inherit extra costs and configuration requirements if you want to do something like expose it over http (route53/cloudfront/s3), credential storage is a nightmare for distribution compared to rsync/ssh etc. Ugh.

Please note I have used Google Storage as well and all of the above also apply.

The only thing that is positive is capital expenditure is low.

3 comments

I think AWS CLI is pretty handy for lot of tasks. Not sure if you are referring to which multipart API of s3[1]. There is a higher level API for quite some time now which is very convenient. Agree about documentation for iam/vpc/roles being arcane but tbh I have no idea how that could be simplified.

[1] http://docs.aws.amazon.com/AmazonS3/latest/dev/usingHLmpuJav...

Edit: Added link to high level API

> credential storage is a nightmare for distribution compared to rsync/ssh.

I don't understand this argument. Why is it harder to store credentials for S3 or GCS than it is to store credentials for rsync or ssh?

Because everything has no ownership, no standard place to put things and no logical consistency.
I think I'm even more confused now. If you're going to have an app ssh or rsync, it's going to have a password or a private key to use that will be associated with some account, right? And if you want to use cloud storage, you'll also need a credential or a private key to use that will be associated with some account, right? What's the difference?
Is there a stand for object storage?
openstack swift supposedly but this is a slightly incompatible S3 implementation that nothing else talks.

The filesystem works pretty well though ;)

Ceph / librados provides a compatible layer on top of S3 and Swift.

http://ceph.com/ceph-storage/object-storage/

Swift3 speaks aws4 auth now. The real problem is S3 clients that expect an amazon signed cert. No good way to fake that nicely.