Ask HN: Best document storage solution in 2023

[ID1452319]

We have a application which requires a document store. This need to hold up to 10 million documents and be accessible via APIs to retrieve documents to display in our application.

We are considering everything from Dropbox-type solutions to blob storage in GCP.

What kind of document storage solutions are people using in 2023 to meet this use case?

[tothrowaway]

I use B2 and Wasabi because I don't like relying on a single cloud provider. Files are uploaded to both. OpenResty (Nginx+Lua) sits in front to provide caching, and the logic for deciding which provider to pull from.

Wasabi gives you a free bandwidth allowance equal to the number of bytes stored per month. When I use up most of that, I start pulling from B2. And of course, if one of them is down, I pull from the other.

It's more time up front to build instead of just relying completely on GCP/Azure/AWS. But I don't have to worry as much about spontaneous account terminations destroying my business.

[s1k3s]

It's too much of a generic question to be answered right. Do you need global availability? Do you need high speed downloads? Are you worried about bandwidth costs? etc.

We use S3 + Cloudfront for documents that we want to be quickly accessed by our customers. We use SFTP for our internal docs when we don't care that much about availability and speed.

[ID1452319]

No, not global. Each office only really need access to their own documents, there are over 2,000 offices though. It does need to be high speed and yes cost is a factor.

[didgetmaster]

If you are willing to try out some beta software, give https://didgets.com/ a look. You can load in hundreds of millions of documents or other kinds of files and put meta-data tags on each of them. Queries for individual documents or sets of them execute with sub-second speeds. The software is in beta and available for free download.

Currently working on the indexing feature to index the contents of the files too.

[aborsy]

What SFTP client do you use in Linux?

The command line isn’t very convenient.

[s1k3s]

We don't, we're accessing it programmatically from php, it has built in support. I'm not sure how it works under the hood.

[speedgoose]

I would go with an S3 compatible object store by default.

In Open-source Ceph and Minio are common. Garage is newer and has good potential too and it has a simpler design.

https://ceph.com/en/ https://min.io/ https://garagehq.deuxfleurs.fr/

[fpdavis]

The file system was designed to hold documents and does a pretty good job of it, there are several to choose from depending on what OS you run. Backing them up and restoring them is easy. An API to retrieve documents is trivial to write and customize or there are a few tools and APIs already available.

[giaour]

There are a number of fine options for blob storage (S3, R2, Ceph, Azure Storage, etc.), but with that many documents it's likely access control and audit logging will be important. If that's the case, something heavyweight like SharePoint may be a better choice.

[ID1452319]

Are those blob storage options significantly cheaper than Google? I fear we are being pushing down a GCP route as it is what out consultancy know, rather than the best solution.

[giaour]

R2 would probably be significantly cheaper because they don't bill for bandwidth, but S3, GCP storage, and Azure storage would likely be in the same ballpark. Blob storage is a commodity service, so the major providers mostly match each other's prices and features.

If you are already using a particular cloud provider for the web application that will be accessing these documents, you should use their blob storage unless you have a compelling reason not to. Communication between a cloud provider's compute and storage services within the same region will have minimal latency.

[locustmostest]

One possibility is to use our open-core document management API build to deploy in your AWS account: https://github.com/formkiq/formkiq-core

The files are stored in S3, with customizable metadata storage in DynamoDB. As the system is designed to run on AWS Serverless and Managed Services, the majority of the cost will come from S3 storage fees.

[LLcolD]

Documents need to be indexed?

[ID1452319]

No. The application gives each document a unique filename and they are then only accessed via the application doing forwards.