[simonw]

I've heard from a couple of people who are using it for bioinformatics. It's not an area I know anything about myself but I'm excited to hear it's being applied there.

How big are we talking here?

My rule of thumb for SQLite and Datasette is that anything up to 1GB will Just Work. Up to 10GB works OK too but you need to start thinking a little bit about your indexes.

Beyond 10GB works in theory, but you need to start throwing more hardware at the problem (mainly RAM) if you're going to get decent response times.

The theoretical maximum for a single SQLite database file is 280TB - it used to be 140TB but someone out there in the world ran up against that limit and the SQLite developers doubled it for them!

[pletnes]

Lots of science is «big data, small but important metadata». Also «big raw data, small result data» use cases are out there. (I used to do hyperspectral stuff for a while, which lets you record tons of sensor data to get a small and neat result, think TB -> kB). So GB might not be the best or only metric, as such.

[simonw]

My story for Datasette and Big Data at the moment is that you can use Big Data tooling - BigQuery, Parquet, etc, but then run aggregate queries against that which produce an interesting ~10MB/~100MB/~1GB summary that you then pipe into Datasette for people to explore.

I've used that trick myself a few times. Most people don't need to be able to interactively query TBs of data, they need to be able to quickly filter against a useful summary of it.

[cubes]

I have a friend that's super smart, currently works in biotech, and studied computer science, would you be interested in chatting with him about possible applications? Happy to make an introduction if you like!

[nlarusstone]

I know I’m not Simon but I’ve been in this space for a while and would love to chat with your friend about what applications they’re thinking about and how they’ve been solving this problem at their current company.

Nicholas at sphinxbio dot com

[simonw]

Yes, absolutely! I'm swillison @ Google's email provider.

[cubes]

Cool, I'll reach out, and let you know if he's interested!