[sampo]

> you need to preprocess your input data, or deserialize it, normalize it, join it or transform it based on some rdbms data, then you do your calculations, and then you need to graph it, serialize it, etc.

In large scale number crunching, like climate models, numerical weather prediction, the typical case is that input data is conceptually in a regular 2d or 3d grid, and stored in binary format files (like NetCDF or HDF), as that is more efficient and saves space.

Then the heavy lifting number crunching code runs on a cluster as a batch job, reads in the data, crunches the numbers, and writes results out again in NetCDF or HDF files.

The output files are then downloaded to a desktop PC, and graphing is done with Matlab (Python is also getting more popular) or especially in meteorology with some dedicated meteorology graphing software.

The binary format input and output is probably about as efficient as it can be. Also, heavy number crunching scientist probably don't have much use for relational databases.