[duanem]

I process weather simulation data from around the world (22 data sources). My servers process around 400-500 GB of data a day and I store about 120 GB which gives me about 5 days of historic simulation data and up to 16 days forecast data.

Storage and processing is relatively easy since it's just a matter of throwing resources at it. However, it'll be damn expensive to store months or years of historic simulation data.

The difficult part is writing and maintaining the processing scripts. The different weather services store data in different formats, have different ways to download the data and sometimes change data structures or fall over.

Edit: removed a spurious sentence.

[thisisbrians]

This is very helpful. Thank you for sharing.

If you wouldn't mind: how do you go about creating a simulation around the empirical data? This seems to be very core to how Dark Sky is able to provide such accurate realtime data for geographies in between weather stations.