[_joel]

We used historical flow meter readings (started out with 15 minute intervals but it worked much better with a higher frequency) and used that to train Recurrent Neural Networks (RNN's) to predict where areas were likely to flood. I was the devops lead on it for the prototype, not the data scientist unfortunately so can't give you the in's and outs but can tell you that we used tensorflow together with pandas/df/timescaledb. We then displayed that using plotly, all this was stuffed inside several containers. It was a great project to work on actually. The whole setup was pretty much a joy to work with.

[msaharia]

That is awesome. I work in flood forecasting and previously worked in NCAR/NASA. If you would be so kind, feel free to share any white papers, links etc to the project you are referring to.

msaharia[a]iitd.ac.in

[_joel]

I don't I'm afraid, for starters I contract and no longer work on the project, plus there's IP invovled etc. Also I'm no data scientist, just a hacker really :)

The main difficulty was getting access to the data (and ensuring it was valid, as with all ML projects), lucking we managed to get that from several sources (councils, water companies etc). The flow data was in TimescaleDB, pandas dataframes so we could use varying levels of frequency of data and we used HDF5 iirc as well (detail is hazy, it was a few years back now). We did demo it to the Met Office here in the UK too. They were interested but already had their own thing cooking up so the project never really got out of prototype, but it was making accurate predictions. I think there were some other areas that might turn out to be flaky over time using this method (such as rapid changes to catchment areas) but that could maybe be factored in someway with more thought on the model and verification on a larger set/timeframes. Feel free to hit me up if you want any more detail on the tech side, but don't ask me about stats, maths fu I ain't :)

joel [at] smashthesystems.com