[soravux]

As a complement of information, the question of colorization was recently revisited using deep learning methods [1]. All very interesting work!

[1] Zhang, Richard, Phillip Isola, and Alexei A. Efros. "Colorful image colorization." European Conference on Computer Vision (2016) http://richzhang.github.io/colorization/

[stefanpie]

You could always optimize it by combining both methods in the process somehow.