[f0ti]

Have been doing something similar to this using image to image translation (XYZ rendered images to RGB space domain). Most of the information is contained in the Z-axis which gives you the height information, e.g. helps to distinguish the grass and buildings color. However I am skeptical if the X and Y is noise and how much spatial information it provides during Conv blocks. Anyone who had previous experience on this?

https://github.com/f0ti/thesis

[W0lf]

As pointed out in my other comment, using a single image for point coloring is prone to errors due to noise, specular reflection and occlusion. I'd consider using a (normalized) cross-correlation approach with several images.