[bemusedthrow75]

But convert them to grayscale how?

Black and white film doesn't have one single colour sensitivity. Play around with something like DxO FilmPack sometime (it has excellent measurement-based representations of black and white film stocks).

It's a much more complex problem than it might seem on the surface.

[atorodius]

fair, but can’t you just randomize the grayscale generation for training?

[bemusedthrow75]

I wanted to say no, that can't work.

And I think it can't work. But now I am not sure!

The other day I was working on a mono photo to prove a point: that a model (a photographic artist's model!) with very striking pink hair was of little concern to a photographer who worked in black and white only, and might actually present some opportunities for choosing tonal separation that are not present in those with non-tinted hair.

In different circumstances (film and filter) her hair could appear (in black and white) to the viewer as if it was likely brunette or likely blonde, before any local (as opposed to image wide) adjustments were made.

The question you are asking, I think, is could you get the hair colour right based on the impact of those same circumstances on other known objects in the scene.

I think the answer is no, in the main, generally because those objects likely don't survive to make colour comparisons from (and there are known cases where the colourisation of a building has been completely wrong because it had simply been repainted). And also because it's sometimes not even obvious what a structure actually is, without its colour. People who colourise by hand make this mistake too.

But I concede that given that we have to work with contemporary images to have a colour source, randomising the tone curve is the only thing that could work.