How would they know that the vehicles, say, were supposed to be the same colour. I'm thinking The Italian Job - the same model of cars are used and are distinguished by their colours. Also what about when different vehicles are painted by the production team to look the same (for stunts say), the computer could rightly recognise them as different vehicles - how would it then know that they're supposed to be the same.
For such things it seems you'd need to check every shot change.
BW filmmakers were, of course, well aware that they were filming in BW and would select colors for sets and costumes that would look good in BW. For example, chocolate milk was used for blood. You wouldn't want a colorizer to determine the actual colors, but the intended colors!
Even today, directors rarely seem to want to film in actual color. They'll tint everything sepia, or that hideous blue-orange scheme that is so popular these days.
That TIJ cars were distinguishable only by color was made possible by filming in color. A BW movie would not make a film requiring distinguishing colors that filmed as equal shades of grey.
In any case, any such colorizing system would be designed to accept a bit of guidance here and there from the artist. This is much like when an OCR'd document needs a bit of touch-up.
And even if it wasn't perfect, many BW movies would be made much more watchable, like the 1927 Wings, which is crying out to be colorized (and have a soundtrack added).
For such things it seems you'd need to check every shot change.