[emilwallner]

It could try to classify the apple tree or the context, but it would require a lot of training data. If it's out of context, it should select a color based on probability. But it's hard to solve this with just input and output data. The simple solution is to use noncontradictory training data, i.e. only having green apples.

I have an urge to teach it simple logic. Instead of making it brown, it selects the color with the highest probability from a range of colors. However, I haven't come across a deep learning implementation like this to mimic.