Right, if you take type-safety first, you shouldn't be able to. But a list of abstract fruits has very limited usage without the ability to accessing concrete fruit instance. The adoption of downcasting in some OO languages came out from such needs, given that they lacked generic and/or algebraic types. And with that regard, I thought your solution didn't address the original covariance/contravariance problem (that is, want to have heterogeneous list of fruits and allowing to access concrete types of individual elements).
There are type-safe ways, like making a fruit a sum type of apple and banana, or using traits or type classes, etc. But is-a/has-a discussion seems a bit off from that.
For this context, what I'm saying is that if you don't need to extract an apple from a list of fruits, then you can do it safely with inheritance, so whether using inheritance or delegation is irrelevant. Isn't it?