[so_tired]

I never really tried multi-task learning. Is it so catastrophic ?

If I have several tasks, plenty of samples for each task, and a network that converges well for each task.

Cant i just mix and task-label these samples, train from scratch a slightly bigger network, and ta dam .. a multi modal network or what ever?

[iandanforth]

Yes you can! The "mix" part is key. It's the sequential learning which screws up networks today. If you randomly sample from tasks you're fine, or if you can replay older tasks while you're learning new ones (essentially another form of random sampling) the network can learn multiple tasks. But the moment you drop a task from a distribution of training data you're going to start losing competency on it. By default neural networks don't have mechanisms to protect data (weights) from being over-written.