So when I was doing stuff with it, I realised I don't wanted to save in my palette all the color I used for the illustration. So add colors to the palette has to be a manual action from the user.
But Im open to suggestions, you're not the first and not the last to complain about it :) [some people find it extremely confusing]