IgE allergies show up quickly - but many allergy (mast cell) reactions, are not IgE reactions and may take considerably longer to show up (for example, MCAS reactions generally peak in the middle of the night, unlike IgE reactions.)
Our immune system is complex, IgE is just a tiny part of it. Some introductory reading: