Your scenario makes me suspecting that it is Xilinx flavored Yocto causing the problem. I think that removing some unused Xilinx-specific layers/recipes can reduce the prologue and epilogue execution time.
Well, Xilinx layers are definitely not the most light-weight ones, though they still shouldn't cause parsing/task init to take minutes. By any chance, are you using WSL? I have heard some complaints about storage I/O performance, when it comes to this...
Nope, mine runs on a native Linux box featuring Intel i7-1370P and 32GB RAM. Maybe Xilinx has some tuning which makes Yocto become slow at parsing the recipe dependency and the likes.
I've tried asking the Xilinx community, and got only a reply saying that there is a database in Yocto which limits the scalability.