[theaiquestion]

I think the method of merging the pipelines via img2img should use controlnet. Possibly needing to be finetuned specifically for this, although existing controlnet models might work fine for this.

This is exactly what you'd want to use controlnet for - mapping semantic information onto the perceived structure.

[tmabraham]

Yes we've been looking into ControlNet as well, and I think there is one recent fMRI-to-image paper that also has tried ControlNet. Maybe we'll use ControlNet in MindEye v2 :)

[atom_101]

Yes controlnet will be used in the next version. For this one we couldn't get it working in time.