There's already been preprints released showing fMRI reconstructions that appear to do better than an implicit multi-class classifier [1] [2]. But also, even if the result is an implicit multi-class classifier, if the n is sufficiently high then that would still be quite impressive!
I see no evidence that they do better than multi-class classification, in fact they both work as I described. They learn embeddings of fMRI which perform an implicit classification of the data (which can be recovered just by quantizing the embedding space to get its modes) and put very large generation models on top.
The only reason the reconstructions are much better than before is because they use the latest generation models. Those models have internal models of the classes which allow them to fill in the high-frequency details in the reconstruction. The only information they get from the fMRI is the same low-frequency signal that previous papers already had, and indeed the only things the reconstructions get right are low-frequency: class of object/scene, broad position of object, broad shape of object.
fMRI scans are aggregates of brain information, they act like low-pass filters over the brain state. You can put as big a model on top as you want it won't make it more truthful a reconstruction.
I think, as you say, that detecting as many classes as possible is already a pretty good goal, developing new embeddings and techniques to see how much juice we can squeeze out of the scans. I like the arxiv preprint you posted in particular since it does just that and evaluates accuracy (although the way it does it is flawed since it uses an image classifier on the reconstruction which presents the same problems). What I don't like is the misrepresentation of what's going on when people put those large generative models on top of this kind of data.
The only reason the reconstructions are much better than before is because they use the latest generation models. Those models have internal models of the classes which allow them to fill in the high-frequency details in the reconstruction. The only information they get from the fMRI is the same low-frequency signal that previous papers already had, and indeed the only things the reconstructions get right are low-frequency: class of object/scene, broad position of object, broad shape of object.
fMRI scans are aggregates of brain information, they act like low-pass filters over the brain state. You can put as big a model on top as you want it won't make it more truthful a reconstruction.
I think, as you say, that detecting as many classes as possible is already a pretty good goal, developing new embeddings and techniques to see how much juice we can squeeze out of the scans. I like the arxiv preprint you posted in particular since it does just that and evaluates accuracy (although the way it does it is flawed since it uses an image classifier on the reconstruction which presents the same problems). What I don't like is the misrepresentation of what's going on when people put those large generative models on top of this kind of data.