Multimodal natural scene reconstruction from fMRI using generative AI
Abstract
Reconstructing natural images from functional Magnetic Resonance Imaging (fMRI) signals is a challenging problem due to the high dimensionality, noise, and indirect nature of neu- ral measurements. Recent diffusion-based approaches have improved reconstruction quality by leveraging pretrained generative priors, but most rely on fixed conditioning strategies that use a single static neural representation throughout the denoising process. This project pro- poses a timestep-conditioned adaptive fusion framework for fMRI-based image reconstruc- tion, designed to dynamically integrate structural, semantic, and hierarchical neural repre- sentations during diffusion-based generation. Several conditioning strategies, including ex- plicit shape conditioning, CLIP-based semantic guidance, and multi-scale feature extraction, are investigated to understand their impact on reconstruction quality. Building on these find- ings, the proposed adaptive fusion method is shown to improve reconstruction quality over fixed-conditioning baselines on the Generic Object Decoding (GOD) dataset across percep- tual, structural, and semantic evaluation metrics. The results demonstrate that aligning neural conditioning with the temporal dynamics of diffusion models can improve brain-to-image re- construction and provide a more effective framework for neural decoding.
Read the paper
Related studies
- Learning from First-Generation Qualitative Approaches in the IS Discipline: An Evolutionary View and Some Implications for Authors and Evaluators (PART 1/2)
- Commentary on the clinical and preclinical dosage limits of interstitially administered magnetic fluids for therapeutic hyperthermia based on current practice and efficacy models
- Prosthesis: New Technological Opportunities and Innovative Biomedical Devices
- The history and future of neural modeling for cochlear implants
- Ryanodine receptors: waking up from refractoriness
- Novel Transgenic Rabbit Model Sheds Light on the Puzzling Role of Matrix Metalloproteinase-12 in Atherosclerosis
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.