Exploring The Visual Feature Space for Multimodal Neural Decoding
Weihao Xia
Cengiz Öztireli
University of Cambridge
[Paper]
[Code]
Method
Our method builds upon UMBRAE, which aligns brain activity with image features for zero-shot captioning and grounding. The learned brain encoder maps brain activations into an image feature space, which is then processed through an adapter and an LLM for downstream tasks. We further explore the impact of different image feature spaces on fine-grained multimodal decoding, drawing inspiration from prior MLLM research that leverage different feature representations.
We investigate four types of image feature spaces: (a) Single Encoder (SE): uses features from a single pre-trained vision encoder (e.g., CLIP) for brain alignment—commonly used in MLLMs.; (b) Mixture of Encoders (ME): integrates features from multiple task-specific vision experts such as CLIP and DINO; (c) Aggregated Feature (AF): combines dense features from various layers (shallow, middle, and deep) to capture different image characteristics.; (d) Nested Features (NF): uses a variable-length set of visual tokens, downscaled using multiple downsampling factors to produce a hierarchical, nested representation—encoding visual content from coarse to fine-grained details.
Advantages:
- Versatile and model-agnostic: VINDEX is a flexible brain decoding framework that can be easily adapted to various subjects, image sizes, visual encoders, MLLM settings, input prompts, and downstream tasks. The single trained encoder can be applied to all LLaVA-based variants due to architectural consistency.
- Zero-shot multimodal decoding: VINDEX performs zero-shot multimodal brain decoding by leveraging diverse visual feature spaces from vision expert encoders, enabling brain decoding at multiple levels of detail.
- New evaluation metric: We decompose decoded captions into objects, attributes, and relationships for evaluation. Semantic discrepancies are reflected in precision and recall: precision penalizes hallucinations and recall captures omissions, , offering a clearer view of what is genuinely decoded from brain activity.
Benchmark
MG-BrainDUB (Multi-Granularity Brain Descriptive Understanding Benchmark) contains two tasks: (a) detailed captioning and (b) salient question answering, each with corresponding metrics.
Descriptive Caption Understanding
Motivation:
(a) Existing studies are limited to coarse interpretations, lacking essential details such as object descriptions, locations, attributes, and their relationships. This results in imprecise and ambiguous reconstructions when these cues are used for visual decoding.
(b) Evaluating brain-decoded detailed captions—often hundreds of words long—poses a challenge, as conventional metrics become unsuitable. Traditional captioning metrics, which mostly rely on n-gram matching between candidate and reference captions, are highly sensitive to stylistic variations, leading to inconsistent evaluations. Model-based metrics either depend on outdated text encoders or struggle with long texts due to token length limitations.
(c) Decoding methods that rely on pretrained models benefit from powerful generative capabilities but are also vulnerable to systematic biases. They often introduce hallucinated details, such as prototypical co-occurrence patterns, raising concerns about what is genuinely decoded from brain activity versus what is hallucinated by the generative model.
Our strategies: (a) we investigate brain alignment across different image feature spaces to identify informative representations; (b) we decompose decoded captions into objects, attributes, and relations for evaluation. This enables us to assess alignment at a finer semantic granularity and evaluate which types of information are truly recoverable from brain activity. Such semantic discrepancies are reflected in precision and recall: precision penalizes hallucinations, while recall captures omissions. Together, they reduce the confounding effects of hallucination from pretrained models and provide a more accurate measure of true information decoding capacity.
Salient Question Answering
Motivation: (a) effective brain decoding methods should be capable of understanding the intricate information embedded in brain signals, but current approaches lack proper evaluation of this ability; (b) brain signal data are often noisy or error-prone due to distractions or fatigue during the recording process—especially when decoding fine-grained details such as small objects.
Our Strategies: (a) We design a salient and complex reasoning task to evaluate the model’s ability to recognize objects, understand their interactions within the image, and answer in-depth reasoning questions that require step-by-step logical processing. Since COCO often overlooks salient categories beyond its 80 common classes, we use SAM and MLLMs to re-identify the main scene or key objects in each test image.
Results
Concise Captioning
Descriptive Captioning
(NF
n refers to the nested feature variant where the token size n is selected from {1, 9, 36, 144} after downsampling.)
Key Observations:
- Direct alignment with image features enables zero-shot decoding and demonstrates improved performance in concise captioning, detailed captioning, and grounding tasks.
- Previous captioning metrics fail to effectively evaluate open-vocabulary, detailed descriptions. Our proposed metric addresses both hallucination and omission, providing a more accurate assessment of the model’s fine-grained and genuinely decoded information.
- While aligning LLMs with complex image feature spaces can improve performance in image-based tasks for MLLMs, brain decoding faces a trade-off: some information present in images may not be captured in brain data, making alignment with salient features more effective.
Acknowledgements
This project was supported by a UKRI Future Leaders Fellowship and also was undertaken with the support received through the Imminent Research Grants—an initiative established to foster innovative language research—by Imminent, the research center of Translated, a pioneering company in AI-driven language technologies dedicated to enabling seamless global communication.
