Skip to content

MIND'S EYE: IMAGE RECOGNITION BY EEG VIA MULTIMODAL SIMILARITY-KEEPING CONTRASTIVE LEARNING

šŸ”„INFO

Blog: 2025/07/31 by IgniSavium

  • Title: MIND'S EYE: IMAGE RECOGNITION BY EEG VIA MULTIMODAL SIMILARITY-KEEPING CONTRASTIVE LEARNING
  • Authors: Chi-Sheng Chen, Chun-Shu Wei (National Yang Ming Chiao Tung University)
  • Published: June 2024
  • Comment: arxiv
  • URL: https://arxiv.org/abs/2406.16910

šŸ„œTLDR: Regularize contrastive loss with similarity-keeping term

Motivation

This paper aims to overcome the challenges of low signal-to-noise ratio and nonstationarity in EEG-based image decoding by proposing a self-supervised contrastive learning framework (MUSE) (similarity-keeping regularization term) with tailored EEG encoders, significantly outperforming prior methods in zero-shot image classification on large-scale datasets.

Model

EEG encoder architecture: (šŸ”„upstream spatial conv + Spatial-Then-Time/Time-Then-Spatial combination + Graph Attention)

image-20250731232009059

regularize the standard contrastive loss with Similarity Keeping term:

šŸ¤”Innate character: Similarity Keeping term keeps the distribution of similarities with negative samples

image-20250731232215133

Evaluation

SK term is helpful in Nerv-series architecture but not very helpful in MUSE-series.

image-20250731232321568

Interpretability

image-20250731232448094

MUSE-SK model highlights the modelā€™s enhanced focus on temporal and occipital areasšŸ§Seems not obvious...

image-20250731232517426

image-20250731232701263