HCFMaNet: A Novel Holistic Cross-Modal Fusion Mamba Network for Multi-Modal Medical Image Fusion

초록

Multi-modal medical image fusion synthesizes functional and structural features from different imaging modalities, providing comprehensive and accurate information for subsequent analysis. Existing fusion techniques based on Transformer often suffer from reduced accuracy and efficiency due to local perceptual limitations and cross-modal computational complexity. Recently, Mamba has proven effective in various uni-modal tasks due to its exceptional ability to model long-range dependency. However, the straightforward and effective cross-modal information flow and interaction based on Mamba remains underdeveloped in multi-modal fusion. In this paper, we propose a novel Holistic Cross-modal Fusion Mamba Network for multi-modal medical image fusion, namely HCFMaNet. HCFMaNet introduces a new local-aware Mamba which perceives local positional relationships while modeling long-range dependency, thereby capturing richer inter-modal local-global feature representations. Additionally, a novel holistic cross-modal fusion Mamba is designed for explicit cross-modal perception and interaction both in spatial and channel dimensions by cross-spatial interaction and the proposed channel exchange embedding mechanism. Extensive experiments across various medical fusion sub-tasks demonstrate the high accuracy (avg.+22.4% ) and effectiveness (avg.+92.7% ) of our proposed method. Furthermore, HCFMaNet can be applied to other image fusion tasks, such as multi-exposure and visual-infrared fusion, yielding precise fusion outcomes.

키워드