This document presents a comprehensive literature review of 3D multi-modal scene understanding, tracing the evolution from geometric deep learning (PointNet++) through transformer-based methods to modern vision-language fusion approaches. It analyzes key fusion strategies including early fusion, late fusion, and query-based architectures, with deep dives into recent benchmarks MMScan and EmbodiedScan. The review identifies critical research gaps in efficiency, data scalability, and real-time deployment, proposing Fast3D-VL as a solution for high-speed multi-modal 3D perception.
Key findings
3D perception has evolved through three distinct eras: geometry-focused (2017-2019), transformer-based (2020-2022), and multi-modal fusion (2023-2024).
Multi-modal fusion is essential for state-of-the-art performance, with RGB-D consistently outperforming single modalities.
Current state-of-the-art methods suffer from severe efficiency limitations, with most operating below 5 FPS compared to real-time requirements of 30+ FPS.
Data scaling shows diminishing returns after initial gains, emphasizing the need for quality over quantity in 3D-VL datasets.
A significant gap exists between research benchmarks and production deployment, particularly regarding edge device optimization.
Limitations & open questions
The proposed Fast3D-VL method is mentioned but lacks detailed architectural description or experimental validation.
Analysis is limited to existing published benchmarks and does not include original empirical experiments.
Focus remains predominantly on indoor scenes despite identified need for outdoor benchmarks.