This research proposal introduces SubMod-RAG, a principled framework applying submodular function optimization for context selection in multi-modal Retrieval-Augmented Generation systems. The method addresses critical limitations in current approaches by formulating context selection as a submodular maximization problem under cardinality and knapsack constraints, enabling efficient selection of diverse yet relevant multi-modal contexts. The proposed framework integrates seamlessly with existing retrieval pipelines while providing theoretical approximation guarantees for managing token budgets and reducing cross-modal redundancy.
Key findings
Context selection is formulated as a submodular maximization problem under cardinality and knapsack constraints to manage token budgets
Modality-aware submodular functions capture cross-modal dependencies and redundancy
The framework provides theoretical approximation guarantees for constrained optimization
Addresses three critical challenges: independent selection without correlations, lack of diversity, and non-monotonic utility
Limitations & open questions
This is a research proposal (Version 1.0) with planned experiments rather than completed empirical validation
Experimental results on knowledge-based visual question answering and image captioning benchmarks are pending
Practical computational overhead of submodular optimization in real-time RAG systems not yet evaluated