ABSTRACT
This paper introduces three sample-efficient variants of Sparton to reduce computational requirements in Learned Sparse Retrieval models while maintaining retrieval effectiveness. The variants include Token-Level Sparse Sampling, Vocabulary-Tiled Importance Sampling, and Adaptive Gradient Sampling, which aim to optimize memory usage and speed during training.
PAPER · PDF
Loading PDF...
Key findings
TLSS-Sparton achieves up to 2.1x speedup and 47% memory reduction with minimal accuracy loss.
VTIS-Sparton enables 2.3x speedup for large vocabularies.
AGS-Sparton automatically finds the optimal accuracy-efficiency tradeoff.
Limitations & open questions
The study focuses on computational and memory efficiency, with less exploration on the impact of these methods on model generalization.