This literature map examines 25 papers on small language models (≤7B parameters) and their inability to effectively utilize retrieved contextual information. Synthesizing evidence from retrieval-augmented generation, parameter-efficient fine-tuning (LoRA/QLoRA), and knowledge distillation research, it identifies a critical utilization bottleneck where small models fail 85-100% of the time even with oracle retrieval. The report provides a taxonomy of adaptation methods and highlights hybrid RAFT-style approaches as promising solutions while identifying evidence gaps in sub-1B model studies.
Key findings
Models ≤7B parameters fail to extract correct answers 85-100% of the time on questions requiring external knowledge, even with perfect retrieval
Adding retrieval context destroys 42-100% of answers the model previously knew correctly, indicating a severe 'distraction effect'
Parameter-efficient methods (LoRA, QLoRA) enable adaptation of small models with minimal computational overhead (0.1-1% trainable parameters)
Hybrid approaches combining fine-tuning with RAG (RAFT-style methods) show promise for improving contextual utilization
Context utilization capability emerges primarily above 10B parameters, posing fundamental challenges for small model deployment
Limitations & open questions
Limited studies available on sub-1B parameter models
Insufficient evaluation of hybrid fine-tuning + RAG approaches at small scales
Lack of standardized benchmarks for context utilization specifically