On-The-Fly Information Retrieval Augmentation for Language Models