Evaluation is problematic

[jiayuww]

Thank you for this work! I may be misunderstanding the evaluation setup, but it seems potentially problematic. From reading the code, my understanding is that these tasks are contextual QA tasks, yet only the question is provided to the model without the accompanying context. Could you please let me know if this interpretation is correct, or if I am missing something?

Thank you very much!

[ViktorAxelsen]

Thank you for your interest and the great question!

You are correct that under the standard evaluation pipeline of these datasets (e.g., RAG-style setups), retrieved context from the corpus is typically provided to the model. In our work, however, the goal is not to evaluate retrieval but to assess whether Router-R1 can effectively leverage the complementary strengths of multiple LLMs.

Providing retrieved evidence that may already contain the answer would significantly reduce the impact of routing, as the final performance would then be dominated by the retrieval component rather than the router’s model selection behavior. By excluding the corpus and using only the question, each LLM must rely on its own internal knowledge and reasoning ability, making differences between models more meaningful for routing.

Furthermore, we agree that this is not the only valid setting, and exploring more reasoning-intensive tasks for multi-LLM routing is a promising direction for future work.