Evaluating Interactive Reasoning in Large Language Models: A Hierarchical Benchmark with Executable Games
arXiv:2606.00103v1 Announce Type: new Abstract: We introduce a multi-turn interactive framework for reasoning evaluation that treats reasoning as active evidence acquisition and belief updating. Wherein, LLMs receive only the task rules, must issue targeted queries to a hidden environment, integrate partial observations over time, and decide when to submit a final answer. Beyond standard success rate and interaction efficiency, we evaluate contextual robustness under controlled contextual perturbations, and metacognitive adaptation through counterfactual revision and necessity judgment. We ins
The rapid advancement and widespread deployment of Large Language Models necessitate more sophisticated and interactive evaluation frameworks to understand their true capabilities and limitations beyond simple benchmarks.
This new interactive reasoning benchmark provides a crucial, granular method for assessing the sophisticated cognitive processes within LLMs, moving beyond mere output scoring to evaluate their ability to acquire evidence and adapt.
The shift to an interactive evaluation framework for LLMs means that future development and competition will increasingly focus on model adaptability, evidence acquisition, and metacognitive abilities, rather than just raw performance on static tasks.
- · AI research institutions specializing in advanced reasoning
- · Developers of highly adaptive and interactive LLMs
- · Platforms providing complex, dynamic environments for AI training
- · LLMs optimized only for static, single-turn tasks
- · Evaluation frameworks focusing solely on superficial task completion
- · Developers unprepared for rigorous, interactive model assessments
This benchmark will drive the development of LLMs with enhanced capabilities in active learning and dynamic problem-solving.
Improved interactive reasoning could accelerate the deployment of more robust and autonomous AI agents in complex, real-world scenarios.
The insights gained from these evaluations may inform new architectural designs for AI, mimicking how biological systems acquire and process information.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.AI