Webpage adapted from nerfies.
CLUE actively resolves contextually uncertain tasks
Abstract
Foundation models provide robots with the ability to interpret natural language and reason about environmental context, yet most language-conditioned policies assume that goals are well-specified and that task-relevant information is provided upfront via a prior map. Operating in unfamiliar environments with underspecified tasks entails high contextual uncertainty—the robot must jointly infer what constitutes task success, what constitutes relevant information, and where (or whether) that information exists. We address these limitations via CLUE (Closed-Loop contextual Uncertainty rEsolution), a framework for actively resolving contextual uncertainty given underspecified tasks in natural language. CLUE uses an LLM-derived policy to hypothesize task-relevant concepts and potential plans. It then uses a language-embedded map, which is constructed online, to ground these hypotheses into actions. The policy sequentially evaluates hypotheses via closed-loop environment interaction and refines its plans as it gathers new information. We deploy CLUE on a Boston Dynamics Spot across three real indoor and outdoor environments spanning 15 tasks that require object disambiguation, functional inference, and occlusion reasoning. CLUE achieves a success rate within 7 percentage points of an oracle policy and outperforms an LLM-enabled planner without closed-loop feedback by a 4x margin. Supporting experiments demonstrate that simply building and then querying a language-enriched map is insufficient to resolve complex contextual planning tasks, which achieves a roughly one third success rate compared to CLUE while requiring over 10x more VLM tokens.
Overview
CLUE uses an LLM to realize underspecified language tasks in a closed-loop manner via two action types - hypothesis grounding and active information gathering. The LLM then constructs actions comprising manipulation, inspection, and navigation to resolve contextual uncertainties and make task progress. Information from the physical world is provided to the LLM via a symbolic graph-based observation space.
Video
BibTeX
@article{ravichandran_contextual_uncertainty,
title={Actively Resolving Contextual Uncertainty for Underspecified Tasks in Natural Language},
author={Zachary Ravichandran and Jonathan Diller and Fernando Cladera and Varun Murali and George J. Pappas and Vijay Kumar}
year={2026},
journal={International Symposium on Robotics Research (ISRR)}
}