A National University of Singapore team has built MIGU, a robot-planning framework that combines spoken instructions with the uncertainty in a person's pointing gesture. It identified the intended target correctly in 83% of 100 real-world scenarios. In a separate robot-planning test, allowing one clarification question raised success from 83.3% to 91.7% when language and pointing still left the target uncertain.
A request such as “put this over there” carries two different kinds of information. Language and visual context suggest which objects or regions make sense, while a finger supplies a geometric direction. Either cue can be wrong or imprecise: vision-language models can be overconfident, and a small hand-direction error expands into a large positional error at distance.
One belief from language and pointing
MIGU keeps those sources separate long enough to model their weaknesses. Its geometric component starts from the estimated eye and fingertip positions, then propagates viewing-direction, depth and hand-estimation uncertainty into a likelihood over 3D scene points. In parallel, a vision-language model proposes candidate objects or regions and assigns relative semantic confidence.
A Bayes-inspired fusion step multiplies the geometric likelihood by the semantic score for each candidate, producing a normalized belief over possible targets. This lets a table remain plausible because it fits the instruction even when the pointing ray also passes near the floor, or lets a pointed object outrank another item that better matches the words alone.
The planner compares direct execution with the expected value of asking a binary question about color, size, shape or relative position. It asks only when the best question is worth more than acting immediately after accounting for an interaction cost. The study permits one clarification round; the answer updates the belief before the robot commits to a target.
Identifying the target before planning a movement
The authors collected 100 scenarios with semantic ambiguity, geometric ambiguity or both. Pointing distances ranged from 0.2 to 3 metres. Full MIGU achieved 83% object-level accuracy and placed the intended target in its top two candidates 95% of the time. Removing semantic evidence reduced accuracy to 51%; removing geometry reduced it to 46%.
The probability distribution also mattered. MIGU recorded a Brier score of 0.25, compared with 0.49 without semantics and 0.57 without geometry; lower scores indicate beliefs that better match the observed outcome. Replacing the uncertainty model with simpler distance-to-ray, angle or fixed-threshold rules also weakened either accuracy, probabilistic quality or both.
For mobile manipulation, the team used an Autolife S2 wheeled humanoid on eight tasks repeated three times. Semantic-only and geometry-only grounding produced planning success rates of 45.8% and 41.7%. Fusion without clarification reached 83.3%, and the one-question policy reached 91.7%. Separate Fetch robot demonstrations connected the grounded targets to task-and-motion plans that moved obstacles or selected placement regions, but those examples were not a broader success-rate study.
The evidence remains a small, author-run evaluation. MIGU treats the vision-language model's confidence as a relative score rather than a calibrated probability, assumes the intended target is represented in the detected scene and limits dialogue to one deterministic yes-or-no answer. Its semantic component also used GPT-5.6 Sol, so results do not establish the same behavior with other vision-language models.
Sources
- Lu, Xiao and Hsu, “MIGU: Multimodal Instruction Grounding under Uncertainty for Manipulation Planning,” arXiv:2609.24995 (submitted September 21, 2026)
- Full paper in arXiv HTML, including methods, real-world results and limitations
- Lead image: “Hold my cup, Baxter!” by QuarkyTale, reused under CC BY-SA 4.0 via Wikimedia Commons.