Imitation learning (IL) enables robots to acquire autonomous skills from human demonstrations. Recent in-context IL (ICIL) methods leverage off-the-shelf large language models (LLMs) as instant policies that infer task context and generate executable trajectories from only a few demonstrations, without retraining model parameters. However, the reliability of LLM-based instant policies is limited by hallucinations, which produce outlier trajectories inconsistent with the demonstrations. This issue is further amplified in multimodal tasks, where multiple distinct yet equally valid strategies exist in human demonstrations. To address these challenges, we propose the Robust Instant Policy (RIP) framework, which improves reliability by robustly aggregating multiple trajectories sampled from an LLM. As a concrete instantiation, we introduce Student’s-t mixture model RIP (SMM-RIP), which captures trajectory multimodality via an SMM and extracts a robust representative trajectory using Student’s-t regression, effectively suppressing hallucinated outliers while preserving valid strategies. Experiments on 15 everyday manipulation tasks in both simulated and real-world environments show that SMM-RIP consistently outperforms state-of-the-art IL methods, particularly in low-data regimes. These results highlight the importance of robustness and flexible multimodal modeling for reliable LLM-based ICIL.
Fig1. Overview of the SMM-RIP applied in a pick-hanger-rack task. Human demonstrations provide paired visual observations and corresponding gripper trajectories from which contextual text tokens are generated. An LLM-based instant policy utilizes the contextual data and new image keypoints to sample multiple candidate trajectories. Using a Student’s-t mixture model followed by Student’s-t regression, the SMM-RIP captures multimodal motion patterns in LLM-sampled trajectories while suppressing outlier behaviors, yielding a robust and reliable trajectory for task execution.
This study makes the following key contributions:
We demonstrate that LLM-based instant policies are inherently prone to hallucinations and that the multimodality of human demonstrations further amplifies this issue.
We introduce the RIP framework, implemented as the SMM-RIP, which aggregates multiple LLM-sampled trajectories while flexibly capturing multimodality and maintaining robustness to hallucinations, thereby yielding reliable trajectories.
We empirically validate the SMM-RIP across a diverse set of everyday tasks, demonstrating its superior robustness and flexibility.
Evaluation
Environments and Tasks
To evaluate the SMM-RIP, a set of 15 everyday manipulation tasks was defined across both simulated and real-world environments, as shown in fig 2.
These tasks were designed to reflect a broad range of household-level manipulations while enabling controlled evaluation of multimodal demonstrations, where multiple distinct optimal trajectories can achieve the same goal.
For simplicity, we focused on multimodal scenarios with up to two optimal trajectories per task, because this reflects the typical data limitation of few-shot demonstrations.
Fig2. Environments and tasks. Human demonstrations were collected in both simulated and real-world environments using a leader–follower robot setup, where a human-operated leader robot was mirrored by a follower robot with a similar embodiment.
Under these settings, the SMM-RIP and baseline imitation learning methods were evaluated on 15 everyday manipulation tasks to assess their robustness and flexibility.
The figure illustrates the simulated (top) and real-world (bottom) setups, each including examples of unimodal (left) and multimodal (right) tasks.
Comparison Methods
Diffusion Policy (DP)[Chi et al.]:
A state-of-the-art IL algorithm that learns robust visuomotor policies through diffusion-based action modeling, demonstrating strong learning efficiency and generalizability across diverse manipulation tasks.
Keypoint Action Tokens (KAT) [Di Palo et al.]:
A state-of-the-art ICIL algorithm that leverages an LLM as an instant policy by representing visual observations and robot actions as discrete keypoint tokens, as described in Section Method:KAT.
KAT-DP[Di Palo et al.]:
A hybrid approach combining KAT and DP. It adopts the keypoint-based observation–action space from KAT while maintaining the diffusion-based action generation of DP, achieving a performance comparable to that of KAT.
Gaussian Neural Network RIP (GNN-RIP):
An ablated variant of our proposed framework in which a standard Gaussian distribution replaces the Student’s t-distribution
(i.e., 𝒩(^at | μθ(t), σθ2(t))), thus removing the robustness property of the Student’s t formulation.
Student’s t Neural Network RIP (SNN-RIP)[Oh et al.]:
Another ablated variant of the SMM-RIP that excludes the mixture modeling component. It performs robust trajectory regression using a single-mode Student’s t-distribution, corresponding to a recent study.
Qualitative results: Simulation
Teleoperation System for Demonstration
Simulation Unimodal Task: Pick Banana
Success condition: at the end of the episode, the banana is picked up
KATFailure
GNN-RIPFailure
SNN-RIPSuccess
SMM-RIPSuccess
Simulation Unimodal Task: Push Cube
Success condition: at the end of the episode, the cube should be pushed out to the goal circle
KATFailure
GNN-RIPFailure
SNN-RIPSuccess
SMM-RIPSuccess
Simulation Unimodal Task: Open Drawer
Success condition: at the end of the episode, the drawer is opened
KATFailure
GNN-RIPFailure
SNN-RIPSuccess
SMM-RIPSuccess
Simulation Unimodal Task: Close Cabinet
Success condition: at the end of the episode, the cabinet is closed completely
KATFailure
GNN-RIPFailure
SNN-RIPSuccess
SMM-RIPSuccess
Simulation Unimodal Task: Turn Faucet
Success condition: at the end of the episode, the faucet is rotated
KATFailure
GNN-RIPFailure
SNN-RIPSuccess
SMM-RIPSuccess
Simulation Multimodal Task: Draw Triangle
Success condition: at the end of the episode, the triangle should be accurately drawn
KATFailure
GNN-RIPFailure
SNN-RIPFailure
SMM-RIPSuccess
Simulation Multimodal Task: Push Pull Cube
Success condition: at the end of the episode, the cube should be outside of the goal area
KATFailure
GNN-RIPFailure
SNN-RIPFailure
SMM-RIPSuccess
Qualitative results: Real-world
Teleoperation System for Demonstration
Real Unimodal Task: Pick Banana
Success condition: at the end of the episode, the banana is picked up
KAT Failure
GNN-RIP Failure
SNN-RIP Success
SMM-RIP Success
Real Unimodal Task: Hanger
Success condition: at the end of the episode, the hanger is hung on the stand
KAT Failure
GNN-RIP Failure
SNN-RIP Success
SMM-RIP Success
Real Unimodal Task: Cup Upright
Success condition: at the end of the episode, the cup is vertically placed on the table
KAT Failure
GNN-RIP Failure
SNN-RIP Success
SMM-RIP Success
Real Unimodal Task: Sweep
Success condition: at the end of the episode, the dust (cubes) is on the dust pan
KAT Failure
GNN-RIP Failure
SNN-RIP Success
SMM-RIP Success
SMM-RIP Success
SMM-RIP Success
Real Unimodal Task: Pick and Place
Success condition: at the end of the episode, the bottle is in the bowl
KAT Failure
GNN-RIP Failure
SNN-RIP Success
SMM-RIP Success
SMM-RIP Success
SMM-RIP Success
Real Multimodal Task: Wipe Plate
Success condition: At the end of the episode, the sponge should have wiped along the entire contour of the plate, maintaining contact throughout, without significantly moving the plate
KAT Failure
GNN-RIP Failure
SNN-RIP Failure
SMM-RIP Success
SMM-RIP Success
Real Multimodal Task: Pick Hanger
Success condition: At the end of the episode, the hanger should be pick up by any side
KAT Failure
GNN-RIP Failure
SNN-RIP Failure
SMM-RIP Success
SMM-RIP Success
Real Multimodal Task: Place Cube
Success condition: At the end of the episode, the cube should be placed in any cup
KAT Failure
GNN-RIP Failure
SNN-RIP Failure
SMM-RIP Success
SMM-RIP Success
Quantitative Results
How does the SMM-RIP compare with state-of-the-art IL and instant policy methods in terms of flexibility for multimodal tasks and robustness against LLM-induced hallucinations?
The results of the quantitative analysis comparing the policy success rates of each method are presented as below:
Result 1.
Comparison of SMM-RIP with baseline methods using
10 demonstrations per task.
(a)Top: Simulation | Bottom: Real world.
Success rates are measured over
50 executions (simulation) and
10 executions (real world).
Missing bars indicate zero success.
(b) Average success rates on
unimodal, multimodal, and all tasks
(simulation + real world).
Shaded regions show mean ± standard deviation.
Methods marked with
*
perform significantly worse than SMM-RIP
(t-test, p < 5×10−2).
Takeaway.
SMM-RIP consistently achieves the highest success rates by
robustly aggregating multiple LLM-generated trajectories and explicitly
handling multimodal solutions.
This enables reliable few-shot robot execution without retraining,
even in the presence of LLM hallucinations.
Result 2.
Success rates as a function of the number of demonstrations
(3, 5, 10, 15, 20) in simulation experiments.
All curves report the mean success rate (solid line)
with standard deviation (shaded region),
measured over 50 executions per task.
Takeaway.
SMM-RIP consistently achieves the highest success rates across all data regimes,
with the largest gains in the low-data setting (3–5 demonstrations).
While unimodal aggregation degrades on multimodal tasks,
SMM-RIP remains robust by explicitly modeling multiple valid strategies,
highlighting the importance of multimodal, outlier-robust aggregation when demonstrations are scarce.
Which RIP configurations yield the best performance?
To evaluate the contribution of each design component in the SMM-RIP, a quantitative analysis was conducted on four factors: the number of queries (Q), degrees of freedom (ν), number of mixture components (M), and the effect of applying SNN-RIP regression after SMM.
Unimodal Tasks
Multimodal Tasks
All Tasks
Result 3.
Design analysis of RIP variants highlighting three key factors:
the Student’s-t degrees of freedom (ν),
the number of trajectory queries (Q),
and the number of mixture components (M). Top → Bottom:
Success rates under different configurations of
ν = {0.1, 0.5, 1.0, 1.5, 3.0, ∞},
Q = {3, 5, 10},
and M = {1, 3, 5}.
Results report the mean success rate with
standard deviation,
evaluated over 50 simulation runs per task
using 20 demonstrations.
Takeaway.
The analysis shows that heavy-tailed modeling (ν < ∞)
and sufficient trajectory sampling (Q ≥ 5)
are critical for suppressing hallucinated outliers,
while multiple mixture components (M > 1)
are necessary to capture distinct valid strategies.
Together, these results justify the design choices of SMM-RIP.
BibTeX
@inproceedings{oh2025rip,
author = {Oh, Hanbit and Salcedo-Vázquez, Andrea M. and Ramirez-Alpizar, Ixchel G. and Domae, Yukiyasu},
booktitle = {2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
title = {Robust Instant Policy: Leveraging Student’s t-Regression Model for Robust In-context Imitation Learning of Robot Manipulation},
year = {2025},
pages = {7973--7980},
doi = {10.1109/IROS60139.2025.11247005},
}
