Talk-to-Robot
Tested where a language model goes wrong when it turns spoken robot commands into target positions, separately from the robot's own controller.
UC San Diego team project (CSE 190)
End-to-end success falls from 98% to 50% while policy success holds at 93 to 100%
- Python
- Gemini
- MuJoCo
- Gymnasium
- Stable-Baselines3 (SAC + HER)
Read the details about Talk-to-Robot
Problem
Natural-language robot commands fail when spatial grounding is mixed with control, making it hard to see where LLM understanding breaks.
What I built
Team project (CSE 190) with a decoupled LLM grounder and SAC+HER controller in MuJoCo FetchPush-v4. Benchmarked a regex baseline against zero-shot, few-shot and chain-of-thought prompting across five instruction tiers, from literal coordinates to functional intent, then retrained the controller to test which grounding failures were recoverable.
Stack
- Python
- Gemini
- MuJoCo
- Gymnasium
- Stable-Baselines3 (SAC + HER)
Result
End-to-end success falls from 98% on literal coordinates to 50% on functional intent, while policy success held between 93 and 100%, which places the failures in grounding rather than control. Each tier fails differently rather than degrading smoothly: relative offsets land goals off the table, where a plain regex beat the LLM 85% to 77%; reference objects are off by a consistent 6 cm bias that controller retraining can absorb; and functional intent is not a coordinate problem at all, since annotators disagreed with each other about as much as the model did.
