Robots Won't Do Chores Until Language Learns to Point

Ask a robot to put the napkins on the couch, and it might grab the same napkin four times. Not because it can't plan — GPT-5.2 is in there trying — but because natural language has no way to say which napkin. "The napkin on the table" refers to all four of them. "The top-left napkin" sounds precise, but the model doesn't know where "top-left" is in its own visual field.

A team from Korea University, Microsoft Research, and the University of Wisconsin-Madison has built a benchmark designed to measure exactly that failure. GroundedPlanBench, posted to arXiv on March 13, 2026, evaluates vision-language models on 308 real-world household scenes from the DROID dataset — 1,009 tasks total, ranging from one action to twenty-six. The benchmark does something existing tests don't: it evaluates planning and spatial grounding jointly. Most prior benchmarks check whether a model can plan a sequence of actions, or whether it can identify where something is in an image. GroundedPlanBench asks both at once, the way a real robot in a real kitchen has to.

The DROID dataset, the data source underlying the benchmark, is itself notable. Collected across 564 scenes and 84 tasks by 50 data collectors in North America, Asia, and Europe over 12 months, it contains 76,000 demonstration trajectories — 350 hours of people doing things with robots. The author list on the DROID paper runs past 100 names and includes Chelsea Finn of Stanford and Sergey Levine of Berkeley. That depth of authorship is not academic vanity; it reflects a dataset built to capture the messiness of real homes, not the clean geometry of a simulation.

The four actions in GroundedPlanBench are grasp, place, open, close — tied to specific locations in the image. Tasks are annotated under both explicit instructions ("pick up the cup on the left and place it in the bin on the right") and implicit ones ("put the dishes away"). The difference matters enormously. Explicit instructions tell the robot where to act. Implicit instructions require the robot to figure it out, and that is where performance collapses.

On short-explicit tasks, models do reasonably well. Qwen3-VL-4B, when fine-tuned with V2GP — a four-stage framework the researchers built to train models on spatially grounded planning — jumped from 39.5 percent to 58.2 percent task success rate, according to the arXiv paper. That result even surpassed the performance of the 32-billion-parameter version of the same model on that setting. But long-horizon implicit tasks tell a different story. Every model degrades as tasks get longer and instructions get vaguer. Spatially grounded long-horizon planning, the researchers write, "remains a major bottleneck for current VLMs."

The napkin failure case is the clearest illustration of why. When asked to place four napkins on a couch, Qwen3-VL-4B referred to each napkin identically as "the napkin on the table," causing every grasp action to target the same physical napkin. GPT-5.2 tried to be more descriptive — "top-left napkin," "upper-center napkin" — but these spatial adjectives were still too imprecise for the model to distinguish between four identical objects. Both approaches failed the same way, at the same place.

V2GP — Video-to-Spatially Grounded Planning — is the researchers' proposed fix. The framework trains on 43,000 grounded plans generated from robot videos: 34,646 short sequences of one to four actions, 4,368 medium sequences of five to eight, and 4,448 long sequences of nine to twenty-six. The four stages are temporal decomposition (breaking a video into action chunks using gripper signals), interactive object identification (finding which objects the robot touches), spatial grounding via the Segment Anything Model 3, and finally spatially grounded task planning with explicit and implicit instruction variants.

The real-world validation used a Franka Research 3 robot with calibrated Intel RealSense D435i RGB-D cameras in an eye-to-hand configuration — the researchers tested whether plans generated by the models could actually execute on hardware, not just score well in simulation.

The benchmark's framing is precise: it measures whether VLM-generated plans correspond to physically feasible robot manipulation behaviors. That is a narrower claim than "this robot can do chores." And the gap between the two is the whole story.

What GroundedPlanBench is really measuring is the brittleness of natural language as a spatial interface. Humans solve the napkin problem unconsciously — we look, we point, we gesture, we use context. A robot receiving an instruction has none of that. The phrase "the cup next to the plate" requires the robot to first identify the plate, then find the cup, then determine what "next to" means in pixels. Get any of those wrong and the robot does something baffling to a human watching.

The authors are Sehun Jung, HyunJee Song, and Dong-Hee Kim of Korea University; Reuben Tan and Jianfeng Gao of Microsoft Research; Yong Jae Lee of UW-Madison; and Donghyun Kim of Korea University, corresponding author. GroundedPlanBench is a research artifact — a benchmark, not a product. But benchmarks are how a field measures itself, and this one quantifies a problem researchers have been working around: the robot can plan. It cannot specify where.

The DROID dataset is research data — curated, cleaned, collected by people who knew what they were looking for. Real homes are messier, darker, and full of objects the dataset never saw. A model that scores well on GroundedPlanBench is not a robot ready for deployment. It is a model that has learned to fail less catastrophically on a curated benchmark. The napkin problem, in other words, is not solved. It is measured.