How Many Trials Back a Force-Aware Manipulation Claim?

A Statistical Audit of 2026 Real-Robot Evaluations

Abstract

Papers on manipulation with force, torque, tactile or motor-current feedback usually support their claims with real-robot success rates measured over a handful of trials. We audited how much statistical evidence those numbers carry. Following a protocol frozen before any data were extracted, we screened 89 arXiv papers retrieved by six fixed queries (first posted January–September 2026), included 58, and extracted 377 binary comparisons of the proposed method against its strongest baseline or against its own ablation without the force/touch signal (337 in the primary set). A second, blinded extraction of every paper, by separate instances of the AI assistant used for the first, checked the data before analysis, and the author re-coded a seeded random 20% of papers by hand (19 of 19 inclusion decisions and 100% of 88 binary rows matched). The median condition was evaluated on 20 trials (interquartile range 15–30); 17% of papers reported any interval or error bar on success, 5% any significance test and 5% the number of training seeds. The proposed method scored higher in 91% of comparisons, but the 95% interval for the difference excluded zero in only 53% (49% against the strongest baseline). After a Holm correction within each paper, 23% of papers had every headline comparison resolved. At the reported trial counts the median comparison could detect, with 80% power, only a gap of about 34 percentage points; only 6% of comparisons had 80% power to detect a true 20-point improvement (median power 0.19). In simulation, two equally good methods compared on 10 trials each show an apparent gain of 10 points or more 41% of the time. None of this says the methods do not work: most reported gaps are large and in the expected direction. It says that the field’s typical evaluation cannot distinguish a real 20-point improvement from noise, and we give a short checklist and a sequential-testing illustration that would cost few additional trials. All data, code and the frozen protocol are released.