RGB versus Early-Fusion RGB-D Glass Segmentation

A Controlled Benchmark-to Robot Evaluation on Pioneer P3-DX

Abstract

Transparent glass is a persistent perception hazard for indoor mobile robots: RGB boundaries can be visually ambiguous, while commodity depth sensors often return missing or distorted measurements. This study isolates the effect of adding raw depth to a lightweight transformer segmenter. Two SegFormer B0 models were trained on identical partitions of the 3,009-image RGB-D Glass Surface Detection dataset using the same optimization and checkpoint-selection protocol: a three-channel RGB model and a four channel early-fusion RGB-D model. On the fixed 609-image test split, RGB outperformed RGB-D on seven of eight reported measures, reaching 0.9448 pixel accuracy, 0.8234 mean intersection over union, 0.7108 glass IoU, 0.8376 recall, and 0.8310 F1. RGB-D retained higher precision (0.8374 versus 0.8244). RGB also reduced mean inference latency by 10.7% (10.24 versus 11.47 ms) and increased throughput by 12.0% (97.63 versus 87.18 frames/s). The frozen checkpoints were then evaluated in 20 Pioneer P3-DX navigation runs. At the selected center-ratio threshold of 0.35, each model produced glass-triggered stops in both of its two glass runs (4/4 combined). Across all recorded thresholds, RGB triggered in 5/5 glass runs and RGB-D in 3/5; the independent distance safety stop terminated the remaining two RGB-D runs. Neither model produced a glass trigger in the ten non-glass runs. Under this controlled early-fusion design, raw depth did not improve unseen-test segmentation and reduced threshold robustness during robot deployment.