Beyond RGB-D: Perception of Glass, Mirrors, and See-Through Scenes for Robotics

Abstract

Perception of transparent and reflective objects remains one of the most challenging problems in robotics because such materials often violate the assumptions used by conventional vision and depth sensing systems. Reflections, refractions, and missing depth information frequently lead to unreliable scene understanding, affecting important robotic tasks such as navigation, mapping, obstacle avoidance, and manipulation.

This paper presents a survey of perception techniques designed for transparent, reflective, and see-through environments, with a focus on robotics-oriented perception and decision-making. The survey reviews recent developments in transparent object segmentation, depth completion, multimodal sensing, polarization-based geometry recovery, neural rendering, event-based vision, foundation-model-based depth estimation, uncertainty estimation, sim-to-real transfer, and embodied robotic intelligence.

Rather than treating perception tasks separately, this survey emphasizes a unified perception pipeline perspective in which sensing, scene understanding, reasoning, and task-level interaction are treated as interconnected stages. By combining insights from recent research, the paper provides a structured overview of current progress, identifies major research gaps, and highlights future directions toward building more robust, adaptive, and reliable robotic perception systems for real-world transparent and reflective environments.