Automated Multi-Class Structural Damage Detection and Segmentation for Unmanned Aerial Vehicle-Based Infrastructure Inspection Using Deep Learning

Abstract

This study presents a three-stage deep learning framework for automated structural damage classification, segmentation, and unmanned aerial vehicle (UAV)-based inspection simulation targeting bridges, offshore platforms, and other civil infrastructure. Stage 1 fine-tunes ResNet50 with differential learning rates and class weighting to classify crack, corrosion, spalling, and intact surfaces, achieving 85.69% accuracy on a held-out test set of 1,600 images drawn from multi-source public datasets, with the differential learning rate strategy contributing an 8.25% improvement over the frozen-backbone baseline. A targeted augmentation strategy addresses minority-class imbalance and yields a spalling F1-score of 0.886 under the balanced evaluation protocol. Stage 2 applies DeepLabV3+ with a partially unfrozen ResNet50 backbone for pixel-level damage localization, achieving mean intersection over union (mIoU) values of 0.7890 for three-class segmentation and 0.6446 for four-class segmentation, the latter incorporating 315 real UAV crack images to extend the training distribution to the aerial perspective absent from most public crack datasets. Stage 3 integrates frame-level inference into a boustrophedon UAV grid inspection simulation that produces a severity-coded damage map and a structured JSON inspection report, identifying 16 of the 20 damaged zones. Average CPU inference time is 16.29 ms per frame (61 frames per second), consistent with near-real-time deployment on modest hardware. Ablation studies of backbone unfreezing depth and Grad-CAM visualizations provide additional support for the robustness and interpretability of the proposed pipeline. The framework demonstrates that multi-class damage recognition, pixel-level localization, and inspection workflow integration can be combined in a single reproducible system built entirely on open data.