A Hierarchical YOLO-Based Pipeline for Document Analysis in Natural Images

Abstract

Extracting structured information from documents embedded in natural images remains challenging due to diverse layouts and environmental noise. We propose a hierarchical three-stage deep learning pipeline for robust document understanding: (1) document localization via instance segmentation, (2) layout analysis through semantic block-level detection, and (3) text transcription using Optical Character Recognition (OCR). We systematically benchmark recent YOLO variants (YOLOv5, v8, v11, and v12) on the localization and layout-analysis stages. Our results show that YOLOv11-Seg achieves a mask mAP of 0.935 for localization, while YOLOv11-Det attains a box mAP of 0.671 for layout analysis. In the transcription stage, EasyOCR provides strong character recovery, whereas Tesseract offers higher word-level accuracy and lexical coherence for structured images. A primary contribution is a new menu-image dataset with complex semantic structures, establishing a resource for evaluation and benchmarking. Together, our findings set a high-performance baseline for document analysis and highlight performance differences.