Grounding Large Language Models (LLMs) in Agency Standards: A Controlled Comparison of Closed-Book, Retrieval-Augmented, and Full-Document Question Answering for Construction Inspection

Abstract

Objectives: Construction inspectors on transportation projects must verify work against a fragmented body of specifications under schedule pressure. Transportation agencies are now deciding how, and whether, to deploy Large Language Models (LLMs) to support this task. This study examines whether an LLM must be grounded in agency documents at all, and if so, whether grounding should take the form of Retrieval-Augmented Generation (RAG) or simply providing the full source document within the model's context window.

Methods: The effect of document grounding was isolated through a controlled, within-question comparison. Three contemporary LLMs from different model families each answered 70 validated inspection questions, drawn from seven public Connecticut Department of Transportation (CTDOT), Occupational Safety and Health Administration (OSHA), and Federal Highway Administration (FHWA) documents, under three conditions: closed-book, RAG, and full-document context. Responses were scored with token-level F1, ROUGE-L, and semantic similarity, together with an LLM-judge rubric validated against blinded ratings by a licensed professional engineer. Consistency across systems was quantified with the Gini coefficient.

Findings: Grounding proved decisive: relative to closed-book prompting, RAG raised the judge composite by 1.31 to 1.48 points on the five-point scale and full-document context by 1.40 to 1.66 points ( p < 0.000001), while 56 to 64 percent of closed-book answers were judged factually unacceptable. The two grounding strategies were comparable, with full-document context ahead by average 0.14 points despite requiring roughly 35 times the input tokens. Grounding gains were largest for project-specific documents and smallest for widely published standards (r = -0.83).

Novelty: This is the first controlled study to isolate the causal effect of document grounding on transportation inspection question answering, and to compare RAG against full-document prompting for agency standards.

Practical Applications: The findings offer agencies evidence-based guidance on when retrieval infrastructure is warranted for inspection support and when simpler deployment suffices.