A Demand-Side Benchmark for Consumer-Facing Construction Cost Questions: Price-Figure Span, Output Consistency, and the Case for a Verifiable Reference Layer

Abstract

Consumers facing home-renovation quotes operate in a classic credence-goods market: they cannot readily verify whether a quoted price is fair, and general-purpose large language models (LLMs) are now a zero-cost place to ask. Whether LLM answers are actionable for this purpose is untested. Demand-side benchmarks exist for medical, legal, and financial advice, but not for construction costs. We present, to our knowledge, the first consumer-question benchmark for construction costs. Forty Japanese renovation-price questions were posed to frontier LLMs, with repeated-trial sets measuring output stability. A matched re-run at bare provider defaults with a current frontier model (gpt-5.5) was added to remove a settings confound present in the original configuration. Two findings are robust across models, generations, and settings: no LLM answer contained an explicit over-charge decision threshold, and repeated runs of the same question returned materially different price figures. Within-answer price spans are also wide, with a median of 10x under bare defaults. A deterministic structured engine over an open cost database is included as an existence proof that a citable reference layer is constructible. Its consistency is a design property and its accuracy is not validated here; validating it against completed real-world quotations is the next study. All questions, raw outputs, harness, and scoring code are public.