Leveraging Retrieval-Augmented Prompting for Enhanced Comment Feedback Prediction with Large Language Models

Abstract

User comment feedback prediction is an important problem in online platforms, which have impact on the automatic moderation, quality evaluation, and user experience enhancement etc. Despite strong general reasoning capabilities of large language models (LLMs) such as GPT-5, Claude, Gemini and Qwen3-7B, their performance on domain restriction tasks can vary significantly and is commonly very sensitive to context provided and external knowledge. This paper presents an innovative Retrieval-Augmented Prompting (RAP) framework for comment feedback prediction. Our approach incrementally improves the quality of LLM prompts by retrieving k of the semantically most similar historical comment-feedback pairs from an external dataset as in-context few-shot examples. We experiment on the specialized dataset of 50000 comment ratings, obtained from various online scenarios. Experimental results show that our GPT-5 + RAP model outperforms state-of-the-art LLMs such as Qwen3-7B, Claude, Gemini and GPT-5 baseline on accuracy, Macro-F1 and explanation consistency and a strong prompt engineered baseline, GPT-5 + PE.