Evaluating and Pricing Advertisements in AI-Generated Responses

• URL: http://arxiv.org/abs/2607.27686v1
• Date: Thu, 30 Jul 2026 05:07:27 GMT
• Title: Evaluating and Pricing Advertisements in AI-Generated Responses
• Abstract summary: Key challenge is click-through intent, behavioural logs are unavailable, human annotation resists calibration, and frontier LLM judges conflate intent with linguistic fluency.<n>We construct missing supervision through a psychologically grounded agent simulation framework, and distil it into a parameter-efficient evaluator that predicts click-through intent, together with the three companion dimensions of ad quality, as smooth, differentiable estimates.<n>We build the pricing layer directly, deriving the unique payment rule under which truthful bidding is optimal, demonstrating it on a best-of-k allocation, and extending the mechanism to non-monotone allocations.
• Score: 5.675676304782969
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As search increasingly shifts toward LLM-driven answer engines, advertising is becoming embedded within the generated response itself and should therefore be evaluated for both user utility and commercial value. The key challenge is click-through intent: behavioural logs are unavailable, human annotation resists calibration, and frontier LLM judges conflate intent with linguistic fluency. These gaps compound, as principled pricing presupposes a continuous intent signal, while generating such a signal presupposes supervision that is currently unavailable. We construct the missing supervision through a psychologically grounded agent simulation framework, and distil it into a parameter-efficient evaluator that predicts click-through intent, together with the three companion dimensions of ad quality, as smooth, differentiable estimates. Validated through sign-certain behavioural perturbations, the evaluator surpasses frontier zero-shot judges on relevance sensitivity (79% versus 60-67%), tracks graded content degradation, generalises without error to 103 fictional products, and agrees with human preference in 86% of pairwise judgements across five annotators, with agreement rising in the evaluator's confidence. Upon its estimates we build the pricing layer directly, deriving the unique payment rule under which truthful bidding is optimal, demonstrating it on a best-of-k allocation, and extending the mechanism to non-monotone allocations. The same differentiable signal stands ready as a training objective for ad generation.

Related papers

• CARRE: Counterfactual Action Retrieval and Reason Evaluation for Explainable Churn Prescription [5.823033362443801]
  We present CARRE, a three-stage framework that combines retrieval-augmented candidate generation, cost-aware counterfactual scoring, and large language model (LLM) reasoning.<n>On the IBM Telco Customer Churn dataset, CARRE achieves 79.8% greater mean model-predicted risk reduction than the plain SHAP baseline.
  arXiv  Detail & Related papers  (2026-09-09T06:08:02Z)
• Rolling With Resistance: Preference-Optimized LLM Counselors Can Trade Goal Persistence for Relational Attunement in Motivational Interviewing [0.6372261626436676]
  In Motivational Interviewing (MI), a client's sustain talk (arguments for the status quo) calls for the counselor to roll with resistance, a move that can fail in two ways.<n>We introduce a two-axis evaluation of counselor responses of Motivational Interviewing Treatment Integrity (MITI) code, Goal Persistence (GP) and Attunement (RA)<n>We ask whether penalizing one failure through preference optimization teaches rolling resistance provokes its opposite.
  arXiv  Detail & Related papers  (2026-07-30T20:16:03Z)
• Cybersecurity Detection Classification with Reasoning-enabled Language Models [50.72675435043546]
  A major issue in Security Operations Centers (SOCs) is alert fatigue.<n>Prior work prompts or fine-tunes large language models (LLMs) to emit a triage label directly, but does not train them to reason about whether a detection is a genuine threat.<n>We train a chain-of-thought (CoT) reasoning-enabled triage classifier on real, human-labeled Windows endpoint detections.
  arXiv  Detail & Related papers  (2026-07-30T16:22:13Z)
• PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception [75.70273182182397]
  PerceptionRubrics is a rubric-based evaluation framework.<n>It addresses the gap between saturated benchmark scores and real-world brittleness.<n>It pairs 1,038 information-dense images with over 10,000 instance-specific rubrics.
  arXiv  Detail & Related papers  (2026-06-26T17:59:15Z)
• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape [79.30826980815927]
  rubrics are explicit criteria sets that transform complex quality judgments into structured and actionable standards.<n>We systematically organize existing rubric designs, examine their construction and optimization, and analyze their role across evaluation and training.
  arXiv  Detail & Related papers  (2026-06-07T13:34:55Z)
• Multi-Dimensional Behavioral Evaluation of Agentic Stock Prediction Systems Using Large Language Model Judges with Closed-Loop Reinforcement Learning Feedback [1.2362187555287152]
  Forecast evaluation in finance has relied on aggregate accuracy metrics and predictive-accuracy tests built on point-forecast errors.<n>We propose a behavioral forecast-evaluation methodology that complements accuracy tests by assessing the intermediate decision process itself.
  arXiv  Detail & Related papers  (2026-05-07T06:31:34Z)
• Are we still able to recognize pearls? Machine-driven peer review and the risk to creativity: An explainable RAG-XAI detection framework with markers extraction [7.723181091241251]
  We introduce an explainable framework (RAG-XAI) for assessing review quality and detecting automated patterns.<n>The framework achieves near-perfect detection performance, with XGBoost, Random Forest and LightGBM reaching 99.61% accuracy, AUC-ROC above 0.999 and F1-scores of 0.9925 on the test set.
  arXiv  Detail & Related papers  (2026-04-09T08:25:49Z)
• Reviewing the Reviewer: Graph-Enhanced LLMs for E-commerce Appeal Adjudication [0.7136933021609076]
  We propose a minimal representation for adjudication reasoning that prevents hallucination through operational grounding.<n>We develop a conflict-aware graph reasoning framework that performs top-down deductive reasoning for new cases.<n>We evaluate the framework in large-scale e-commerce seller appeal adjudication.
  arXiv  Detail & Related papers  (2026-02-27T00:43:59Z)
• Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models [108.26461635308796]
  We introduce Rationale Consistency, a fine-grained metric that quantifies the alignment between the model's reasoning process and human judgment.<n>Our evaluation of frontier models reveals that rationale consistency effectively discriminates among state-of-the-art models.<n>We introduce a hybrid signal that combines rationale consistency with outcome accuracy for GenRM training.
  arXiv  Detail & Related papers  (2026-02-04T15:24:52Z)
• Ad Insertion in LLM-Generated Responses [16.434649348378706]
  We propose a practical framework that resolves tensions through two decoupling strategies.<n>First, we decouple ad insertion from response generation to ensure safety and explicit disclosure.<n>Second, we decouple bidding from specific user queries by using genres'' (high-level semantic clusters) as a proxy.
  arXiv  Detail & Related papers  (2026-01-27T10:16:03Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• A Confidence-Diversity Framework for Calibrating AI Judgement in Accessible Qualitative Coding Tasks [0.0]
  Confidence-diversity calibration is a quality assessment framework for accessible coding tasks.<n>Analysing 5,680 coding decisions from eight state-of-the-art LLMs, we find that mean self-confidence tracks inter-model agreement closely.
  arXiv  Detail & Related papers  (2025-08-04T03:47:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.