Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty

• URL: http://arxiv.org/abs/2608.25660v1
• Date: Wed, 26 Aug 2026 11:42:12 GMT
• Title: Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty
• Abstract summary: Think-Probe-Respond (TPR) is a lightweight approach that probes latent novelty judgments from hidden states during the reasoning phase and uses the probed judgments to condition the final response.<n>Across strong baselines, TPR improves novelty judgment performance by 22.30% and successfully mitigates the prevalent "medium novelty" bias.
• Score: 17.10692838025252
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automated novelty judgment can accelerate scientific discovery by enabling efficient evaluation, refinement, and comparison of research ideas. While large language models are increasingly adopted for this task, we investigate a previously overlooked limitation in their judgment capabilities: despite generating reasoning rationales that closely mirror those of human experts, their final novelty judgments often diverge substantially. We demonstrate that this miscalibration stems from a systematic bias towards judging ideas as "medium novel". To mitigate this, we propose Think-Probe-Respond (TPR), a lightweight approach that probes latent novelty judgments from hidden states during the reasoning phase and uses the probed judgments to condition the final response. Across strong baselines, TPR improves novelty judgment performance by 22.30% and successfully mitigates the prevalent "medium novelty" bias.

Related papers

• Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews [69.66583722746904]
  We introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions.<n>We propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews.<n>Our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy.
  arXiv  Detail & Related papers  (2026-04-21T14:21:15Z)
• Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas [5.4020309622966325]
  RINoBench is the first comprehensive benchmark for large-scale evaluation of research idea novelty judgments.<n>It comprises 1,381 research ideas derived from and judged by human experts as well as nine automated evaluation metrics.<n>Using this benchmark, we evaluate several state-of-the-art large language models on their ability to judge the novelty of research ideas.
  arXiv  Detail & Related papers  (2026-03-11T00:54:10Z)
• What Is Novel? A Knowledge-Driven Framework for Bias-Aware Literature Originality Evaluation [4.14197005718384]
  We introduce a literature-aware novelty assessment framework that learns how humans judge novelty from peer-review reports.<n>Using nearly 80K novelty-annotated reviews from top-tier AI conferences, we fine-tune a large language model to capture reviewer-aligned novelty evaluation behavior.
  arXiv  Detail & Related papers  (2026-01-14T16:49:39Z)
• Re-Rankers as Relevance Judges [65.37611299805856]
  We reproduce re-rankers in a re-ranker-as-relevance-judge setup.<n>We perform experiments on TREC-DL 2019 to 2023 with 8 re-rankers from 3 families, ranging from 220M to 32B, and analyse the evaluation bias exhibited by re-ranker-based judges.
  arXiv  Detail & Related papers  (2026-01-08T00:02:59Z)
• CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards [72.44810390478229]
  CompassJudger-2 is a novel generalist judge model that overcomes limitations via a task-driven, multi-domain data curation strategy.<n> CompassJudger-2 achieves superior results across multiple judge and reward benchmarks.
  arXiv  Detail & Related papers  (2025-07-12T01:34:24Z)
• Literature-Grounded Novelty Assessment of Scientific Ideas [23.481266336046833]
  We propose the Idea Novelty Checker, an LLM-based retrieval-augmented generation framework.<n>Our experiments demonstrate that our novelty checker achieves approximately 13% higher agreement than existing approaches.
  arXiv  Detail & Related papers  (2025-06-27T08:47:28Z)
• Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Models [130.5487886246353]
  Extending thinking traces using prompts like "Wait" or "Let me rethink" can improve performance.<n>This raises a natural question: Does thinking more at test-time truly lead to better reasoning?<n>We show a consistent pattern of initial performance improvements from additional thinking followed by a decline, due to "overthinking"
  arXiv  Detail & Related papers  (2025-06-04T17:55:09Z)
• Re-evaluating Open-ended Evaluation of Large Language Models [50.23008729038318]
  We show that the current Elo-based rating systems can be susceptible to and even reinforce biases in data, intentional or accidental.<n>We propose evaluation as a 3-player game, and introduce novel game-theoretic solution concepts to ensure robustness to redundancy.
  arXiv  Detail & Related papers  (2025-02-27T15:07:47Z)
• Socratic Reasoning Improves Positive Text Rewriting [49.32560132826547]
  Reframing a negative into a positive thought is at the crux of several cognitive approaches to mental health and psychotherapy.<n>We develop a novel framework called textscSocraticReframe to rationalize the thought rewriting process.
  arXiv  Detail & Related papers  (2024-03-05T15:05:06Z)
• Probabilistically Robust Recourse: Navigating the Trade-offs between Costs and Robustness in Algorithmic Recourse [34.39887495671287]
  We propose an objective function which simultaneously minimizes the gap between the achieved (resulting) and desired recourse invalidation rates. We develop novel theoretical results to characterize the recourse invalidation rates corresponding to any given instance. Experimental evaluation with multiple real world datasets demonstrates the efficacy of the proposed framework.
  arXiv  Detail & Related papers  (2022-03-13T21:39:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.