KC-Bench: A Dynamic Interactive Benchmark for Evaluating Knowledge Conflicts in LLM Agents

• URL: http://arxiv.org/abs/2609.03588v1
• Date: Thu, 03 Sep 2026 09:35:14 GMT
• Title: KC-Bench: A Dynamic Interactive Benchmark for Evaluating Knowledge Conflicts in LLM Agents
• Abstract summary: We introduce KC-Bench, a controlled multi-turn benchmark for measuring this capability.<n>Its 238 tasks are manually screened from more than 1,000 generated candidates.<n>No model handles factual correction, identity consistency checking, and temporal conflict resolution reliably across all settings.
• Score: 5.481534695653804
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As LLMs increasingly act through tools, they must reconcile user instructions, parametric knowledge, and dynamic environmental observations before taking actions. We introduce KC-Bench, a controlled multi-turn benchmark for measuring this capability across world-knowledge conflicts, input inconsistencies, and multi-source temporal conflicts. Its 238 tasks are manually screened from more than 1,000 generated candidates and combine a user simulator, stateful tools, deterministic environment assertions, an open-source natural-language evaluator, and human trajectory verification. Evaluation of nine models, including DeepSeek-V4-Flash, GLM-5.2, and MiniMax-M3, shows substantial cross-domain variation: no model handles factual correction, identity consistency checking, and temporal conflict resolution reliably across all settings. In the simulated environments, missed conflicts can propagate to tool calls or synthetic protected-data flows. KC-Bench isolates this model-level behavior rather than ranking complete agent frameworks, and provides a reproducible diagnostic for developing conflict-aware reasoning and execution safeguards.

Related papers

• DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows [69.12339622053588]
  We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions.<n>The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories.<n>We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy.
  arXiv  Detail & Related papers  (2026-08-27T02:36:23Z)
• CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs [100.38986535324284]
  We introduce textbfCIAware-Bench, a benchmark for measuring textbfcontrol textbfintervention (CI) awareness across frontier models.<n>CIAware-Bench tests whether models can distinguish their own trajectories from those modified by a control intervention.
  arXiv  Detail & Related papers  (2026-06-09T16:24:16Z)
• A Unified Framework for the Evaluation of LLM Agentic Capabilities [36.43241368835721]
  We present a unified framework for the fair evaluation of LLM agentic capabilities.<n> Driven by a unified configuration system, the framework integrates diverse benchmarks into a standardized instruction--tool--environment format.<n>We conduct a large-scale empirical analysis over 400K rollouts and 5B tokens on 15 models.
  arXiv  Detail & Related papers  (2026-05-27T03:20:45Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• ClawArena: Benchmarking AI Agents in Evolving Information Environments [61.664633997138004]
  ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
  arXiv  Detail & Related papers  (2026-04-05T17:55:23Z)
• Quantifying Divergence in Inter-LLM Communication Through API Retrieval and Ranking [0.0]
  Large language models (LLMs) increasingly operate as autonomous agents that reason over external APIs to perform complex tasks.<n>We present a unified benchmarking framework to quantify inter-LLM divergence, defined as the extent to which models differ in API discovery and ranking under identical tasks.
  arXiv  Detail & Related papers  (2026-03-09T20:28:24Z)
• ConflictBench: Evaluating Human-AI Conflict via Interactive and Visually Grounded Environments [43.08480724370872]
  We introduce ConflictBench, a benchmark for evaluating human-AI conflict through 150 multi-turn scenarios.<n>We show that ConflictBench integrates a text-based simulation engine with a visually grounded world model, enabling agents to perceive, plan, and act under dynamic conditions.
  arXiv  Detail & Related papers  (2026-03-09T06:59:48Z)
• AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition [72.24180896265192]
  We introduce AgentNoiseBench, a framework for evaluating robustness of agentic models under noisy environments.<n>We first conduct an in-depth analysis of biases and uncertainties in real-world scenarios.<n>We then categorize environmental noise into two primary types: user-noise and tool-noise.<n>Building on this analysis, we develop an automated pipeline that injects controllable noise into existing agent-centric benchmarks.
  arXiv  Detail & Related papers  (2026-02-11T20:33:10Z)
• InspectCoder: Dynamic Analysis-Enabled Self Repair through interactive LLM-Debugger Collaboration [71.18377595277018]
  Large Language Models (LLMs) frequently generate buggy code with complex logic errors that are challenging to diagnose.<n>We present InspectCoder, the first agentic program repair system that empowers LLMs to actively conduct dynamic analysis via interactive debugger control.
  arXiv  Detail & Related papers  (2025-10-21T06:26:29Z)
• Real-Time Anomaly Detection with Synthetic Anomaly Monitoring (SAM) [2.055524866851853]
  Anomaly detection is essential for identifying rare and significant events across diverse domains such as finance, cybersecurity, and network monitoring.<n>This paper presents Synthetic Anomaly Monitoring (SAM), an innovative approach that applies synthetic control methods from causal inference to improve the accuracy and interpretability of anomaly detection processes.
  arXiv  Detail & Related papers  (2025-01-30T15:15:17Z)
• AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledge [57.66282463340297]
  Knowledge conflict arises from discrepancies between information in the context of a large language model and the knowledge stored in its parameters.<n>We propose a fine-grained, instance-level approach called AdaCAD, which dynamically infers the weight of adjustment based on the degree of conflict.<n>We show that ADACAD consistently outperforms other decoding baselines with average QA accuracy gains of 14.21% (absolute) over a static contrastive baseline, and improves the factuality of summaries by 6.19 (AlignScore)
  arXiv  Detail & Related papers  (2024-09-11T16:35:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.