TRACE: A Multi-Layer Benchmark for Human AI Controller Coordination Under Drift and Failure

• URL: http://arxiv.org/abs/2608.06657v1
• Date: Fri, 07 Aug 2026 00:03:55 GMT
• Title: TRACE: A Multi-Layer Benchmark for Human AI Controller Coordination Under Drift and Failure
• Abstract summary: Modern cyber-physical and AI-assisted systems couple human operators, AI decision modules, and automated controllers in a single control loop.<n>No standard benchmark captures time-aligned, multi-layer traces of how drift and failures propagate across these layers.<n>We construct a benchmark by injecting controlled drift into traces derived from ALFRED, a grounded-instruction benchmark for everyday household tasks.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern cyber-physical and AI-assisted systems couple human operators, AI decision modules, and automated controllers in a single control loop, so trustworthiness depends on the whole loop, not any one model. Yet no standard benchmark captures time-aligned, multi-layer traces of how drift and failures propagate across these layers, so we cannot diagnose where coordination breaks down, why, or how to recover. This paper targets one facet of that gap: drift, a deviation that can originate in any stack layer and that conventional single-modality monitoring cannot localize to a layer or pin to an onset time. We construct a benchmark by injecting controlled drift into traces derived from ALFRED, a grounded-instruction benchmark for everyday household tasks, yielding 1,918 drifted traces. Each trace is a time-aligned sequence of per-step records across five execution layers (state, observation, decision, rules, control), labeled with the drift type, affected layer, onset time, responsible actor, and causal mechanism, and validated by independent raters with inter-annotator agreement reported. We pair the dataset with a leak-aware protocol that removes a near-perfect onset leak, and a baseline study across classical, recurrent, and attention-based model families. Under this honest protocol, drift is identifiable and attributable well above random and majority baselines across every family (affected layer macro-F1 near 0.70, responsible actor near 0.85, causal mechanism near 0.49), and heavy attention offers no advantage over simpler models on this symbolic benchmark.

Related papers

• Sensitivity Analysis of GRU, LSTM and Transformer Encoder in Classification of Automated Driving Systems [0.5156484100374058]
  Monitoring systems to independently verify which automated driving system is active are important for safety monitoring, regulatory compliance, insurance assessment, and anomaly detection.<n>We first evaluate the effectiveness of three sequence-based classification models: Gated Recurrent Units (GRU), Long Short-Term Memory (LSTM) networks, and a Transformer encoder model.<n>All three models achieve strong clean-data performance with macro F1-scores of 0.92 (GRU), 0.90 (LSTM), and 0.93 (Transformer encoder model) when trained on clean data.
  arXiv  Detail & Related papers  (2026-07-25T01:31:37Z)
• Modeling Normal Is All You Need: Joint Latent Clustering for Anomaly Detection in Multimodal Cyber-Physical Systems [47.027290803102666]
  Normal behaviour is the union of many imbalanced, curved, thin-fringed operating regimes.<n>We model the normal law with a jointly learned latent representation plus explicit Gaussian-mixture mode clustering.<n>We evaluate under a deliberately fair protocol: raw point-wise metrics with no point adjustment, a trivial-detector difficulty split, prevalence-matched F1, and train-normal-only calibration.
  arXiv  Detail & Related papers  (2026-07-07T10:10:00Z)
• Do Transformers Actually Help Intrusion Detection? A Temporal Sequence Evaluation on CIC-IDS2017 [1.2934180951771597]
  We reformulate CIC-IDS 2017 as a temporal intrusion-detection task by constructing ordered flow sequences from network conversations.<n>The central finding is that padding convention, not architecture, determines the Transformer's performance.<n>We advocate leakage-free splits, explicit padding disclosure, and sequence-aware benchmarking as standard practice in future IDS research.
  arXiv  Detail & Related papers  (2026-06-09T16:57:10Z)
• CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs [100.38986535324284]
  We introduce textbfCIAware-Bench, a benchmark for measuring textbfcontrol textbfintervention (CI) awareness across frontier models.<n>CIAware-Bench tests whether models can distinguish their own trajectories from those modified by a control intervention.
  arXiv  Detail & Related papers  (2026-06-09T16:24:16Z)
• The Saturation Trap and the Subjectivity of Intervention Timing: Why Affect-Based Triggers and LLM Judges Fail to Time Interventions on Autonomous Agents [0.0]
  Intervention timing is a low-reliability construct, making single-annotator F1 an unsuitable optimization target.<n>Our contribution is the joint mapping of this problem across human inter-rater reliability, four detector architectures, a cross-model LLM-judge sweep, and a reproduced saturation effect.
  arXiv  Detail & Related papers  (2026-06-02T23:54:27Z)
• Multi-scale Coarse-to-fine Modeling for Test-time Human Motion Control [51.92884966472683]
  MSCoT is a multi-scale, coarse-to-fine model for test-time human motion synthesis and control.<n> MSCoT discretizes motion into a multi-scale hierarchical representation and predicts the entire token sequence at each temporal scale.
  arXiv  Detail & Related papers  (2026-05-14T15:09:49Z)
• Metacognitive Self-Correction for Multi-Agent System via Prototype-Guided Next-Execution Reconstruction [58.51530390018909]
  Large Language Model based multi-agent systems excel at collaborative problem solving but remain brittle to cascading errors.<n>We present MASC, a metacognitive framework that endows MAS with real-time, unsupervised, step-level error detection and self-correction.
  arXiv  Detail & Related papers  (2025-10-16T05:35:37Z)
• Hierarchical Self-Supervised Representation Learning for Depression Detection from Speech [51.14752758616364]
  Speech-based depression detection (SDD) is a promising, non-invasive alternative to traditional clinical assessments.<n>We propose HAREN-CTC, a novel architecture that integrates multi-layer SSL features using cross-attention within a multitask learning framework.<n>The model achieves state-of-the-art macro F1-scores of 0.81 on DAIC-WOZ and 0.82 on MODMA, outperforming prior methods across both evaluation scenarios.
  arXiv  Detail & Related papers  (2025-10-05T09:32:12Z)
• Unsupervised Detection of Spatiotemporal Anomalies in PMU Data Using Transformer-Based BiGAN [0.0]
  We introduce T-BiGAN, a framework that window-attention Transformers within a bidirectional Generative Adversarial Network (BiGAN)<n>Its encoderdecoder captures architecture while discriminator enforces cycle consistency to align latent space with the true data distribution.<n>Anomalies are flagged in real-time using an adaptive score that combines reconstruction error, latent space drift, and discriminator confidence.
  arXiv  Detail & Related papers  (2025-09-30T00:16:35Z)
• Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion [56.38386580040991]
  Consistency Trajectory Model (CTM) is a generalization of Consistency Models (CM) CTM enables the efficient combination of adversarial training and denoising score matching loss to enhance performance. Unlike CM, CTM's access to the score function can streamline the adoption of established controllable/conditional generation methods.
  arXiv  Detail & Related papers  (2023-10-01T05:07:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.