Nautilus Compass: Black-box Persona Drift Detection for Production LLM Agents

• URL: http://arxiv.org/abs/2605.09863v1
• Date: Mon, 11 May 2026 01:49:17 GMT
• Title: Nautilus Compass: Black-box Persona Drift Detection for Production LLM Agents
• Abstract summary: Nautilus Compass is a black-box drift detector and agent memory layer for production coding agents.<n>The system ships as a Claude Code plugin, an MCP 20245 A2A server, a CLI, and an API on one daemon.
• Score: 2.417342411475111
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Production LLM coding agents drift over long sessions: they forget user-specified constraints, slip into mistakes the user already flagged, and confabulate prior agreements. White-box approaches such as persona vectors require model weights and so cannot be applied to closed APIs (Claude, GPT-4) that most users actually interact with. We present Nautilus Compass, a black-box persona drift detector and agent memory layer for production coding agents. The method operates entirely at the prompt-text layer: cosine similarity between user prompts and behavioral anchor texts, aggregated by a weighted top-k mean using BGE-m3 embeddings. Compass is, to our knowledge, the only public agent memory layer (among Mem0, Letta, Cognee, Zep, MemOS, smrti verified May 2026) that does not call an LLM at index time to extract facts or build a graph; raw conversation text is embedded directly. The system ships as a Claude Code plugin, an MCP 2024-11-05 A2A server (Cursor, Cline, Hermes), a CLI, and a REST API on one daemon, with a Merkle-chained audit log for tamper-evident anchor updates. On a held-out test set built from real Claude Code session traces and labeled by an independent LLM judge, Compass reaches ROC AUC 0.83 for drift detection. The embedded retrieval pipeline scores 56.6% on LongMemEval-S v0.8 and 44.4% on EverMemBench-Dynamic (n=500), topping the four published EverMemBench Table 4 baselines. LongMemEval-S 56.6% is ~30 points below recent white-box leaders (90+%); we treat that as the architectural ceiling of the no-extraction design. End-to-end reproduction cost is $3.50 (~14x cheaper than GPT-4o-judged stacks). A paired cross-vendor behavior A/B accompanies these numbers as preliminary system-level evidence. Code, anchors, frozen test data, and audit-log tooling are MIT-licensed at github.com/chunxiaoxx/nautilus-compass.

Related papers

• Codebook Agent: Amortized Topology Design for LLM Multi-Agent Systems [67.04448659688579]
  We develop a query-independent 16-entry codebook that ranks the top decoded candidates in a single batched forward pass.<n>With no iterative search and no message passing at test time, Codebook Agent is the most accurate method on all six benchmarks.
  arXiv  Detail & Related papers  (2026-09-02T08:10:22Z)
• JavaVulBench: A Java Vulnerability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-Aware Evaluation Mode [0.0]
  textscVulBench is a benchmark and evaluation harness for Java vulnerability evaluation.<n>The dataset contains $sim,600 Java methods spanning 1-14, 700+ projects, labelled at both line and publication dates.<n>The harness provides a single ttexttLlm schema across backend families.
  arXiv  Detail & Related papers  (2026-07-02T23:32:43Z)
• NeuroLog: Reasoning You Can Audit -- Neuro-Symbolic Vulnerability Discovery via LLM Facts, Datalog, and SMT [0.0]
  Vulnerability discovery on C/C++ source asks analyst to choose between static analysers, which need a working build before a single query runs, and free-form LLMs, which read source but details and lose track of cross-function dataflow on reals.<n>We present NeuroLog, an-to-end build-to-end pipeline thatdiscovers typed dataflow good at each iteration.
  arXiv  Detail & Related papers  (2026-05-30T10:50:06Z)
• Do Proactive Agents Really Need an LLM to Decide When to Wake and What to Anchor? [36.54616681758304]
  Proactive agents read user activity as text and call an LLM on every event to decide whether to act.<n>We treat the always-on signal as graph updates rather than text.<n>One forward pass yields a per-event trigger probability and a per-entity routing score.
  arXiv  Detail & Related papers  (2026-05-28T16:10:32Z)
• Skill-as-Pseudocode: Refactoring Skill Libraries to Pseudocode for LLM Agents [50.78646963535282]
  We propose Skill-as-Pseudocode (SaP), an automatic conversion of markdown skill libraries into typed pseudocode.<n>For each cluster of similar procedural passages drawn from one or more skills, SaP extracts a typed contract and filters it through a four-check deterministic verifier.<n>Promoted contracts are inlined into a rewritten skill skeleton together with restored concrete action templates, giving the agent two complementary signals.
  arXiv  Detail & Related papers  (2026-05-27T04:48:40Z)
• CommitDistill: A Lightweight Knowledge-Centric Memory Layer for Software Repositories [0.5872014229110213]
  Software accumulate large amounts of unstructured knowledge in commit messages, pull-request discussions, and issue threads.<n>Recent work on typed-memory architectures for agents (MemGPT, generative agents, and the PlugMem module of Yang et al.) argues that agent memory should be distilled, typed knowledge rather than raw interaction text.<n>We adapt that stance to a software repository's own git history under a constrained regime: deterministic, dependency-free, local-only, no embeddings.
  arXiv  Detail & Related papers  (2026-05-18T12:14:28Z)
• WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation [88.10947115397971]
  This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories.<n>Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls.<n> Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification.
  arXiv  Detail & Related papers  (2026-05-11T17:49:43Z)
• LLM-Redactor: An Empirical Evaluation of Eight Techniques for Privacy-Preserving LLM Requests [0.0]
  Coding agents and LLM-powered applications routinely send potentially sensitive content to cloud LLM APIs where it may be logged, retained, used for training, or subpoenaed.<n>We present a systematic empirical evaluation of eight techniques for privacy-preserving LLM requests.<n>We implement all eight (or a tractable research-stage subset where deployment is not yet feasible) in an open-source shim compatible with MCP and any OpenAI-compatible API.
  arXiv  Detail & Related papers  (2026-04-13T21:05:42Z)
• VibeTensor: System Software for Deep Learning, Fully Generated by AI Agents [42.56489784841984]
  "fully generated" refers to code provenance: implementation changes were produced and applied as agent-proposed diffs.<n>We describe the architecture, summarize the workflow used to produce and validate the system, and evaluate the artifact.
  arXiv  Detail & Related papers  (2026-01-21T19:29:00Z)
• eSapiens's DEREK Module: Deep Extraction & Reasoning Engine for Knowledge with LLMs [10.667949307405983]
  DEREK module ingests heterogeneous content (PDF, Office, web), splits it into 1,000-token overlapping chunks, and indexes them in a hybrid HNSW+BM25 store.<n>LangGraph verifier enforces citation overlap, regenerating answers until every claim is grounded.<n>All components run in containers, enforce end-to-end TLS 1.3 and AES-256.
  arXiv  Detail & Related papers  (2025-07-13T05:54:01Z)
• Towards Realistic Evaluation of Commit Message Generation by Matching Online and Offline Settings [77.20838441870151]
  We use an online metric - the number of edits users introduce before committing the generated messages to the VCS - to select metrics for offline experiments.<n>We collect a dataset with 57 pairs consisting of commit messages generated by GPT-4 and their counterparts edited by human experts.<n>Our results indicate that edit distance exhibits the highest correlation with the online metric, whereas commonly used similarity metrics such as BLEU and METEOR demonstrate low correlation.
  arXiv  Detail & Related papers  (2024-10-15T20:32:07Z)
• Get my drift? Catching LLM Task Drift with Activation Deltas [55.75645403965326]
  Task drift allows attackers to exfiltrate data or influence the LLM's output for other users.<n>We show that a simple linear classifier can detect drift with near-perfect ROC AUC on an out-of-distribution test set.<n>We observe that this approach generalizes surprisingly well to unseen task domains, such as prompt injections, jailbreaks, and malicious instructions.
  arXiv  Detail & Related papers  (2024-06-02T16:53:21Z)
• ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code [76.84199699772903]
  ML-Bench is a benchmark rooted in real-world programming applications that leverage existing code repositories to perform tasks. To evaluate both Large Language Models (LLMs) and AI agents, two setups are employed: ML-LLM-Bench for assessing LLMs' text-to-code conversion within a predefined deployment environment, and ML-Agent-Bench for testing autonomous agents in an end-to-end task execution within a Linux sandbox environment.
  arXiv  Detail & Related papers  (2023-11-16T12:03:21Z)
• Tracking by Instance Detection: A Meta-Learning Approach [99.66119903655711]
  We propose a principled three-step approach to build a high-performance tracker. We build two trackers, named Retina-MAML and FCOS-MAML, based on two modern detectors RetinaNet and FCOS. Both trackers run in real-time at 40 FPS.
  arXiv  Detail & Related papers  (2020-04-02T05:55:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.