Towards Observation Lakehouses: Living, Interactive Archives of Software Behavior

• URL: http://arxiv.org/abs/2512.02795v1
• Date: Tue, 02 Dec 2025 14:12:36 GMT
• Title: Towards Observation Lakehouses: Living, Interactive Archives of Software Behavior
• Abstract summary: In prior work, we addressed representation with Sequence Sheets, StimulusResponse Matrices, and StimulusResponse Cubes.<n>In this paper, we introduce observation lakehouses that operationalize continual SRCs.<n>We ingest data from controlled pipelines (SOLAS) and CILAS, enabling n-version assessment, behavioral clustering, and consensus oracles.
• Score: 2.6397379133308214
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Code-generating LLMs are trained largely on static artifacts (source, comments, specifications) and rarely on materializations of run-time behavior. As a result, they readily internalize buggy or mislabeled code. Since non-trivial semantic properties are undecidable in general, the only practical way to obtain ground-truth functionality is by dynamic observation of executions. In prior work, we addressed representation with Sequence Sheets, Stimulus-Response Matrices (SRMs), and Stimulus-Response Cubes (SRCs) to capture and compare behavior across tests, implementations, and contexts. These structures make observation data analyzable offline and reusable, but they do not by themselves provide persistence, evolution, or interactive analytics at scale. In this paper, therefore, we introduce observation lakehouses that operationalize continual SRCs: a tall, append-only observations table storing every actuation (stimulus, response, context) and SQL queries that materialize SRC slices on demand. Built on Apache Parquet + Iceberg + DuckDB, the lakehouse ingests data from controlled pipelines (LASSO) and CI pipelines (e.g., unit test executions), enabling n-version assessment, behavioral clustering, and consensus oracles without re-execution. On a 509-problem benchmark, we ingest $\approx$8.6M observation rows ($<$51MiB) and reconstruct SRM/SRC views and clusters in $<$100ms on a laptop, demonstrating that continual behavior mining is practical without a distributed cluster of machines. This makes behavioral ground truth first-class alongside other run-time data and provides an infrastructure path toward behavior-aware evaluation and training. The Observation Lakehouse, together with the accompanying dataset, is publicly available as an open-source project on GitHub: https://github.com/SoftwareObservatorium/observation-lakehouse

Related papers

• Automata from Agent Traces: Failure and Next-Step Prediction [1.16693435589939]
  We collapse an entire trace corpus into a single, compact finite-state machine (FSM) that serves as a structural substrate for the behavior of LLM agents.<n>For failure prediction, per-state behavioral features reach held-out AUROC up to 0.94, and an online monitor ranks failing runs above passing ones.
  arXiv  Detail & Related papers  (2026-08-24T17:58:01Z)
• TELLER: Non-intrusive Cross-Layer Root-Cause Analysis for LLM Inference [47.62941623025392]
  We present TELLER, a non-intrusive and Log-aware LLM inference Root-cause analysis framework.<n>TELLER collects NVTX/CUPTI traces and service logs without modifying model binaries.<n>It reconstructs per-request call-chain trees and aligns log lines with corresponding execution steps.<n>On top of these representations, TELLER combines numeric candidate localization with a multimodal root-cause model.
  arXiv  Detail & Related papers  (2026-08-03T09:39:11Z)
• Scaling Self-Evolving Agents via Parametric Memory [69.96398842169002]
  Existing memory-augmented LLM agents store past experience exclusively in prompt space.<n>We introduce textttTMEM, a self-evolving parametric memory framework.<n>textttTMEM consistently outperforms summary-based and retrieval-based baselines across different model scales.
  arXiv  Detail & Related papers  (2026-06-03T07:18:31Z)
• Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors [61.610957638373826]
  We introduce a method to measure the distributional gap between real and simulated user behaviors.<n>Given a dataset of real and simulated conversations, our method extracts representations of user behavior from each conversation.<n>We show that most simulators behave similarly, while a few stand apart.
  arXiv  Detail & Related papers  (2026-05-08T15:09:25Z)
• Logic-Regularized Verifier Elicits Reasoning from LLMs [63.65875399266337]
  We propose LOVER, an unsupervised verifier regularized by logical rules.<n>Lover treats theverifier as a binary latent variable, utilizinginternal activations and enforcing three logical constraints.<n>Lover significantly outperforms unsupervised baselines.
  arXiv  Detail & Related papers  (2026-05-07T09:03:49Z)
• Treating Run-time Execution History as a First-Class Citizen: Co-Versioning Run-time Behavior alongside Code [2.6397379133308214]
  This paper argues that this mismatch contributes to a blind spot in software evolution analysis and CI.<n>We propose textitBehavioral Co-Versioning, a paradigm that couples the Git history with a textitBehavioral Archive.<n>This enables semantic diffing, behavior-aware regression localization, and retrospective auditing by querying historical executions.
  arXiv  Detail & Related papers  (2026-04-18T09:36:12Z)
• AgentSkiller: Scaling Generalist Agent Intelligence through Semantically Integrated Cross-Domain Data Synthesis [30.512393568258105]
  Large Language Model agents demonstrate potential in solving real-world problems via tools, yet generalist intelligence is bottlenecked by scarce high-quality, long-horizon data.<n>We propose AgentSkiller, a fully automated framework synthesizing multi-turn interaction data across realistic, semantically linked domains.
  arXiv  Detail & Related papers  (2026-02-10T03:21:42Z)
• MirrorBench: An Extensible Framework to Evaluate User-Proxy Agents for Human-Likeness [0.4893345190925178]
  Large language models (LLMs) are increasingly used as human simulators.<n> naive "act-as-a-user" often yields verbose, unrealistic utterances.<n>We present MIRRORBENCH, a benchmarking framework that evaluates user proxies solely on their ability to produce human-like user utterances.
  arXiv  Detail & Related papers  (2026-01-13T01:16:13Z)
• EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation [9.472124187479915]
  In on-chain transaction scenarios, even a minor error can cause irreversible loss for users.<n>We introduce EVM-QuestBench, an execution-grounded benchmark for natural-language transaction-script generation.<n>We evaluate 20 models and find large performance gaps, with split scores revealing persistent asymmetry between single-action precision and multi-step workflow completion.
  arXiv  Detail & Related papers  (2026-01-10T13:25:27Z)
• RAGVUE: A Diagnostic View for Explainable and Automated Evaluation of Retrieval-Augmented Generation [1.564663326217051]
  RAGVUE is a framework for evaluation of Retrieval-Augmented Generation (RAG) systems.<n>It decomposes RAG behavior into retrieval quality, answer relevance and completeness, strict claim-level faithfulness, and judge calibration.<n> RAGVUE supports both manual metric selection and fully automated agentic evaluation.
  arXiv  Detail & Related papers  (2025-12-03T07:42:49Z)
• CausalVerse: Benchmarking Causal Representation Learning with Configurable High-Fidelity Simulations [14.751682308668391]
  Causal Representation Learning aims to uncover the data-generating process and identify the underlying causal variables and relations.<n>The dataset comprises around 200 thousand images and 3 million video frames across 24 sub-scenes in four domains.
  arXiv  Detail & Related papers  (2025-10-15T19:39:22Z)
• QueryGym: Step-by-Step Interaction with Relational Databases [30.757678338337055]
  We introduce QueryGym, an interactive environment for building, testing, and evaluating LLM-based query planning agents.<n>Existing frameworks often tie agents to specific query language dialects or obscure their reasoning.<n>QueryGym requires agents to construct explicit sequences of relational algebra operations.
  arXiv  Detail & Related papers  (2025-09-25T22:48:49Z)
• PARALLELPROMPT: Extracting Parallelism from Large Language Model Queries [19.780108308260214]
  We introduce PARALLELPROMPT, the first benchmark for measuring intra-query parallelism in natural user prompts.<n>Our dataset comprises over 37,000 real-world prompts from public LLM chat logs.<n>We provide an execution suite that benchmarks serial vs. parallel strategies, measuring latency, structural adherence, and semantic fidelity.
  arXiv  Detail & Related papers  (2025-06-23T15:05:54Z)
• COrAL: Order-Agnostic Language Modeling for Efficient Iterative Refinement [80.18490952057125]
  Iterative refinement has emerged as an effective paradigm for enhancing the capabilities of large language models (LLMs) on complex tasks. We propose Context-Wise Order-Agnostic Language Modeling (COrAL) to overcome these challenges. Our approach models multiple token dependencies within manageable context windows, enabling the model to perform iterative refinement internally.
  arXiv  Detail & Related papers  (2024-10-12T23:56:19Z)
• NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking [65.24988062003096]
  We present NAVSIM, a framework for benchmarking vision-based driving policies. Our simulation is non-reactive, i.e., the evaluated policy and environment do not influence each other. NAVSIM enabled a new competition held at CVPR 2024, where 143 teams submitted 463 entries, resulting in several new insights.
  arXiv  Detail & Related papers  (2024-06-21T17:59:02Z)
• Understanding Emergent In-Context Learning from a Kernel Regression Perspective [55.95455089638838]
  Large language models (LLMs) have initiated a paradigm shift in transfer learning.<n>This paper proposes a kernel-regression perspective of understanding LLMs' ICL bahaviors when faced with in-context examples.<n>We find that during ICL, the attention and hidden features in LLMs match the behaviors of a kernel regression.
  arXiv  Detail & Related papers  (2023-05-22T06:45:02Z)
• Going beyond research datasets: Novel intent discovery in the industry setting [60.90117614762879]
  This paper proposes methods to improve the intent discovery pipeline deployed in a large e-commerce platform. We show the benefit of pre-training language models on in-domain data: both self-supervised and with weak supervision. We also devise the best method to utilize the conversational structure (i.e., question and answer) of real-life datasets during fine-tuning for clustering tasks, which we call Conv.
  arXiv  Detail & Related papers  (2023-05-09T14:21:29Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)
• How Useful is Self-Supervised Pretraining for Visual Tasks? [133.1984299177874]
  We evaluate various self-supervised algorithms across a comprehensive array of synthetic datasets and downstream tasks. Our experiments offer insights into how the utility of self-supervision changes as the number of available labels grows.
  arXiv  Detail & Related papers  (2020-03-31T16:03:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.