Was It Never Collected, or Rewritten Away? A Commit-Provenance Dataset Separating Ingestion Gaps from Upstream History Edits across the World of Code

• URL: http://arxiv.org/abs/2607.02774v1
• Date: Thu, 02 Jul 2026 21:12:43 GMT
• Title: Was It Never Collected, or Rewritten Away? A Commit-Provenance Dataset Separating Ingestion Gaps from Upstream History Edits across the World of Code
• Abstract summary: We release a commit-provenance dataset that separates missing commits at scale.<n>Over 1,118,116,350 advertised commits, 53.35% are present in World of Code (WoC)<n>About one missing-commit case in fifteen is a rewrite the project erased, not a mirror gap.
• Score: 2.788646205877688
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Any global mirror of open-source version control is incomplete, but the reasons a commit is missing are not interchangeable: a project may have force-pushed it away (so it no longer exists upstream), or the mirror may never have ingested it (a true collection gap). We release a commit-provenance dataset that separates these cases at scale by comparing two views of the same commit graph: the GHArchive event stream, a historical witness of what GitHub advertised at push time, and the World of Code (WoC) V2604 object database, an accumulated union of periodic fetches that never deletes a collected commit. Walking each reference's PushEvent chain reconstructs force-pushes structurally (a before that is not the prior head breaks the fast-forward chain; no recorded flag exists across eras), and joining every advertised commit against WoC membership yields a three-way label. Over 1,118,116,350 advertised commits, 53.35% are present in WoC, 6.47% are rewritten (orphaned by a later force-push, an upstream edit and a correct absence), and 40.18% are never-ingested (the candidate collection gap). About one missing-commit case in fifteen is a rewrite the project erased, not a mirror gap. We release the force-push witness (166,710,831 events over 19,926,250 repositories) and a per-project rollup (78,125,788 repositories; force-pushing observed in 25.47%, a rewritten commit in 12.85%). We treat the 40% never-ingested share as an upper bound on the collection gap and state five reasons it overcounts. The dataset makes mirror-completeness reporting honest, flags rewritten commits as duplicate patch sources for contribution counting, and corrects a 10.82% corpus-wide undercount that history editing imposes on commit-based productivity, with a released per-project correction factor. All artifacts ship as a self-contained, independently hosted replication package keyed to WoC V2604.

Related papers

• VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement [57.86962208273888]
  We present an auditable physical-verification system in which a text-only planner compiles the prompt into typed physical obligations.<n>Each action returns a provenance-carrying evidence record whose payload, when usable, is either a typed measurement or an explicitly tagged learned state.<n>We anchor evaluation in a 1,500-clip corpus of human-annotated flaw records that localize real generation failures in prompt reference, space, and time.
  arXiv  Detail & Related papers  (2026-09-02T20:36:45Z)
• SkillGLoW: Procedural-Family Skill Consolidation for Self-Improving Agents on Long-Horizon Task Streams [55.90739575113059]
  We argue the missing unit of reuse is the solving procedure shared by a cluster of related tasks, and build SkillGLoW around it.<n>Across four benchmarks (mathematical reasoning, terminal automation, software repair, and embodied control) and three models, the priors gain 17.2 points (hard) over the no-skill baseline on average.
  arXiv  Detail & Related papers  (2026-09-02T07:31:18Z)
• RosettaBitcoin: An Artifact-Backed Experience Report on Verification Infrastructure for Agent-Assisted Consensus Validators [0.0]
  This experience report studies RosettaBitcoin, a single-developer project that built twelve separately implemented Bitcoin testnet4 consensus validators.<n>We analyze the snapshot's tracked evidence database, curated artifact index, canonical conformance fixtures, validation scripts, blocker records, and version history.<n>The case suggests that explicit failure records, fixtures, port-owned, and validating imports can make agent-assisted systems more auditable.
  arXiv  Detail & Related papers  (2026-09-01T17:22:07Z)
• From Traceability to Justifiability: Accountability Structures in Agentic Software Engineering [0.0]
  We measure, from public material only, whether AI records can express a claim and whether they hold where declared.<n>Across 188 double-graded cells we found no platform whose default record emits a content identity of a behavioral system.<n>An instrument computes realized assurance depth from a pipeline's published exhaust alone and compares it with the declared depth.
  arXiv  Detail & Related papers  (2026-08-21T16:45:00Z)
• Turning Interaction History into Execution State: A Runtime Layer for Long-Horizon Coding Agents [48.967927222079965]
  Ledger is a runtime layer that distills an agent's completed interactions into an explicit execution state.<n> Ledger keeps this state in an online execution ledger and applies it at two boundaries of every step.<n>Across all 500 SWE-bench Verified instances, Ledger raises Pass@1 from 56.2% to 64.2% with GPT-5 mini and from 75.8% to 81.0% with MiniMax M2.5.
  arXiv  Detail & Related papers  (2026-08-01T18:25:00Z)
• Why Git Is the Memory Solution for the Agentic Development Lifecycle [0.0]
  We argue memory should be git-bound, built into the repository's version control.<n>On this ledger we solve two problems separately, then combine them.<n>A router dispatches breadth to a git-anchored structural map, pointed lookups to confidence-gated episodes, and rationale to decision synthesis.
  arXiv  Detail & Related papers  (2026-07-15T22:06:59Z)
• Scaling Author Identity Disambiguation to the World of Code: A Methodology [2.788646205877688]
  We describe the methodology used to alias the World of Code (WoC) collection (version V2604, 107M distinct author strings over 6B commits) into canonical persons.<n>At this scale the central problem is over-merge, not missed merges.<n>We report the full experimental record (more than twenty experiments, including unsuccessful ones) behind the deployed design.
  arXiv  Detail & Related papers  (2026-07-08T02:30:26Z)
• Deforking the World of Code: A Project-Provenance Map that Recovers Cross-Forge Fork Families that Platform Graphs Cannot See [2.788646205877688]
  We release a curated deforking map for the World of Code (WoC) version V2604: p2PFull.<n>It collapses every raw repository p into the deforked project P to which it belongs.<n>We validate the map against GitHub's declared fork graph reconstructed from GH ForkEvents.
  arXiv  Detail & Related papers  (2026-06-28T18:25:17Z)
• SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers [55.39407031861402]
  We introduce SCDBench, a dataset and benchmark methodology for smart contract decompilation.<n>The dataset contains 600 real-world Solidity contracts with paired bytecode inputs, ground-truth source code, and replayable semantic checkpoints.<n>We evaluate Claude Opus 4.7, GPT-5.3-Codex, and GLM-5 in a zero-shot decompilation setting, including GLM-5 variants with and without extended reasoning.
  arXiv  Detail & Related papers  (2026-05-27T20:08:47Z)
• CommitDistill: A Lightweight Knowledge-Centric Memory Layer for Software Repositories [0.5872014229110213]
  Software accumulate large amounts of unstructured knowledge in commit messages, pull-request discussions, and issue threads.<n>Recent work on typed-memory architectures for agents (MemGPT, generative agents, and the PlugMem module of Yang et al.) argues that agent memory should be distilled, typed knowledge rather than raw interaction text.<n>We adapt that stance to a software repository's own git history under a constrained regime: deterministic, dependency-free, local-only, no embeddings.
  arXiv  Detail & Related papers  (2026-05-18T12:14:28Z)
• An Extensive Replication Study of the ABLoTS Approach for Bug Localization [47.89260192608388]
  ABLoTS is a proposed IRBL approach to identify buggy source code snippets with promising results.<n>We conduct a replication study of this approach with the original dataset and on two extended datasets.<n>We find that the TraceScore component, which is the core of ABLoTS, produces comparable or even better results with the extended datasets.
  arXiv  Detail & Related papers  (2026-05-12T08:54:13Z)
• How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings [56.70440596502351]
  The past year has seen over 20 open document parsing models, yet benchmarks almost exclusively on OmniDocBench.<n>We present PureDocBench, a benchmark thatrenders document images from HTML/CSS, covering 10 domains, 66 pages, and 1,475 pages.
  arXiv  Detail & Related papers  (2026-05-08T09:30:31Z)
• WiCER: Wiki-memory Compile, Evaluate, Refine Iterative Knowledge Compilation for LLM Wiki Systems [0.0]
  We characterize the compilation gap across 17 RepLiQA domains (6,800 questions)<n>We propose WiCER, an iterative algorithm inspired by counterexample-guided abstraction refinement (CEGAR) that closes this gap.<n>An ablation across all 17 topics confirms that targeted diagnosis (+0.95), not generic pinning (+0.16), drives the gains.
  arXiv  Detail & Related papers  (2026-05-08T00:25:16Z)
• RepoDoc: A Knowledge Graph-Based Framework to Automatic Documentation Generation and Incremental Updates [41.56228301287882]
  We propose RepoDoc, a system that uses a repository knowledge graph (RepoKG) as the semantic foundation for the entire documentation lifecycle.<n>For incremental updates, it cuts update time by 73% and token usage by 77%, and achieves 10.2% higher update recall.<n> Evaluated on 24 repositories across 8 languages, RepoDoc substantially outperforms state-of-the-art alternatives.
  arXiv  Detail & Related papers  (2026-04-29T10:43:02Z)
• Analysis of Commit Signing on Github [2.1045887299238855]
  We studied 71,694 active GitHub users, defined as accounts that have authored at least one commit, across all their repositories and their entire commit history, spanning 16 million commits and 874,198 repositories.<n>Fewer than 6% of developers have ever signed a commit themselves, and the vast majority of apparent signers have never signed outside a web browser.<n>Among the minority who do sign locally, signing rarely persists over time or across repositories, and roughly one in eight developer-managed signatures fails verification because signing keys are never uploaded to GitHub.
  arXiv  Detail & Related papers  (2026-04-15T15:57:07Z)
• His2Trans: A Skeleton First Framework for Self Evolving C to Rust Translation with Historical Retrieval [45.246293154277886]
  His2Trans is a framework that combines a deterministic, build-aware skeleton with self-evolving knowledge extraction to support stable, incremental migration.<n> Experiments on industrial OpenHarmony modules show that His2Trans reaches a 99.75% incremental compilation pass rate.
  arXiv  Detail & Related papers  (2026-03-03T05:42:08Z)
• From Commit Message Generation to History-Aware Commit Message Completion [49.175498083165884]
  We argue that if we could shift the focus from commit message generation to commit message completion, we could significantly improve the quality and the personal nature of the resulting commit messages. Since the existing datasets lack historical data, we collect and share a novel dataset called CommitChronicle, containing 10.7M commits across 20 programming languages. Our results show that in some contexts, commit message completion shows better results than generation, and that while in general GPT-3.5-turbo performs worse, it shows potential for long and detailed messages.
  arXiv  Detail & Related papers  (2023-08-15T09:10:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.