The Single-File Test: A Longitudinal Public-Interface Evaluation of First-Output LLM Web Generation with Social Reach Tracking

• URL: http://arxiv.org/abs/2605.06707v1
• Date: Wed, 06 May 2026 18:19:11 GMT
• Title: The Single-File Test: A Longitudinal Public-Interface Evaluation of First-Output LLM Web Generation with Social Reach Tracking
• Abstract summary: This paper presents an eight-week observational comparison of 68 single-file HTML generations collected across 17 public experiments in the "HTML AI Battle" project.<n>Four reasoning model families, GPT, Gemini, Grok, and Claude, were compared under a fixed public-interface protocol with no custom instructions, no personality tuning, and no repair prompts.<n> Claude was the strongest and most consistent family, leading mean performance and winning 9/17 prompts under the primary human weighted score.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents an eight-week observational comparison of 68 single-file HTML generations collected across 17 public experiments in the "HTML AI Battle" project between December 10, 2025 and February 4, 2026. Four reasoning model families, GPT, Gemini, Grok, and Claude, were compared under a fixed public-interface protocol with no custom instructions, no personality tuning, and no repair prompts. Each output was evaluated from a rendered browser video using human scores and a Gemini LLM-as-a-judge layer for prompt adherence, functional correctness, and UI quality, then packaged into a standardized social-media protocol spanning X (Twitter), TikTok, and YouTube. The tracker was also used for two supervised predictive analyses: an experiment-level model for 24-hour X impressions and a generation-level model for HTML verbosity. Under this protocol, Claude was the strongest and most consistent family, leading mean performance and winning 9/17 prompts under the primary human weighted score. Longer measured reasoning time was not associated with higher quality overall. Gemini as a judge was significantly more lenient than the human evaluator on functional correctness and overall performance, while stable self-favoring bias remained unresolved. The exploratory X-impressions model remained weak under post-screen cross-validation (MAE = 46,874, R^2 = -0.377), whereas the HTML-lines model performed better, with a model-family-only baseline outperforming prompt-aware alternatives (MAE = 135.2, R^2 = 0.576). Overall, selected pre-publication technical/audio variables were not sufficient to predict 24-hour X reach, while code verbosity was driven much more by model family than by prompt wording. The comparisons remain observational and are limited by public-interface drift, access-path differences, and one primary human scorer.

Related papers

• DashArena: Benchmarking LLMs on Interactive Analytic Dashboard Generation [8.452268647068166]
  We introduce DashArena, to our knowledge the first benchmark for open-ended, task-grounded generation of interactive analytic dashboards.<n>DashArena requires each system to generate both a dashboard and a replayable interaction trajectory.<n>A browser replays the trajectory and turns the system's intended analytical workflow into reproducible visual and execution evidence.
  arXiv  Detail & Related papers  (2026-08-11T06:54:37Z)
• Language Models Agree With Each Other, Not With Readers [1.8620637029128544]
  We measure convergence against a human reference nobody built for the purpose.<n>2,523 reader mark sets across 120 web documents.<n>No model agrees with readers detectably more than a reader does.
  arXiv  Detail & Related papers  (2026-07-31T10:44:10Z)
• Style over Substance: A Shortcut Audit of Emotion-Description Preference Evaluation [18.639828982880108]
  We conduct a systematic shortcut audit of EmoPrefer using content-blind probes.<n>A simple logistic regression using only description length and generator identity performs comparably to LoRA-finetuned 7B text and audio-visual judges.<n>Results do not imply that human preferences are inherently stylistic or that the descriptions contain no emotional information.
  arXiv  Detail & Related papers  (2026-07-20T21:05:08Z)
• Modality-Driven Search with Holistic Trace Judging for ARC-AGI-2 [0.0]
  I present a solver for ARC-AGI-2, a few-shot visual reasoning benchmark.<n>It generates diverse candidates independently across text, image, and code channels.<n>Unlike self-consistency or majority voting, it reliably recovers correct minority hypotheses on tasks where the modal answer is wrong.
  arXiv  Detail & Related papers  (2026-06-30T11:55:33Z)
• vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models [58.633451339058986]
  VLA models are typically evaluated using per benchmark scripts maintained independently by each model repository.<n>We present vla eval, an open source evaluation harness that decouples model inference from benchmark execution.<n>A complete evaluation requires only two commands: vla eval serve and vla eval run.
  arXiv  Detail & Related papers  (2026-03-14T14:38:53Z)
• VB: Visibility Benchmark for Visibility and Perspective Reasoning in Images [0.0]
  We present VB, a benchmark that tests whether vision-language models can determine what is and is not visible in a photograph.<n>Items are organized into 100 families using a 2x2 design that crosses a minimal image edit with a minimal text edit.<n>We score models on confidence-aware accuracy with abstention (CAA), minimal-edit flip rate (MEFR), confidence-ranked selective prediction (SelRank), and second-order perspective reasoning.
  arXiv  Detail & Related papers  (2026-03-03T23:03:11Z)
• Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Test [62.17144846428715]
  We introduce the Embodied Turing Test benchmark: WoW-World-Eval (Wow,wo,val)<n>Wow-wo-val examines five core abilities, including perception, planning, prediction, generalization and execution.<n>For the Inverse Dynamic Model Turing Test, we first use an IDM to evaluate the video foundation models' execution accuracy in the real world.
  arXiv  Detail & Related papers  (2026-01-07T17:50:37Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• Ensuring Reproducibility in Generative AI Systems for General Use Cases: A Framework for Regression Testing and Open Datasets [0.0]
  We introduce GPR-bench, a benchmark that operationalizes regression testing for general purpose use cases.<n>We show that newer models generally improve correctness, but the differences are modest and not statistically significant.<n>In contrast, the concise-writing instruction significantly enhances conciseness, demonstrating the effectiveness of prompt engineering.
  arXiv  Detail & Related papers  (2025-05-02T12:31:43Z)
• Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens [53.99177152562075]
  Scaling up autoregressive models in vision has not proven as beneficial as in large language models. We focus on two critical factors: whether models use discrete or continuous tokens, and whether tokens are generated in a random or fixed order using BERT- or GPT-like transformer architectures. Our results show that while all models scale effectively in terms of validation loss, their evaluation performance -- measured by FID, GenEval score, and visual quality -- follows different trends.
  arXiv  Detail & Related papers  (2024-10-17T17:59:59Z)
• Large Language Models are not Fair Evaluators [60.27164804083752]
  We find that the quality ranking of candidate responses can be easily hacked by altering their order of appearance in the context. This manipulation allows us to skew the evaluation result, making one model appear considerably superior to the other. We propose a framework with three simple yet effective strategies to mitigate this issue.
  arXiv  Detail & Related papers  (2023-05-29T07:41:03Z)
• Perception Test: A Diagnostic Benchmark for Multimodal Video Models [78.64546291816117]
  We propose a novel multimodal video benchmark to evaluate the perception and reasoning skills of pre-trained multimodal models. The Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and types of reasoning (descriptive, explanatory, predictive, counterfactual) across video, audio, and text modalities. The benchmark probes pre-trained models for their transfer capabilities, in a zero-shot / few-shot or limited finetuning regime.
  arXiv  Detail & Related papers  (2023-05-23T07:54:37Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.