Architecture-Sensitive Supervised Fine-Tuning for Screen-Conditioned Action Prediction: A PiSAR Benchmark

• URL: http://arxiv.org/abs/2605.29400v1
• Date: Thu, 28 May 2026 05:49:36 GMT
• Title: Architecture-Sensitive Supervised Fine-Tuning for Screen-Conditioned Action Prediction: A PiSAR Benchmark
• Abstract summary: We benchmark three supervised fine-tuned models against frontier zero-shot baselines on a 661-row held-out slice of PiSAR.<n>Every model, frontier or fine-tuned, is evaluated on the same 661-row slice with the same scoring pipeline.
• Score: 4.626261940793026
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We benchmark three supervised fine-tuned models against frontier zero-shot baselines on a 661-row held-out slice of PiSAR (Persona, intent, Screen, Action, Rationale), a 12,929-tuple corpus of screen-anchored behavioural rationales curated from public app-store reviews, Pew American Trends Panel demographics, and the OPeRA shopper traces. Every model, frontier or fine-tuned, is evaluated on the same 661-row slice with the same scoring pipeline. Two findings. First, frontier zero-shot baselines (Claude Opus 4.7 and GPT-5.5) reach sem_sim 0.459 and 0.482 respectively; a fine-tuned Qwen3-VL-8B-Instruct reaches 0.783 and clears sem_sim >= 0.7 on 79% of rows, against 1-2% for either frontier baseline, a gap of 0.30 absolute on the same test set. Second, the same training data and recipe on Gemma-4-26B-A4B-IT scores only 0.441, in the same band as the frontier zero-shot baselines rather than the fine-tuned Qwen. We read this as a recipe-vs-model mismatch: the reasoning-tuned high-parameter model resists displacement and would likely need either more data or a stronger fine-tuning method.

Related papers

• GroundBench: A Factorized, Counterfactual Benchmark for Locating VLM Affordance Failures [0.0]
  A companion evaluation found that naming the target part in a manipulation prompt increased action accuracy by 0.32-0.63 across eight vision-language models.<n>However, naming the part supplies information that a real system must infer, confounding visual grounding, mechanical reasoning, and category-to-action association.<n>We introduce GroundBench, a diagnostic benchmark that separates these explanations through six branch-and-merge conditions.
  arXiv  Detail & Related papers  (2026-09-10T14:52:26Z)
• From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix [55.08143827481755]
  We consolidate traffic from over 200 internal applications onto a single model.<n>We train a separate GRPO expert per axis and merge them via two-stage SLERP.<n>The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.
  arXiv  Detail & Related papers  (2026-09-01T17:39:26Z)
• TrustRoboReward: Preference-Ordered Isotonic Score Editing for Multi-Paradigm Robot Reward Models [40.42873860258522]
  We propose TrustRoboReward, a multi-paradigm reward modeling framework equipped with Preference-Ordered Isotonic Score Editing (POISE)<n>POISE rectifies pointwise scores and eliminates cross-paradigm reversal conflicts unresolved by TrustJudge.<n>Qwen3-VL-4B trained with POISE achieves an overall reward score of 77.96%, nearly matching GPT-5-mini (78.09%, gap 0.13%) and outperforming the strongest RoboReward-4B baseline by 10.13%.
  arXiv  Detail & Related papers  (2026-08-09T05:25:22Z)
• TokenPrint: A Calibrated Token-Space Fingerprint for Language-Model Provenance [8.657685524975246]
  We introduce a training-free fingerprint based on the top-$k$ vocabulary projections of late hidden states elicited by 250 fixed knowledge probes.<n>We evaluate the method on 32 open-weight models from nine families (0.6B--32B) with documented relationships.
  arXiv  Detail & Related papers  (2026-08-08T13:56:39Z)
• On-Policy Self-Distillation without Any Supervision [47.315224514076334]
  We show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency.<n>We propose unsupervised on-policy self-distillation (U-OPSD)
  arXiv  Detail & Related papers  (2026-08-06T17:18:23Z)
• Sub-Billion, Super-Frontier: Small Language Models Rival Zero-Shot Frontier LLMs on General and Literary Relation Extraction [5.070542698701158]
  Large language models (LLMs) achieve strong relation extraction (RE)<n>We investigate how far small language models (SLMs) can close this gap across general-domain and literary text.
  arXiv  Detail & Related papers  (2026-06-21T17:24:31Z)
• Knowledge Index of Noah's Ark [63.143852586221534]
  We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
  arXiv  Detail & Related papers  (2026-06-03T17:06:49Z)
• Benchmarking Patent Embeddings: A Multi-Task Evaluation of 22 Models Across Retrieval, Classification, and Clustering [0.0]
  We evaluate 22 pre-trained embedding models on three tasks -- information retrieval, classification, and clustering.<n>We find that two results cast doubt on the prevailing wisdom.
  arXiv  Detail & Related papers  (2026-05-22T23:51:13Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Step-wise Rubric Rewards for LLM Reasoning [72.17879367869503]
  Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in large language models.<n>We find 18.2% of steps in correct responses are wrong yet positively rewarded.<n>Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points.
  arXiv  Detail & Related papers  (2026-05-17T07:08:14Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Optimised Support Vector Regression for California Housing Price Prediction: The Critical Role of Feature Engineering and Hyperparameter Tuning [0.0]
  Support Vector Regression has been cited as one of the weakest performers on the California Housing benchmark dataset.<n>This paper examines whether the previously reported performance reflects experimental configuration choices rather than an inherent algorithmic limitation.<n> tuned SVR achieves a test R2 of 0.723, a 0.123-point absolute improvement over the previously reported SVR result.
  arXiv  Detail & Related papers  (2026-05-09T03:58:50Z)
• Paired-CSLiDAR: Height-Stratified Registration for Cross-Source Aerial-Ground LiDAR Pose Refinement [46.13526676100776]
  We introduce Paired-CSLiDAR, a cross-source aerial-ground LiDAR benchmark for single-scan pose refinement.<n>The benchmark contains 12,683 ground-aerial pairs across 6 evaluation sites.<n>We propose Residual-Guided Stratified Registration (RGSR), a training-free, geometry-only refinement pipeline.
  arXiv  Detail & Related papers  (2026-05-01T13:14:20Z)
• Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment [53.72927532626824]
  We show that subsets of just 50 examples (0.3% of data) can achieve over 0.93 Pearson correlation with full benchmark scores.<n>To better predict preferences, we trained regression models on these selected subsets, achieving 0.98 correlation.<n>This demonstrates that in regression modeling, well-curated subsets outpredict the full benchmark, showing quality over quantity.
  arXiv  Detail & Related papers  (2026-04-20T00:57:31Z)
• MIRROR: A Hierarchical Benchmark for Metacognitive Calibration in Large Language Models [0.0]
  MIRROR is a benchmark that evaluates whether large language models can use self-knowledge to make better decisions.<n>We evaluate 16 models from 8 labs across approximately 250,000 evaluation instances.
  arXiv  Detail & Related papers  (2026-04-15T08:41:12Z)
• Gemma 4, Phi-4, and Qwen3: Accuracy-Efficiency Tradeoffs in Dense and MoE Reasoning Language Models [6.396911723204044]
  Mixture-of-experts (MoE) language models are often expected to offer better quality-efficiency tradeoffs than dense models.<n>We present a benchmark of seven recent reasoning-oriented instruction-tuned models spanning dense and MoE designs.
  arXiv  Detail & Related papers  (2026-04-08T12:50:52Z)
• Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels [83.99688944263843]
  Weight-De Low-Rank Adaptation (DoRA) extends LoRA by decoupling weight magnitude from direction.<n>At d_in = 8192 and rank r = 384, a single module's norm requires about 512 MB of transient working memory in bf16.<n>A factored norm decomposes the squared norm into base, cross, and Gram terms computable through O(d_out r + r2) intermediates, eliminating the dense product.
  arXiv  Detail & Related papers  (2026-03-23T17:57:24Z)
• EdgeJury: Cross-Reviewed Small-Model Ensembles for Truthful Question Answering on Serverless Edge Inference [0.0]
  We present EdgeJury, a lightweight ensemble framework that improves truthfulness and robustness.<n>On TruthfulQA (MC1), EdgeJury achieves 76.2% accuracy.<n>On a 200-question adversarial EdgeCases set, EdgeJury yields +48.2% relative gains.
  arXiv  Detail & Related papers  (2025-12-29T14:48:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.