MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes

• URL: http://arxiv.org/abs/2609.10016v1
• Date: Wed, 09 Sep 2026 10:46:56 GMT
• Title: MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes
• Abstract summary: We introduce MetroLLM-Bench, a 955-case benchmark for testing language models as the policy layer of a transit kiosk.<n>It covers six real metro systems, ranging from 37 to 414 stations, and eleven categories that include routing, fare calculation, disruptions, accessibility, and adversarial input.<n>14 deterministic scoring components form Tier 1; eight semantic-quality components form Tier 2, six of which use a language-model judge.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce MetroLLM-Bench, a 955-case benchmark for testing language models as the policy layer of a transit kiosk. It covers six real metro systems, ranging from 37 to 414 stations, and eleven categories that include routing, fare calculation, disruptions, accessibility, and adversarial input. In each case, the model must call structured tools and submit a machine-renderable terminal state containing an outcome, a per-ticket fare quote when applicable, and a kiosk action. Fourteen deterministic scoring components form Tier 1; eight semantic-quality components form Tier 2, six of which use a language-model judge. We report Tier 1 and the combined score of both tiers. A stratified 75/25 split reserves 717 cases for training-data generation and 238 for held-out evaluation. We evaluate twenty-six models from six vendors, of which twenty-three are ranked. On the held-out partition, a 4B Qwen 3.5 student trained through parameter-efficient fine-tuning (PEFT) exceeds both GPT-5.6 tiers on Tier 1 (91.3 against 90.6 and 90.0) and matches GPT-5.4 full at maximum reasoning effort (91.4), with a 2.6 GB Q4_K_M footprint. Larger 9B and 27B students provide no further Tier 1 improvement over the 4B student at this training scale. Across the four Qwen sizes, the PEFT gain over the corresponding base model decreases from +7.03 points at 2B (three training seeds) to -0.91 at 27B; every seed shows the same direction at every size. A deterministic rule-based baseline reaches 84.6 on Tier 1, with the remaining language-model advantage concentrated in policy adaptation, compound scenarios, accessibility, and temporal reasoning. Muse Glimmer 30B leads the composite ranking, and serving configuration alone moves the Qwen 3.5-to-3.8 comparison by 2.7 Tier 1 points. The benchmark, harness, reproduction guide, and fine-tuned students are released at https://github.com/continker/metrollm-bench.

Related papers

• NeoHorse-1: Towards Recursive Self-Improvement via Agentic Post-Training with Routing Harness [76.6324710887112]
  NeoHorse-1 is a family of agent-native models developed to explore this path through agentic post-training.<n>Our system combines a heterogeneous model pool with intelligent routing, recording the predicted capability demand, selected service tier, and subsequent interaction for each user turn.
  arXiv  Detail & Related papers  (2026-09-08T03:14:54Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM-to-MLM Curriculum [0.0]
  MoganBert-TR is trained over 237.3B tokens with a two-stage CLM-to-MLM curriculum.<n>MoganBert-TR attains 78.41 on TrGLUE, the best among the Turkish ModernBERT models compared.
  arXiv  Detail & Related papers  (2026-08-26T13:12:51Z)
• FlavourBench: Executable Culinary Reward Maps for Language Model Evaluation and Post-Training [0.0]
  Open-ended language-model evaluation often substitutes another model or a small preference panel for a missing answer key.<n>We introduce FlavourBench, which compiles dense answer maps from a versioned culinary environment.<n>We evaluate 27 frontier endpoints on the same 534 substitution, pairing, and constraint tasks, yielding 14,418 complete model-task observations.
  arXiv  Detail & Related papers  (2026-08-20T21:14:52Z)
• Knowledge Index of Noah's Ark [63.143852586221534]
  We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
  arXiv  Detail & Related papers  (2026-06-03T17:06:49Z)
• Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning [51.88950852117154]
  Chunk-Level Guided Generation is a training-free alternative that uses an off-the-shelf large language model as a process scorer.<n>We show that scoring variable-length reasoning steps with large-model likelihoods is unreliable due to a systematic length bias.<n>Chunk-Level Guided Generation produces substantially shorter reasoning traces than PRM guided search.
  arXiv  Detail & Related papers  (2026-06-01T04:43:36Z)
• Architecture-Sensitive Supervised Fine-Tuning for Screen-Conditioned Action Prediction: A PiSAR Benchmark [4.626261940793026]
  We benchmark three supervised fine-tuned models against frontier zero-shot baselines on a 661-row held-out slice of PiSAR.<n>Every model, frontier or fine-tuned, is evaluated on the same 661-row slice with the same scoring pipeline.
  arXiv  Detail & Related papers  (2026-05-28T05:49:36Z)
• TabH2O: A Unified Foundation Model for Tabular Prediction [12.882141870947706]
  We present TabH2O, a model that performs classification and regression in a single forward pass via in-context learning.<n>We evaluate TabH2O v1 (29.2M parameters) on the TALENT benchmark, where it achieves an average rank of 2.55 out of 6 evaluated methods.
  arXiv  Detail & Related papers  (2026-05-18T13:27:54Z)
• The autoPET3 Challenge: Automated Lesion Segmentation in Whole-Body PET/CT $\unicode{x2013}$ Multitracer Multicenter Generalization [67.92281859568168]
  We report the design and results of the third autoPET challenge (MICCAI 2024)<n>It benchmarked automated lesion segmentation in whole-body PET/CT under a compositional generalization setting.<n>The top-ranked algorithm achieved a mean DSC of 0.66, FNV of 3.18 mL, and FPV of 2.78 mL across all four test conditions.
  arXiv  Detail & Related papers  (2026-05-07T07:07:09Z)
• CheeseBench: Evaluating Large Language Models on Rodent Behavioral Neuroscience Paradigms [0.0]
  CheeseBench is a benchmark that evaluates large language models (LLMs) on nine classical behavioral neuroscience paradigms.<n>Each task is grounded in peer-reviewed rodent protocols with approximate animal baselines.<n>We evaluate six open-weight LLMs on text-based ASCII renderings and compare against both a random baseline and a graph-based reinforcement learning agent.
  arXiv  Detail & Related papers  (2026-04-12T21:37:26Z)
• Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems [63.713297451300086]
  We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B. Their subsequent distillation into smaller models ranging from 17M-170M parameters, and their application to the Natural Language Understanding (NLU) component of a virtual assistant system.
  arXiv  Detail & Related papers  (2022-06-15T20:44:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.