ATLAS: All-round Testing of Long-context Abilities across Scales

• URL: http://arxiv.org/abs/2605.28079v1
• Date: Wed, 27 May 2026 07:33:47 GMT
• Title: ATLAS: All-round Testing of Long-context Abilities across Scales
• Abstract summary: ATLAS is a benchmarking framework that redefines long-context evaluation as length-dependent capability profiling.<n> ATLAS contributes three methodological principles: (i) a layered taxonomy separating foundational operations from application workloads so failures can be attributed, (ii) length-aware AUC scoring that integrates score-length curves over a fixed 8K-1M grid, replacing single-point metrics with full degradation profiles, and (iii) ATLAScore, a harmonic-mean aggregate over taxonomy categories that penalizes imbalanced profiles.
• Score: 42.83200405769639
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-context language models now advertise context windows up to millions of tokens, yet evaluations typically report a single length or a narrow task family, masking two failure modes: performance can collapse as length grows, and strong retrieval need not transfer to downstream use. We present ATLAS, a benchmarking framework that redefines long-context evaluation as length-dependent capability profiling. ATLAS contributes three methodological principles:(i) a layered taxonomy separating foundational operations from application workloads so failures can be attributed, (ii) length-aware AUC scoring that integrates score-length curves over a fixed 8K-1M grid, replacing single-point metrics with full degradation profiles, and (iii) ATLAScore, a harmonic-mean aggregate over taxonomy categories that penalizes imbalanced profiles, with end-to-end uncertainty propagation from subset scores through the nonlinear final aggregate. We instantiate the framework across eight capability dimensions with nine auditable components and 6,438 instances, and evaluate 26 models. Gemini-3.1-Pro-Preview leads at 128K, Claude-Opus-4.6 leads at 1M. Rankings reshuffle substantially between ATLASscore@8K-128K and ATLASscore@8K-1M: 7 models move by at least two ranks, and the two taxonomy layers share only 61% of cross-model variance, with individual rank gaps up to 12 positions. These results support reporting long-context quality by capability and length, not by a single headline score.

Related papers

• ATLAS: Agentic Taxonomy of Large-Scale Software Ecosystems [14.128311996090352]
  GitHub lacks a systematic hierarchical taxonomy of software repositories.<n>We present ATLAS, the first framework that automatically constructs a hierarchical taxonomy for software repositories and classifies projects into it end-to-end.
  arXiv  Detail & Related papers  (2026-06-19T16:51:10Z)
• FailureScope: Cross-Regime Behavioral Diagnosis of Language Model Weaknesses [0.0]
  FailureScope is a behavioral-diagnosis method that clusters evaluation probes by their cross-model pass/fail patterns.<n>We show it yields stable, interpretable failure across three regimes usually studied separately: single-turn benchmarks, multi-turn dialogue, and adversarial agent attacks.
  arXiv  Detail & Related papers  (2026-06-03T01:28:00Z)
• Benchmarking Speech-to-Speech Translation Models [55.00303727199927]
  Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol.<n>We introduce a benchmarking framework integrating 46 metrics across eight dimensions.<n>We deploy it on 1,248 model-language configurations from FLEURS and CVSS.
  arXiv  Detail & Related papers  (2026-06-02T07:01:33Z)
• Fine-Tuning Over Architectural Complexity: Broad-Coverage PII Detection on PIIBench with DeBERTa [0.0]
  PII detection systems are frequently trained within narrow source or domain boundaries, limiting coverage when deployed on heterogeneous text.<n>We study model fine-tuning on a corrected multi-source PIIBench preparation spanning 82 retained entity types across ten source datasets.<n>We evaluate three DeBERTa-based approaches: direct token classification fine-tuning, a source-conditioned hierarchical model (SC+H), and a three-phase curriculum extension (SC+H+Curr)
  arXiv  Detail & Related papers  (2026-05-25T13:12:42Z)
• Benchmarking Patent Embeddings: A Multi-Task Evaluation of 22 Models Across Retrieval, Classification, and Clustering [0.0]
  We evaluate 22 pre-trained embedding models on three tasks -- information retrieval, classification, and clustering.<n>We find that two results cast doubt on the prevailing wisdom.
  arXiv  Detail & Related papers  (2026-05-22T23:51:13Z)
• Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment [53.72927532626824]
  We show that subsets of just 50 examples (0.3% of data) can achieve over 0.93 Pearson correlation with full benchmark scores.<n>To better predict preferences, we trained regression models on these selected subsets, achieving 0.98 correlation.<n>This demonstrates that in regression modeling, well-curated subsets outpredict the full benchmark, showing quality over quantity.
  arXiv  Detail & Related papers  (2026-04-20T00:57:31Z)
• The Evaluation Blind Spot: A Stereological Theory of Benchmark Coverage for Large Language Models [0.0]
  Empirically, three independent leaderboards (Open LLM v2, an extended 12-benchmark suite, LiveBench) all have d_eff in [2.86, 4.80] on their competitive frontier.<n>A submodular greedy algorithm with the Nemhauser (1 - 1/e) guarantee finds a stable core of 4 benchmarks.<n>A counterfactual validation across 12 internal benchmarks and 27 categories confirms that the eigenstructure predicts which evaluations are irreplaceable.
  arXiv  Detail & Related papers  (2026-04-15T08:56:58Z)
• APEX-EM: Non-Parametric Online Learning for Autonomous Agents via Structured Procedural-Episodic Experience Replay [7.370176470430802]
  LLM-based autonomous agents lack persistent procedural memory.<n>We present APEX-EM, a non-parametric online learning framework that accumulates, retrieves, and reuses structured procedural plans.
  arXiv  Detail & Related papers  (2026-03-31T00:24:56Z)
• Binary Token-Level Classification with DeBERTa for All-Type MWE Identification: A Lightweight Approach with Linguistic Enhancement [1.8429656136522097]
  We present a comprehensive approach for multiword expression (MWE) identification that combines binary token-level classification, linguistic feature integration, and data augmentation.<n>Our DeBERTa-v3-large model achieves 69.8% F1 on the CoAM dataset, surpassing the best results (Qwen-72B, 57.8% F1) on this dataset by 12 points while using 165x fewer parameters.
  arXiv  Detail & Related papers  (2026-01-27T08:42:54Z)
• Hierarchical Self-Supervised Representation Learning for Depression Detection from Speech [51.14752758616364]
  Speech-based depression detection (SDD) is a promising, non-invasive alternative to traditional clinical assessments.<n>We propose HAREN-CTC, a novel architecture that integrates multi-layer SSL features using cross-attention within a multitask learning framework.<n>The model achieves state-of-the-art macro F1-scores of 0.81 on DAIC-WOZ and 0.82 on MODMA, outperforming prior methods across both evaluation scenarios.
  arXiv  Detail & Related papers  (2025-10-05T09:32:12Z)
• Improving Long-Tailed Object Detection with Balanced Group Softmax and Metric Learning [0.0]
  We tackle the problem of long-tailed 2D object detection using the LVISv1 dataset.<n>We employ a two-stage Faster R-CNN architecture and propose enhancements to the Balanced Group Softmax framework.<n>Our approach achieves a new state-of-the-art performance with a mean Average Precision (mAP) of 24.5%, surpassing the previous benchmark of 24.0%.
  arXiv  Detail & Related papers  (2025-09-02T00:38:13Z)
• LCFO: Long Context and Long Form Output Dataset and Benchmarking [50.44679440167169]
  Long Context and Form Output (LCFO) benchmark is a novel framework for assessing gradual summarization and summary expansion capabilities.<n>LCFO consists of long input documents (5k words average length) with three summaries of different lengths.<n> GPT-4o-mini achieves best human scores among automatic systems in both summarization and summary expansion tasks.
  arXiv  Detail & Related papers  (2024-12-11T10:35:45Z)
• Law of the Weakest Link: Cross Capabilities of Large Language Models [102.91861246827797]
  We show that Large Language Models (LLMs) exhibit the "Law of the Weakest Link," where cross-capability performance is significantly constrained by the weakest component. These results highlight the under-performance of LLMs in cross-capability tasks.
  arXiv  Detail & Related papers  (2024-09-30T05:12:01Z)
• Balanced Classification: A Unified Framework for Long-Tailed Object Detection [74.94216414011326]
  Conventional detectors suffer from performance degradation when dealing with long-tailed data due to a classification bias towards the majority head categories. We introduce a unified framework called BAlanced CLassification (BACL), which enables adaptive rectification of inequalities caused by disparities in category distribution. BACL consistently achieves performance improvements across various datasets with different backbones and architectures.
  arXiv  Detail & Related papers  (2023-08-04T09:11:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.