The Order Is The Message

• URL: http://arxiv.org/abs/2603.25047v1
• Date: Tue, 24 Mar 2026 22:14:33 GMT
• Title: The Order Is The Message
• Abstract summary: Two fixed-ordering strategies achieve 99.5% test accuracy by epochs 487 and 659 respectively from a training set comprising 0.3% of the input space.<n>We discuss implications for training efficiency, the reinterpretation of grokking, and the safety risks of a channel that evades all content-level auditing.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In a controlled experiment on modular arithmetic ($p = 9973$), varying only example ordering while holding all else constant, two fixed-ordering strategies achieve 99.5\% test accuracy by epochs 487 and 659 respectively from a training set comprising 0.3\% of the input space, well below established sample complexity lower bounds for this task under IID ordering. The IID baseline achieves 0.30\% after 5{,}000 epochs from identical data. An adversarially structured ordering suppresses learning entirely. The generalizing model reliably constructs a Fourier representation whose fundamental frequency is the Fourier dual of the ordering structure, encoding information present in no individual training example, with the same fundamental emerging across all seeds tested regardless of initialization or training set composition. We discuss implications for training efficiency, the reinterpretation of grokking, and the safety risks of a channel that evades all content-level auditing.

Related papers

• OPEN-1B: A Fully Auditable Training Run [0.7223361655030193]
  We introduce a new tier of model transparency, fully auditable, in which every operation on every data sample during training is independently reproducible on commodity hardware.<n>We make it possible to replay any individual step of a large, distributed training run on a single piece of commodity hardware and check it against the published trajectory.<n>We release Open-1B, a model trained under this regime, together with its full pretraining dataset, every intermediate checkpoint, the training, and the audit harness needed to reproduce and verify any step of its training.
  arXiv  Detail & Related papers  (2026-09-15T16:23:08Z)
• SAM3-LoRA: Parameter-Efficient Adaptation of a Concept-Promptable Foundation Model for Multi-Class Structural Defect Segmentation [0.0]
  This study applies Low-Rank Adaptation (LoRA) to SAM3 for multi-class structural defect segmentation.<n>We describe a supervision procedure that trains a concept-promptable model directly from COCO-style class-labeled instance segmentation.
  arXiv  Detail & Related papers  (2026-08-31T23:19:59Z)
• Verification-Aware Training for Speculative Decoding [57.84976863792784]
  We introduce Verification-Aware Training (VAT), a plug-in framework that simulates verification at every training step and turns the resulting accept and reject patterns into supervision.<n>VAT modifies only the training objective, so it can be layered on top of existing methods without changing the draft architecture, the target model, or the inference procedure.<n>VAT improves average acceptance length by up to 11.4% and wall-clock speedup by up to 8.7%, with consistent gains across math, code, and chat benchmarks.
  arXiv  Detail & Related papers  (2026-08-31T01:42:10Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• Bit-Identical Medical Deep Learning via Structured Orthogonal Initialization [0.0]
  Deep learning training is non-deterministic: identical code with different random seeds produces models that disagree on aggregate metrics but disagree on individual predictions.<n>We present a framework for verified bit-identical training that eliminates three sources of randomness.
  arXiv  Detail & Related papers  (2026-03-30T05:04:57Z)
• Data Repetition Beats Data Scaling in Long-CoT Supervised Fine-Tuning [43.11305591635628]
  Olmo3-7B trained for 128 epochs on 400 samples outperforms the equivalent 1 epoch on 51200 samples by 12-26 percentage points.<n>We find that training token accuracy reliably signals when repetition has saturated.
  arXiv  Detail & Related papers  (2026-02-11T18:58:54Z)
• Architecture-Agnostic Curriculum Learning for Document Understanding: Empirical Evidence from Text-Only and Multimodal [13.329839705160927]
  We investigate whether progressive data scheduling yields consistent efficiency gains across document understanding models.<n>We establish that this schedule reduces wall-clock training time by approximately 33%, commensurate with the reduction from 6.67 to 10.0 effective epoch-reductions of data.
  arXiv  Detail & Related papers  (2026-02-02T10:09:26Z)
• Rethinking Zero-Shot Time Series Classification: From Task-specific Classifiers to In-Context Inference [34.15455500037426]
  We propose TIC-FM, an in-context learning framework that treats the labeled training set as context and predicts labels for all test instances in a single forward pass.<n>TIC-FM pairs a time series encoder and a lightweight projection adapter with a split-masked latent memory Transformer.<n>Experiments on 128 UCR datasets show strong accuracy, with consistent gains in the extreme low-label situation.
  arXiv  Detail & Related papers  (2026-01-31T09:17:45Z)
• Leveraging Randomness in Model and Data Partitioning for Privacy Amplification [8.52745154080651]
  We study how inherent randomness in the training process can be leveraged for privacy amplification.<n>This includes (1) data partitioning, where a sample participates in only a subset of training iterations, and (2) model partitioning, where a sample updates only a subset of the model parameters.<n>Our results demonstrate that randomness in the training process, which is structured rather than i.i.d. and interacts with data in complex ways, can be systematically leveraged for significant privacy amplification.
  arXiv  Detail & Related papers  (2025-03-04T22:49:59Z)
• The Right Time Matters: Data Arrangement Affects Zero-Shot Generalization in Instruction Tuning [86.19804569376333]
  We show that zero-shot generalization happens very early during instruction tuning.<n>We propose a more grounded training data arrangement framework, Test-centric Multi-turn Arrangement.
  arXiv  Detail & Related papers  (2024-06-17T16:40:21Z)
• Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models [107.05966685291067]
  We propose test-time prompt tuning (TPT) to learn adaptive prompts on the fly with a single test sample. TPT improves the zero-shot top-1 accuracy of CLIP by 3.6% on average. In evaluating cross-dataset generalization with unseen categories, TPT performs on par with the state-of-the-art approaches that use additional training data.
  arXiv  Detail & Related papers  (2022-09-15T17:55:11Z)
• BatchFormer: Learning to Explore Sample Relationships for Robust Representation Learning [93.38239238988719]
  We propose to enable deep neural networks with the ability to learn the sample relationships from each mini-batch. BatchFormer is applied into the batch dimension of each mini-batch to implicitly explore sample relationships during training. We perform extensive experiments on over ten datasets and the proposed method achieves significant improvements on different data scarcity applications.
  arXiv  Detail & Related papers  (2022-03-03T05:31:33Z)
• Uncertainty-aware Self-training for Text Classification with Few Labels [54.13279574908808]
  We study self-training as one of the earliest semi-supervised learning approaches to reduce the annotation bottleneck. We propose an approach to improve self-training by incorporating uncertainty estimates of the underlying neural network. We show our methods leveraging only 20-30 labeled samples per class for each task for training and for validation can perform within 3% of fully supervised pre-trained language models.
  arXiv  Detail & Related papers  (2020-06-27T08:13:58Z)
• Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning [61.32992639292889]
  Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks. We introduce a new scoring method that casts a plausibility ranking task in a full-text format. We show that our method provides a much more stable training phase across random restarts.
  arXiv  Detail & Related papers  (2020-04-29T10:54:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.