Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D

• URL: http://arxiv.org/abs/2607.16072v1
• Date: Fri, 17 Jul 2026 15:56:52 GMT
• Title: Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D
• Abstract summary: 2D-RoPE organizes text into a 2D grid rather than a 1D sequence and assigns each token a row ID and a column ID.<n>In synthetic copy experiments, shallow Transformers with 2D-RoPE achieve perfect copying at input lengths hundreds of times longer than those seen during training.
• Score: 19.684508420641816
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While large language models (LLMs) can solve advanced reasoning problems in seconds, we show that even frontier models fail to perform a much simpler operation: exactly copying an input string that lies well within their context windows. We attribute this failure to positional encodings in Transformer architectures, whose inductive bias favors copying through a shortcut based on matching local contexts rather than carefully locating the corresponding input positions. To address this issue, we introduce 2D-RoPE, which organizes text into a 2D grid rather than a 1D sequence and assigns each token a row ID and a column ID. Under this view, copying becomes simply retrieving input tokens at a fixed column offset, which makes the task easy to learn. In synthetic copy experiments, shallow Transformers with 2D-RoPE achieve perfect copying at input lengths hundreds of times longer than those seen during training, whereas standard positional encodings fall far behind. We further show that the advantage of 2D-RoPE language models on copy tasks consistently holds in large-scale pretraining on DCLM with model sizes up to 1.4B parameters. Overall, our results suggest that viewing text in 2D can benefit language modeling, and we hope this encourages future work to further explore the potential of 2D positional encodings.

Related papers

• When 2D Tasks Meet 1D Serialization: On Serialization Friction in Structured Tasks [49.84804136268971]
  Large language models process structured inputs as 1D token sequences.<n>Row-column alignment and local neighborhoods are no longer directly expressed in the input.<n>We compare a text-only language pathway over serialized inputs with a vision-augmented pathway.
  arXiv  Detail & Related papers  (2026-04-29T23:58:45Z)
• PatchAlign3D: Local Feature Alignment for Dense 3D Shape understanding [67.15800065888887]
  Current foundation models for 3D shapes excel at global tasks (retrieval, classification) but transfer poorly to local part-level reasoning.<n>We introduce an encoder-only 3D model that produces language-aligned patch-level features directly from point clouds.<n>Our 3D encoder achieves zero-shot 3D part segmentation with fast single-pass inference without any test-time multi-view rendering.
  arXiv  Detail & Related papers  (2026-01-05T18:55:45Z)
• Adapting Human Mesh Recovery with Vision-Language Feedback [17.253535686451897]
  We leverage vision-language models to generate interactive body part descriptions.<n>We train a text encoder and a pose VQ-VAE, aligning texts to body poses in a shared latent space.<n>The model can produce poses with accurate 3D perception and image consistency.
  arXiv  Detail & Related papers  (2025-02-06T07:42:00Z)
• Translatotron-V(ison): An End-to-End Model for In-Image Machine Translation [81.45400849638347]
  In-image machine translation (IIMT) aims to translate an image containing texts in source language into an image containing translations in target language. In this paper, we propose an end-to-end IIMT model consisting of four modules. Our model achieves competitive performance compared to cascaded models with only 70.9% of parameters, and significantly outperforms the pixel-level end-to-end IIMT model.
  arXiv  Detail & Related papers  (2024-07-03T08:15:39Z)
• SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models [84.71887272654865]
  We present SparseCtrl to enable flexible structure control with temporally sparse signals. It incorporates an additional condition to process these sparse signals while leaving the pre-trained T2V model untouched. The proposed approach is compatible with various modalities, including sketches, depth maps, and RGB images.
  arXiv  Detail & Related papers  (2023-11-28T16:33:08Z)
• Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignment [81.73717488887938]
  Language-Quantized AutoEncoder (LQAE) learns to align text-image data in an unsupervised manner by leveraging pretrained language models. LQAE learns to represent similar images with similar clusters of text tokens, thereby aligning these two modalities without the use of aligned text-image pairs. This enables few-shot image classification with large language models (e.g., GPT-3) as well as linear classification of images based on BERT text features.
  arXiv  Detail & Related papers  (2023-02-02T06:38:44Z)
• Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders [52.91248611338202]
  We propose an alternative to obtain superior 3D representations from 2D pre-trained models via Image-to-Point Masked Autoencoders, named as I2P-MAE. By self-supervised pre-training, we leverage the well learned 2D knowledge to guide 3D masked autoencoding. I2P-MAE attains the state-of-the-art 90.11% accuracy, +3.68% to the second-best, demonstrating superior transferable capacity.
  arXiv  Detail & Related papers  (2022-12-13T17:59:20Z)
• MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval [43.2299969152561]
  Methods for text-to-video retrieval on four datasets with both zero-shot and fine-tune evaluation protocols. Our method outperforms state-of-the-art methods for text-to-video retrieval on four datasets with both zero-shot and fine-tune evaluation protocols.
  arXiv  Detail & Related papers  (2022-04-26T16:06:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.