When 2D Tasks Meet 1D Serialization: On Serialization Friction in Structured Tasks

• URL: http://arxiv.org/abs/2604.27272v1
• Date: Wed, 29 Apr 2026 23:58:45 GMT
• Title: When 2D Tasks Meet 1D Serialization: On Serialization Friction in Structured Tasks
• Abstract summary: Large language models process structured inputs as 1D token sequences.<n>Row-column alignment and local neighborhoods are no longer directly expressed in the input.<n>We compare a text-only language pathway over serialized inputs with a vision-augmented pathway.
• Score: 49.84804136268971
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) conventionally process structured inputs as 1D token sequences. While natural for prose, such linearization may introduce additional representational burden for tasks whose computation depends directly on explicit 2D structure, because row--column alignment and local neighborhoods are no longer directly expressed in the input. We study this setting, which we refer to as serialization friction, on a small diagnostic testbed of synthetic tasks with explicit 2D structure: matrix transpose, Conway's Game of Life, and LU decomposition. To examine this question, we compare a text-only language pathway over serialized inputs with a vision-augmented pathway, built on the same language backbone, that receives the same underlying content rendered in task-faithful 2D layout, yielding a system-level comparison between two end-to-end input pathways. Across the tasks and settings we study, the visual pathway consistently outperforms the textual pathway; the gap often widens at larger dimensions, and error patterns under serialization become increasingly spatially structured. These findings indicate that the relationship between input representation and model performance on such tasks warrants further investigation, and suggest that preserving task-relevant 2D layout is a promising direction for structured 2D tasks.

Related papers

• Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D [19.684508420641816]
  2D-RoPE organizes text into a 2D grid rather than a 1D sequence and assigns each token a row ID and a column ID.<n>In synthetic copy experiments, shallow Transformers with 2D-RoPE achieve perfect copying at input lengths hundreds of times longer than those seen during training.
  arXiv  Detail & Related papers  (2026-07-17T15:56:52Z)
• How to Utilize Complementary Vision-Text Information for 2D Structure Understanding [55.98339043221623]
  DiVA-Former is a lightweight architecture designed to effectively integrate vision and text information.<n> evaluated across 13 table benchmarks, DiVA-Former improves upon the pure-text baseline by 23.9%.
  arXiv  Detail & Related papers  (2026-03-17T08:30:01Z)
• How Modality Shapes Perception and Reasoning: A Study of Error Propagation in ARC-AGI [7.226300346775942]
  ARC-AGI and ARC-AGI-2 measure generalization-through-composition on small color-quantized grids.<n>Recent instruction-first systems translate grids into concise natural-language or DSL rules executed in generate-execute-select loops.
  arXiv  Detail & Related papers  (2025-11-11T19:06:41Z)
• IGGT: Instance-Grounded Geometry Transformer for Semantic 3D Reconstruction [82.53307702809606]
  Humans naturally perceive the geometric structure and semantic content of a 3D world as intertwined dimensions.<n>We propose InstanceGrounded Geometry Transformer (IGGT) to unify the knowledge for both spatial reconstruction and instance-level contextual understanding.
  arXiv  Detail & Related papers  (2025-10-26T14:57:44Z)
• InstructLayout: Instruction-Driven 2D and 3D Layout Synthesis with Semantic Graph Prior [23.536285325566013]
  Comprehending natural language instructions is a charming property for both 2D and 3D layout synthesis systems.<n>Existing methods implicitly model object joint distributions and express object relations, hindering generation's controllability synthesis systems.<n>We introduce Instruct, a novel generative framework that integrates a semantic graph prior and a layout decoder.
  arXiv  Detail & Related papers  (2024-07-10T12:13:39Z)
• Explicitly Representing Syntax Improves Sentence-to-layout Prediction of Unexpected Situations [21.636786771793364]
  We show that we can predict 2D spatial layouts from language representations that implicitly or explicitly encode sentence syntax.<n>We propose a novel structural loss function that better enforces the syntactic structure of the input sentence.<n>The loss has the potential to be used in other generation tasks where a tree-like structure underlies the conditioning modality.
  arXiv  Detail & Related papers  (2024-01-25T14:53:30Z)
• Multi-task Learning with 3D-Aware Regularization [55.97507478913053]
  We propose a structured 3D-aware regularizer which interfaces multiple tasks through the projection of features extracted from an image encoder to a shared 3D feature space. We show that the proposed method is architecture agnostic and can be plugged into various prior multi-task backbones to improve their performance.
  arXiv  Detail & Related papers  (2023-10-02T08:49:56Z)
• ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting [108.93803186429017]
  End-to-end text-spotting aims to integrate detection and recognition in a unified framework. Here, we tackle end-to-end text spotting by presenting Adaptive Bezier Curve Network v2 (ABCNet v2) Our main contributions are four-fold: 1) For the first time, we adaptively fit arbitrarily-shaped text by a parameterized Bezier curve, which, compared with segmentation-based methods, can not only provide structured output but also controllable representation. Comprehensive experiments conducted on various bilingual (English and Chinese) benchmark datasets demonstrate that ABCNet v2 can achieve state-of-the
  arXiv  Detail & Related papers  (2021-05-08T07:46:55Z)
• A Multi-Task Deep Learning Framework for Building Footprint Segmentation [0.0]
  We propose a joint optimization scheme for the task of building footprint delineation. We also introduce two auxiliary tasks; image reconstruction and building footprint boundary segmentation. In particular, we propose a deep multi-task learning (MTL) based unified fully convolutional framework.
  arXiv  Detail & Related papers  (2021-04-19T15:07:27Z)
• Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting [168.91748514706995]
  We propose two novel cross-modal translation pre-text tasks for self-supervised feature learning: Vectorization and Rasterization. Our learned encoder modules benefit both-based and vector-based downstream approaches to analysing hand-drawn data.
  arXiv  Detail & Related papers  (2021-03-25T09:47:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.