A Study of Adaptive Modeling Towards Robust Generalization

• URL: http://arxiv.org/abs/2602.02780v2
• Date: Thu, 05 Feb 2026 07:57:58 GMT
• Title: A Study of Adaptive Modeling Towards Robust Generalization
• Authors: Zihao Jing, Qiuhao Zeng, Ruiyi Fang, Yan Yi Li, Yan Sun, Boyu Wang, Pingzhao Hu,
• Abstract summary: We present a unified all-atom framework that grounds language reasoning in geometric information while adaptively scaling structural tokens.<n>Across diverse all-atom benchmarks, the proposed approach yields consistent gains in heterogeneous structure-grounded reasoning.
• Score: 14.00955228748485
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) increasingly support reasoning over biomolecular structures, but most existing approaches remain modality-specific and rely on either sequence-style encodings or fixed-length connector tokens for structural inputs. These designs can under-expose explicit geometric cues and impose rigid fusion bottlenecks, leading to over-compression and poor token allocation as structural complexity grows. We present a unified all-atom framework that grounds language reasoning in geometric information while adaptively scaling structural tokens. The method first constructs variable-size structural patches on molecular graphs using an instruction-conditioned gating policy, enabling complexity-aware allocation of query tokens. It then refines the resulting patch tokens via cross-attention with modality embeddings and injects geometry-informed tokens into the language model to improve structure grounding and reduce structural hallucinations. Across diverse all-atom benchmarks, the proposed approach yields consistent gains in heterogeneous structure-grounded reasoning. An anonymized implementation is provided in the supplementary material.

Related papers

• Communication-Inspired Tokenization for Structured Image Representations [74.17163003465537]
  COMmunication inspired Tokenization (COMiT) is a framework for learning structured discrete visual token sequences.<n>Our experiments demonstrate that while semantic alignment provides grounding, attentive sequential tokenization is critical for inducing interpretable, object-centric token structure.
  arXiv  Detail & Related papers  (2026-02-24T09:53:50Z)
• Adaptive Protein Tokenization [0.0]
  Existing protein structure tokenizers create tokens by pooling information from local neighborhoods.<n>We present a method for global tokenization of protein structures in which successive tokens contribute increasing levels of detail to a global representation.<n>We show how adaptive tokens enable inference criteria based on information content, which boosts designability.
  arXiv  Detail & Related papers  (2026-02-06T06:15:14Z)
• <SOG_k>: One LLM Token for Explicit Graph Structural Understanding [57.017902343605364]
  We propose to incorporate one special token SOG_k> to fully represent the Structure Of Graph within a unified token space.<n>SOG_k> empowers LLMs to understand, generate, and reason in a concise and accurate manner.
  arXiv  Detail & Related papers  (2026-02-02T07:55:09Z)
• Improving LLM Reasoning with Homophily-aware Structural and Semantic Text-Attributed Graph Compression [55.51959317490934]
  Large language models (LLMs) have demonstrated promising capabilities in Text-Attributed Graph (TAG) understanding.<n>We argue that graphs inherently contain rich structural and semantic information, and that their effective exploitation can unlock potential gains in LLMs reasoning performance.<n>We propose Homophily-aware Structural and Semantic Compression for LLMs (HS2C), a framework centered on exploiting graph homophily.
  arXiv  Detail & Related papers  (2026-01-13T03:35:18Z)
• From Context to EDUs: Faithful and Structured Context Compression via Elementary Discourse Unit Decomposition [46.36937947958481]
  We introduce a novel explicit compression framework designed to preserve both global structure and fine-grained details.<n>Our approach reformulates a structural context compression as a structure-then-select process.<n>Our method achieves state-of-the-art structural prediction accuracy and significantly outperforms frontier LLMs.
  arXiv  Detail & Related papers  (2025-12-16T09:52:58Z)
• From Static Structures to Ensembles: Studying and Harnessing Protein Structure Tokenization [15.864659611818661]
  Protein structure tokenization converts 3D structures into discrete or vectorized representations.<n>Despite many recent works on structure tokenization, the properties of the underlying discrete representations are not well understood.<n>We show that the successful utilization of structural tokens in a language model for structure prediction depends on using rich, pre-trained sequence embeddings.
  arXiv  Detail & Related papers  (2025-11-13T07:58:24Z)
• Structure-R1: Dynamically Leveraging Structural Knowledge in LLM Reasoning through Reinforcement Learning [29.722512436773638]
  We propose textscStructure-R1, a framework that transforms retrieved content into structured representations optimized for reasoning.<n>We show that textscStructure-R1 consistently achieves competitive performance with a 7B-scale backbone model.<n>Our theoretical analysis demonstrates how structured representations enhance reasoning by improving information density and contextual clarity.
  arXiv  Detail & Related papers  (2025-10-16T23:19:28Z)
• A Hierarchical Quantized Tokenization Framework for Task-Adaptive Graph Representation Learning [8.608851021844576]
  This work presents a hierarchical quantization framework that introduces a self-weighted mechanism for task-adaptive aggregation across multiple scales.<n> Experiments on benchmark datasets for node classification and link prediction demonstrate consistent improvements over strong baselines under comparable computational budgets.
  arXiv  Detail & Related papers  (2025-10-14T10:36:43Z)
• ReSSFormer: A Recursive Sparse Structured Transformer for Scalable and Long-Context Reasoning [0.0]
  We present ReSSFormer, a Recursive Sparse Structured Transformer that integrates three complementary innovations.<n>ReSSFormer replaces conventional depth stacking with recurrent inference, substitutes full attention with token- and expert-level sparsity, and models latent token topology directly from content.
  arXiv  Detail & Related papers  (2025-10-02T02:05:30Z)
• Contextually Structured Token Dependency Encoding for Large Language Models [0.0]
  Self-attention mechanisms capture dynamic contextual dependencies, but their reliance on learned weight distributions limits the preservation of long-range hierarchical structures in generated sequences.<n>Dependency-aware token encoding introduces a structured approach to embedding, ensuring relational constraints are embedded within token representations.<n> Empirical evaluations indicate reductions in perplexity across diverse linguistic benchmarks, suggesting improvements in contextual coherence and predictive consistency in autoregressive text generation.
  arXiv  Detail & Related papers  (2025-01-30T08:51:48Z)
• CBGBench: Fill in the Blank of Protein-Molecule Complex Binding Graph [66.11279161533619]
  CBGBench is a benchmark for structure-based drug design (SBDD) By categorizing existing methods based on their attributes, CBGBench implements various cutting-edge methods. We have adapted these models to a range of tasks essential in drug design, which are considered sub-tasks within the graph fill-in-the-blank tasks.
  arXiv  Detail & Related papers  (2024-06-16T08:20:24Z)
• Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings [60.698130703909804]
  Transformers generalize to novel compositions of structures and entities after being trained on a complex dataset. We propose SQ-Transformer that explicitly encourages systematicity in the embeddings and attention layers. We show that SQ-Transformer achieves stronger compositional generalization than the vanilla Transformer on multiple low-complexity semantic parsing and machine translation datasets.
  arXiv  Detail & Related papers  (2024-02-09T15:53:15Z)
• FoldToken: Learning Protein Language via Vector Quantization and Beyond [56.19308144551836]
  We introduce textbfFoldTokenizer to represent protein sequence-structure as discrete symbols. We refer to the learned symbols as textbfFoldToken, and the sequence of FoldTokens serves as a new protein language.
  arXiv  Detail & Related papers  (2024-02-04T12:18:51Z)
• Physics of Language Models: Part 1, Learning Hierarchical Language Structures [51.68385617116854]
  Transformer-based language models are effective but complex, and understanding their inner workings and reasoning mechanisms is a significant challenge.<n>We introduce a family of synthetic CFGs that produce hierarchical rules, capable of generating lengthy sentences.<n>We demonstrate that generative models like GPT can accurately learn and reason over CFG-defined hierarchies and generate sentences based on it.
  arXiv  Detail & Related papers  (2023-05-23T04:28:16Z)
• DepGraph: Towards Any Structural Pruning [68.40343338847664]
  We study general structural pruning of arbitrary architecture like CNNs, RNNs, GNNs and Transformers. We propose a general and fully automatic method, emphDependency Graph (DepGraph), to explicitly model the dependency between layers and comprehensively group parameters for pruning. In this work, we extensively evaluate our method on several architectures and tasks, including ResNe(X)t, DenseNet, MobileNet and Vision transformer for images, GAT for graph, DGCNN for 3D point cloud, alongside LSTM for language, and demonstrate that, even with a
  arXiv  Detail & Related papers  (2023-01-30T14:02:33Z)
• Autoregressive Structured Prediction with Language Models [73.11519625765301]
  We describe an approach to model structures as sequences of actions in an autoregressive manner with PLMs. Our approach achieves the new state-of-the-art on all the structured prediction tasks we looked at.
  arXiv  Detail & Related papers  (2022-10-26T13:27:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.