LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models

• URL: http://arxiv.org/abs/2606.04485v2
• Date: Thu, 04 Jun 2026 07:27:03 GMT
• Title: LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models
• Abstract summary: LimiX-2M is a 2M- parameter model that outperforms larger TabPFN-v2 and TabICL baselines on widely used benchmarks.<n>We present a unified tokenize-and-route framework for strong Tabular foundation models (TFMs)<n>Results highlight value-aware tokenization and readout-aligned routing as key levers for improving the accuracy--efficiency trade-off in TFMs.
• Score: 56.999481798138625
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tabular foundation models (TFMs) increasingly rival tree ensembles, but their performance is often compute-inefficient: with standard affine scalar tokenization, each feature injects value variation through an essentially one-dimensional channel, and feature IDs/positional signals cannot increase within-feature value degrees of freedom, yielding weak early-layer value sensitivity and redundant hidden states. We present a unified tokenize-and-route framework for strong TFMs: RaBEL expands each scalar into compact localized RBF features (optionally exponent-gated) to improve conditioning and shallow-layer effective rank, while a reordered bidirectional block S->N->F aligns computation with the readout by aggregating cross-sample context before feature mixing and using attention pooling. Together, these changes yield LimiX-2M, a 2M-parameter model that outperforms larger TabPFN-v2 and TabICL baselines on widely used tabular benchmarks while reducing training and inference costs. These results highlight value-aware tokenization and readout-aligned routing as key levers for improving the accuracy--efficiency trade-off in TFMs. Model checkpoints and inference code are available at https://github.com/limix-ldm-ai/LimiX.

Related papers

• Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs [18.457250148137494]
  We present an Adaptive Depth Sparse Framework (AdaDSF) that converts off-the-shelf pre-trained large language models into depth-sparse models without retraining.<n>On GPT-NeoX and Qwen2.5 over language modeling and commonsense reasoning, AdaDSF substantially reduces inference FLOPs while preserving performance close to dense counterparts.
  arXiv  Detail & Related papers  (2026-07-23T13:13:04Z)
• dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formats [1.3357020021155022]
  dMX is a mixed-precision quantization framework for learnable floating-point bit-width assignment.<n>We study its application for the microscaling floating-point (MXFP) family of data types defined by the Open Compute Project standard.
  arXiv  Detail & Related papers  (2026-06-02T18:23:20Z)
• OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond [50.440302567029654]
  Multi-modal intelligence has made the Key-Value cache a dominant memory bottleneck for efficient deployment.<n>In this work, we revisit the inherent limitations of the per-channel quantization paradigm.<n>We propose OScaR, an accurate and lightweight KV cache compression framework for X-LLMs.
  arXiv  Detail & Related papers  (2026-05-19T10:53:03Z)
• MAny: Merge Anything for Multimodal Continual Instruction Tuning [52.50936513604062]
  textbfMAny (textbfMAny) is a framework that merges task-specific knowledge through textbfCross-modal textbfProjection textbfMerging.<n>textbfLow-rank textbfParameter textbfMerging (textbfLPM)
  arXiv  Detail & Related papers  (2026-04-15T15:57:23Z)
• Scaling Attention via Feature Sparsity [50.64995497733461]
  Scaling Transformers to ultra-long contexts is bottlenecked by the $O(n2 d)$ cost of self-attention.<n>We propose Sparse Feature Attention, where queries and keys are represented as $k$-sparse codes that preserve high-dimensional expressivity.<n>Across GPT-2 and Qwen3 pretraining, SFA matches dense baselines while improving speed by up to $2.5times$ and reducing FLOPs and KV-cache by nearly 50%.
  arXiv  Detail & Related papers  (2026-03-17T08:41:50Z)
• FMS$^2$: Unified Flow Matching for Segmentation and Synthesis of Thin Structures [1.3111058225843968]
  We propose FMS$2$, a flow-matching framework with two modules.<n>SegFlow is a 2.96M- parameter segmentation model built on a standard encoder-decoder backbone.<n> SynFlow is a mask-conditioned mask $rightarrow$ image generator that produces pixel-aligned synthetic image-mask pairs.
  arXiv  Detail & Related papers  (2026-03-14T00:04:25Z)
• LINA: Linear Autoregressive Image Generative Models with Continuous Tokens [56.80443965097921]
  Autoregressive models with continuous tokens form a promising paradigm for visual generation, especially for text-to-image (T2I) synthesis.<n>We study how to design compute-efficient linear attention within this framework.<n>We present LINA, a simple and compute-efficient T2I model built entirely on linear attention, capable of generating high-fidelity 1024x1024 images from user instructions.
  arXiv  Detail & Related papers  (2026-01-30T06:44:33Z)
• vLinear: A Powerful Linear Model for Multivariate Time Series Forecasting [28.587343014443576]
  vecTrans is a lightweight module that utilizes a learnable vector to model multivariate correlations.<n>WFMLoss is an effective plug-and-play objective, consistently improving existing forecasters.
  arXiv  Detail & Related papers  (2026-01-20T09:23:10Z)
• Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning [3.884856136722027]
  We introduce Orion-MSP, a new ICL architecture featuring multi-scale processing, block-sparse attention, and Perceiver-style memory.<n>Orion-MSP matches or surpasses state-of-the-art performance while scaling effectively to high-dimensional tables.
  arXiv  Detail & Related papers  (2025-11-04T18:43:44Z)
• PT$^2$-LLM: Post-Training Ternarization for Large Language Models [52.4629647715623]
  Large Language Models (LLMs) have shown impressive capabilities across diverse tasks, but their large memory and compute demands hinder deployment.<n>We propose PT$2$-LLM, a post-training ternarization framework tailored for LLMs.<n>At its core is an Asymmetric Ternary Quantizer equipped with a two-stage refinement pipeline.
  arXiv  Detail & Related papers  (2025-09-27T03:01:48Z)
• Harnessing Optimization Dynamics for Curvature-Informed Model Merging [17.42364575754576]
  In supervised fine-tuning, multiple capability-based SFT checkpoints must be consolidated into a single model.<n>We introduce Optimization Trajectory Aware (OTA) Merging and Fast Fisher Grafting (FFG)<n>OTA+FFG improves merged-model quality over strong weight-space baselines, reduces negative transfer, and remains robust across sparsity levels.
  arXiv  Detail & Related papers  (2025-09-14T08:59:53Z)
• DFedADMM: Dual Constraints Controlled Model Inconsistency for Decentralized Federated Learning [52.83811558753284]
  Decentralized learning (DFL) discards the central server and establishes a decentralized communication network. Existing DFL methods still suffer from two major challenges: local inconsistency and local overfitting.
  arXiv  Detail & Related papers  (2023-08-16T11:22:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.