What Linear Probes Miss: Multi-View Probing for Weight-Space Learning

• URL: http://arxiv.org/abs/2605.23410v1
• Date: Fri, 22 May 2026 09:18:01 GMT
• Title: What Linear Probes Miss: Multi-View Probing for Weight-Space Learning
• Abstract summary: MVProbe is a multi-perspective probing framework that synthesizes first-order signals with interaction-aware views.<n>On the Model Jungle benchmark, MVProbe consistently outperforms the state-of-the-art ProbeX.
• Score: 8.474818775416637
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The explosive growth of open-source model repositories has created a Model Jungle, where checkpoints are frequently shared without adequate documentation or metadata. While weight-space learning offers a pathway to identify and analyze these models directly from their parameters, processing full-scale weights is computationally prohibitive. Probing-based methods have emerged as a lightweight alternative, extracting permutation-equivariant representations via learnable probe vectors. However, existing probing methods are limited by a single-view design: they capture first-order structures but fail to encode the rich, higher-order correlation patterns inherent in row-column interactions. To bridge this gap, we introduce MVProbe, a multi-perspective probing framework that synthesizes first-order signals with interaction-aware (Gram-based) views. Our approach is theoretically grounded; we analyze the scaling laws of different probing orders to derive a principled standardization and fusion strategy that ensures balanced contributions from all branches. On the Model Jungle benchmark, MVProbe consistently outperforms the state-of-the-art ProbeX across diverse architectures, including discriminative backbones (ResNet, SupViT, MAE, DINO) and large-scale generative LoRA adapters (Stable Diffusion LoRA).

Related papers

• PACT: Preserving Anchored Cores in Task-vectors for Model Merging [68.52455853496585]
  Model merging aims to combine multiple task-specific fine-tuned models into a single multi-task model.<n>Most existing model merging approaches follow the Task Arithmetic paradigm.<n>We propose PACT, which preserves the anchored task-specific cores (i.e., LBW dimensions) within task vectors by aligning their complements with the subspace of the pre-trained weights.
  arXiv  Detail & Related papers  (2026-06-17T02:48:35Z)
• Aligning Few-Step Generative Models by Amortizing Sample-based Variational Inference [60.82875447631739]
  We introduce FAV, Few-step Generative Models Alignment via Sample-based Variational Inference.<n>FAV cast alignment as sampling from a reward-tilted distribution anchored to a reference distribution.<n>We evaluate FAV on two domains: robotics manipulation and image generator alignment.
  arXiv  Detail & Related papers  (2026-05-26T05:02:49Z)
• ABLE: Representing and Mapping LLMs via Attribution-Based Large-model Embedding [23.27523995613204]
  We propose ABLE (Attribution-Based Large-model Embedding), a framework that leverages the interpretability space to construct model representations.<n>By aggregating gradient-based feature attributions via a tokenizer-agnostic word-level alignment, ABLE captures model-specific input-sensitivity patterns.<n>Our training-free approach achieves competitive or superior performance in relation prediction, model routing, and benchmark score prediction.
  arXiv  Detail & Related papers  (2026-04-20T04:55:11Z)
• Hierarchical Inference and Closure Learning via Adaptive Surrogates for ODEs and PDEs [15.38864225184245]
  Inverse problems are the task of calibrating models to match data.<n>We develop a principled methodology for leveraging data from collections of distinct yet related physical systems.<n>We learn the shared unknown dynamics in the form of an ML-based closure model.
  arXiv  Detail & Related papers  (2026-03-04T10:30:08Z)
• SimGR: Escaping the Pitfalls of Generative Decoding in LLM-based Recommendation [68.00727783181289]
  A core objective in recommender systems is to accurately model the distribution of user preferences over items to enable personalized recommendations.<n>We observe that existing methods inevitably introduce systematic bias when estimating item-level preference distributions.<n>We propose textbfSimply textbfGenerative textbfRecommendation (textbfSimGR), a framework that directly models item-level preference distributions in a shared latent space.
  arXiv  Detail & Related papers  (2026-02-08T07:26:52Z)
• AdaJudge: Adaptive Multi-Perspective Judging for Reward Modeling [23.81351558826977]
  We propose AdaJudge, a unified framework that jointly adapts representation and aggregation.<n>AdaJudge first refines backbone representations into a discrimination-oriented space via refinement blocks.<n>It then replaces the static readout with an adaptive multi-view pooling module that dynamically routes and combines evidence.
  arXiv  Detail & Related papers  (2026-01-13T00:37:38Z)
• Theoretical Foundations of Scaling Law in Familial Models [46.506708373314375]
  We introduce Granularity (G) as a fundamental scaling variable alongside model size (N) and training tokens (D)<n>Our results reveal that the granularity penalty follows a multiplicative power law with an extremely small exponent.<n> Practically, it validates the "train once, deploy many" paradigm, demonstrating that deployment flexibility is achievable.
  arXiv  Detail & Related papers  (2025-12-29T12:01:58Z)
• Model Merging via Multi-Teacher Knowledge Distillation [11.543771846135021]
  We introduce a novel flatness-aware PAC-Bayes generalization bound specifically for the model merging setting.<n>We frame model merging as multi-teacher knowledge distillation on scarce, unlabeled data.<n>We formally demonstrate that minimizing the student-teacher Kullback-Leibler divergence directly tightens the upper bound on the merged model's excess risk.
  arXiv  Detail & Related papers  (2025-12-24T17:10:44Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• Bilinear Convolution Decomposition for Causal RL Interpretability [0.0]
  Efforts to interpret reinforcement learning (RL) models often rely on high-level techniques such as attribution or probing.<n>This work proposes replacing nonlinearities in convolutional neural networks (ConvNets) with bilinear variants, to produce a class of models for which these limitations can be addressed.<n>We show bilinear model variants perform comparably in model-free reinforcement learning settings, and give a side by side comparison on ProcGen environments.
  arXiv  Detail & Related papers  (2024-12-01T19:32:04Z)
• Single-Stage Visual Relationship Learning using Conditional Queries [60.90880759475021]
  TraCQ is a new formulation for scene graph generation that avoids the multi-task learning problem and the entity pair distribution. We employ a DETR-based encoder-decoder conditional queries to significantly reduce the entity label space as well. Experimental results show that TraCQ not only outperforms existing single-stage scene graph generation methods, it also beats many state-of-the-art two-stage methods on the Visual Genome dataset.
  arXiv  Detail & Related papers  (2023-06-09T06:02:01Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• Unsupervised Controllable Generation with Self-Training [90.04287577605723]
  controllable generation with GANs remains a challenging research problem. We propose an unsupervised framework to learn a distribution of latent codes that control the generator through self-training. Our framework exhibits better disentanglement compared to other variants such as the variational autoencoder.
  arXiv  Detail & Related papers  (2020-07-17T21:50:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.