Jet Expansions of Residual Computation

• URL: http://arxiv.org/abs/2410.06024v1
• Date: Tue, 8 Oct 2024 13:25:08 GMT
• Title: Jet Expansions of Residual Computation
• Authors: Yihong Chen, Xiangxiang Xu, Yao Lu, Pontus Stenetorp, Luca Franceschi,
• Abstract summary: We introduce a framework for expanding residual computational graphs using jets. Our method provides a systematic approach to disentangle contributions of different computational paths to model predictions.
• Score: 25.842534423280185
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a framework for expanding residual computational graphs using jets, operators that generalize truncated Taylor series. Our method provides a systematic approach to disentangle contributions of different computational paths to model predictions. In contrast to existing techniques such as distillation, probing, or early decoding, our expansions rely solely on the model itself and requires no data, training, or sampling from the model. We demonstrate how our framework grounds and subsumes logit lens, reveals a (super-)exponential path structure in the recursive residual depth and opens up several applications. These include sketching a transformer large language model with $n$-gram statistics extracted from its computations, and indexing the models' levels of toxicity knowledge. Our approach enables data-free analysis of residual computation for model interpretability, development, and evaluation.

Related papers

• Bayesian Data Sketching for Varying Coefficient Regression Models [1.6727186769396276]
  We introduce Bayesian data sketching for varying coefficient models to obviate computational challenges presented by large sample sizes.<n>Our approach distinguishes itself from several existing methods for analyzing large functional data.<n>Well-established methods and algorithms for varying coefficient regression models can be applied to the compressed data.
  arXiv  Detail & Related papers  (2025-05-30T22:09:06Z)
• Amortizing intractable inference in diffusion models for vision, language, and control [89.65631572949702]
  This paper studies amortized sampling of the posterior over data, $mathbfxsim prm post(mathbfx)propto p(mathbfx)r(mathbfx)$, in a model that consists of a diffusion generative model prior $p(mathbfx)$ and a black-box constraint or function $r(mathbfx)$. We prove the correctness of a data-free learning objective, relative trajectory balance, for training a diffusion model that samples from
  arXiv  Detail & Related papers  (2024-05-31T16:18:46Z)
• Scaling and renormalization in high-dimensional regression [72.59731158970894]
  This paper presents a succinct derivation of the training and generalization performance of a variety of high-dimensional ridge regression models. We provide an introduction and review of recent results on these topics, aimed at readers with backgrounds in physics and deep learning.
  arXiv  Detail & Related papers  (2024-05-01T15:59:00Z)
• Likelihood Based Inference in Fully and Partially Observed Exponential Family Graphical Models with Intractable Normalizing Constants [4.532043501030714]
  Probabilistic graphical models that encode an underlying Markov random field are fundamental building blocks of generative modeling. This paper is to demonstrate that full likelihood based analysis of these models is feasible in a computationally efficient manner.
  arXiv  Detail & Related papers  (2024-04-27T02:58:22Z)
• Efficient and Generalizable Certified Unlearning: A Hessian-free Recollection Approach [8.875278412741695]
  Machine unlearning strives to uphold the data owners' right to be forgotten by enabling models to selectively forget specific data. We develop an algorithm that achieves near-instantaneous unlearning as it only requires a vector addition operation.
  arXiv  Detail & Related papers  (2024-04-02T07:54:18Z)
• Fusion of Gaussian Processes Predictions with Monte Carlo Sampling [61.31380086717422]
  In science and engineering, we often work with models designed for accurate prediction of variables of interest. Recognizing that these models are approximations of reality, it becomes desirable to apply multiple models to the same data and integrate their outcomes.
  arXiv  Detail & Related papers  (2024-03-03T04:21:21Z)
• Discovering Interpretable Physical Models using Symbolic Regression and Discrete Exterior Calculus [55.2480439325792]
  We propose a framework that combines Symbolic Regression (SR) and Discrete Exterior Calculus (DEC) for the automated discovery of physical models. DEC provides building blocks for the discrete analogue of field theories, which are beyond the state-of-the-art applications of SR to physical problems. We prove the effectiveness of our methodology by re-discovering three models of Continuum Physics from synthetic experimental data.
  arXiv  Detail & Related papers  (2023-10-10T13:23:05Z)
• Discovering interpretable elastoplasticity models via the neural polynomial method enabled symbolic regressions [0.0]
  Conventional neural network elastoplasticity models are often perceived as lacking interpretability. This paper introduces a two-step machine learning approach that returns mathematical models interpretable by human experts.
  arXiv  Detail & Related papers  (2023-07-24T22:22:32Z)
• Generalizing Backpropagation for Gradient-Based Interpretability [103.2998254573497]
  We show that the gradient of a model is a special case of a more general formulation using semirings. This observation allows us to generalize the backpropagation algorithm to efficiently compute other interpretable statistics.
  arXiv  Detail & Related papers  (2023-07-06T15:19:53Z)
• Interpretable and Scalable Graphical Models for Complex Spatio-temporal Processes [3.469001874498102]
  thesis focuses on data that has complex-temporal structure and on probabilistic graphical models that learn the structure in an interpretable and interpretable manner. practical applications of the methodology are considered using real datasets. This includes brain-connectivity analysis using data, space weather forecasting using solar imaging data, longitudinal analysis of public opinions using Twitter data, and mining of mental health related issues using TalkLife data.
  arXiv  Detail & Related papers  (2023-01-15T05:39:30Z)
• Generative Principal Component Analysis [47.03792476688768]
  We study the problem of principal component analysis with generative modeling assumptions. Key assumption is that the underlying signal lies near the range of an $L$-Lipschitz continuous generative model with bounded $k$-dimensional inputs. We propose a quadratic estimator, and show that it enjoys a statistical rate of order $sqrtfracklog Lm$, where $m$ is the number of samples.
  arXiv  Detail & Related papers  (2022-03-18T01:48:16Z)
• Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modeling [54.94763543386523]
  Current autoencoder-based disentangled representation learning methods achieve disentanglement by penalizing the ( aggregate) posterior to encourage statistical independence of the latent factors. We present a novel multi-stage modeling approach where the disentangled factors are first learned using a penalty-based disentangled representation learning method. Then, the low-quality reconstruction is improved with another deep generative model that is trained to model the missing correlated latent variables.
  arXiv  Detail & Related papers  (2020-10-25T18:51:15Z)
• Predicting Multidimensional Data via Tensor Learning [0.0]
  We develop a model that retains the intrinsic multidimensional structure of the dataset. To estimate the model parameters, an Alternating Least Squares algorithm is developed. The proposed model is able to outperform benchmark models present in the forecasting literature.
  arXiv  Detail & Related papers  (2020-02-11T11:57:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.