Are Logistic Models Really Interpretable?

• URL: http://arxiv.org/abs/2406.13427v1
• Date: Wed, 19 Jun 2024 10:36:38 GMT
• Title: Are Logistic Models Really Interpretable?
• Authors: Danial Dervovic, Freddy Lécué, Nicolás Marchesotti, Daniele Magazzeni,
• Abstract summary: One of the simplest AI classification models, Logistic Regression, has an unwieldy interpretation of its model weights. We show via a User Study that skilled participants are unable to reliably reproduce the action of small LR models. We argue that Linearised Additive Models (LAMs) are more interpretable than logistic models.
• Score: 10.239620808245448
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The demand for open and trustworthy AI models points towards widespread publishing of model weights. Consumers of these model weights must be able to act accordingly with the information provided. That said, one of the simplest AI classification models, Logistic Regression (LR), has an unwieldy interpretation of its model weights, with greater difficulties when extending LR to generalised additive models. In this work, we show via a User Study that skilled participants are unable to reliably reproduce the action of small LR models given the trained parameters. As an antidote to this, we define Linearised Additive Models (LAMs), an optimal piecewise linear approximation that augments any trained additive model equipped with a sigmoid link function, requiring no retraining. We argue that LAMs are more interpretable than logistic models -- survey participants are shown to solve model reasoning tasks with LAMs much more accurately than with LR given the same information. Furthermore, we show that LAMs do not suffer from large performance penalties in terms of ROC-AUC and calibration with respect to their logistic counterparts on a broad suite of public financial modelling data.

Related papers

• Scaling Diffusion Language Models via Adaptation from Autoregressive Models [105.70889434492143]
  Diffusion Language Models (DLMs) have emerged as a promising new paradigm for text generative modeling. We show that we can convert AR models ranging from 127M to 7B parameters into diffusion models DiffuGPT and DiffuLLaMA, using less than 200B tokens for training. Our experimental results reveal that these models outperform earlier DLMs and are competitive with their AR counterparts.
  arXiv  Detail & Related papers  (2024-10-23T14:04:22Z)
• REEF: Representation Encoding Fingerprints for Large Language Models [53.679712605506715]
  REEF computes and compares the centered kernel alignment similarity between the representations of a suspect model and a victim model. This training-free REEF does not impair the model's general capabilities and is robust to sequential fine-tuning, pruning, model merging, and permutations.
  arXiv  Detail & Related papers  (2024-10-18T08:27:02Z)
• Investigating the Impact of Model Complexity in Large Language Models [3.7919508292745676]
  Large Language Models (LLMs) based on the pre-trained fine-tuning paradigm have become pivotal in solving natural language processing tasks. In this paper, we focus on autoregressive LLMs and propose to employ Hidden Markov Models (HMMs) to model them.
  arXiv  Detail & Related papers  (2024-10-01T13:53:44Z)
• EMR-Merging: Tuning-Free High-Performance Model Merging [55.03509900949149]
  We show that Elect, Mask & Rescale-Merging (EMR-Merging) shows outstanding performance compared to existing merging methods. EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance.
  arXiv  Detail & Related papers  (2024-05-23T05:25:45Z)
• RewardBench: Evaluating Reward Models for Language Modeling [100.28366840977966]
  We present RewardBench, a benchmark dataset and code-base for evaluation of reward models. The dataset is a collection of prompt-chosen-rejected trios spanning chat, reasoning, and safety. On the RewardBench leaderboard, we evaluate reward models trained with a variety of methods.
  arXiv  Detail & Related papers  (2024-03-20T17:49:54Z)
• Induced Model Matching: How Restricted Models Can Help Larger Ones [1.7676816383911753]
  We consider scenarios where a very accurate predictive model using restricted features is available at the time of training of a larger, full-featured, model. How can the restricted model be useful to the full model? We propose an approach for transferring the knowledge of the restricted model to the full model, by aligning the full model's context-restricted performance with that of the restricted model's.
  arXiv  Detail & Related papers  (2024-02-19T20:21:09Z)
• Model-Based RL for Mean-Field Games is not Statistically Harder than Single-Agent RL [57.745700271150454]
  We study the sample complexity of reinforcement learning in Mean-Field Games (MFGs) with model-based function approximation. We introduce the Partial Model-Based Eluder Dimension (P-MBED), a more effective notion to characterize the model class complexity.
  arXiv  Detail & Related papers  (2024-02-08T14:54:47Z)
• Reusing Pretrained Models by Multi-linear Operators for Efficient Training [65.64075958382034]
  Training large models from scratch usually costs a substantial amount of resources. Recent studies such as bert2BERT and LiGO have reused small pretrained models to initialize a large model. We propose a method that linearly correlates each weight of the target model to all the weights of the pretrained model.
  arXiv  Detail & Related papers  (2023-10-16T06:16:47Z)
• fairml: A Statistician's Take on Fair Machine Learning Modelling [0.0]
  We describe the fairml package which implements our previous work (Scutari, Panero, and Proissl 2022) and related models in the literature. fairml is designed around classical statistical models and penalised regression results. The constraint used to enforce fairness is to model estimation, making it possible to mix-and-match the desired model family and fairness definition for each application.
  arXiv  Detail & Related papers  (2023-05-03T09:59:53Z)
• Rethinking Log Odds: Linear Probability Modelling and Expert Advice in Interpretable Machine Learning [8.831954614241234]
  We introduce a family of interpretable machine learning models, with two broad additions: Linearised Additive Models (LAMs) and SubscaleHedge. LAMs replace the ubiquitous logistic link function in General Additive Models (GAMs); and SubscaleHedge is an expert advice algorithm for combining base models trained on subsets of features called subscales.
  arXiv  Detail & Related papers  (2022-11-11T17:21:57Z)
• Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow [14.422129911404472]
  Bellman aims to fill this gap and introduces the first thoroughly designed and tested model-based RL toolbox. Our modular approach enables to combine a wide range of environment models with generic model-based agent classes that recover state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2021-03-26T11:32:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.