Artificial Interrogation for Attributing Language Models

• URL: http://arxiv.org/abs/2211.10877v1
• Date: Sun, 20 Nov 2022 05:46:29 GMT
• Title: Artificial Interrogation for Attributing Language Models
• Authors: Farhan Dhanani, Muhammad Rafi
• Abstract summary: The challenge provides twelve open-sourced base versions of popular language models and twelve fine-tuned language models for text generation. The goal of the contest is to identify which fine-tuned models originated from which base model. We have employed four distinct approaches for measuring the resemblance between the responses generated from the models of both sets.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents solutions to the Machine Learning Model Attribution challenge (MLMAC) collectively organized by MITRE, Microsoft, Schmidt-Futures, Robust-Intelligence, Lincoln-Network, and Huggingface community. The challenge provides twelve open-sourced base versions of popular language models developed by well-known organizations and twelve fine-tuned language models for text generation. The names and architecture details of fine-tuned models were kept hidden, and participants can access these models only through the rest APIs developed by the organizers. Given these constraints, the goal of the contest is to identify which fine-tuned models originated from which base model. To solve this challenge, we have assumed that fine-tuned models and their corresponding base versions must share a similar vocabulary set with a matching syntactical writing style that resonates in their generated outputs. Our strategy is to develop a set of queries to interrogate base and fine-tuned models. And then perform one-to-many pairing between them based on similarities in their generated responses, where more than one fine-tuned model can pair with a base model but not vice-versa. We have employed four distinct approaches for measuring the resemblance between the responses generated from the models of both sets. The first approach uses evaluation metrics of the machine translation, and the second uses a vector space model. The third approach uses state-of-the-art multi-class text classification, Transformer models. Lastly, the fourth approach uses a set of Transformer based binary text classifiers, one for each provided base model, to perform multi-class text classification in a one-vs-all fashion. This paper reports implementation details, comparison, and experimental studies, of these approaches along with the final obtained results.

Related papers

• Merging Text Transformer Models from Different Initializations [7.768975909119287]
  We investigate the extent to which separate Transformer minima learn similar features. We propose a model merging technique to investigate the relationship between these minima in the loss landscape. Our results show that the minima of these models are less sharp and isolated than previously understood.
  arXiv  Detail & Related papers  (2024-03-01T21:16:29Z)
• The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute [66.84421705029624]
  We introduce an experimental protocol that enables model comparisons based on equivalent compute, measured in accelerator hours. We pre-process an existing large, diverse, and high-quality dataset of books that surpasses existing academic benchmarks in quality, diversity, and document length. This work also provides two baseline models: a feed-forward model derived from the GPT-2 architecture and a recurrent model in the form of a novel LSTM with ten-fold throughput.
  arXiv  Detail & Related papers  (2023-09-20T10:31:17Z)
• What is the best recipe for character-level encoder-only modelling? [2.792030485253753]
  This paper aims to benchmark recent progress in language understanding models that output contextualised representations at the character level. We find that our best performing character-level model exceeds the performance of a token-based model trained with the same settings on the same data. We believe our results demonstrate the readiness of character-level models for multilingual language representation, and encourage NLP practitioners to try them as drop-in replacements for token-based models.
  arXiv  Detail & Related papers  (2023-05-09T14:00:15Z)
• Machine Learning Model Attribution Challenge [2.6532805035238747]
  Fine-tuned machine learning models may derive from other trained models without obvious attribution characteristics. In this challenge, participants identify the publicly-available base models that underlie a set of anonymous, fine-tuned large language models.
  arXiv  Detail & Related papers  (2023-02-13T22:05:27Z)
• Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
  Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. This creates a barrier to fusing knowledge across individual models to yield a better single model. We propose a dataless knowledge fusion method that merges models in their parameter space.
  arXiv  Detail & Related papers  (2022-12-19T20:46:43Z)
• Language Model Cascades [72.18809575261498]
  Repeated interactions at test-time with a single model, or the composition of multiple models together, further expands capabilities. Cases with control flow and dynamic structure require techniques from probabilistic programming. We formalize several existing techniques from this perspective, including scratchpads / chain of thought, verifiers, STaR, selection-inference, and tool use.
  arXiv  Detail & Related papers  (2022-07-21T07:35:18Z)
• DIRECTOR: Generator-Classifiers For Supervised Language Modeling [27.86870968048833]
  Current language models achieve low perplexity but their resulting generations still suffer from toxic responses, repetitiveness and contradictions. We introduce a new architecture, sc Director, that consists of a unified generator-classifier with both a language modeling and a classification head for each output token.
  arXiv  Detail & Related papers  (2022-06-15T17:44:08Z)
• Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training [86.91380874390778]
  We present Generation-Augmented Pre-training (GAP), that jointly learns representations of natural language utterances and table schemas by leveraging generation models to generate pre-train data. Based on experimental results, neural semantics that leverage GAP MODEL obtain new state-of-the-art results on both SPIDER and CRITERIA-TO-generative benchmarks.
  arXiv  Detail & Related papers  (2020-12-18T15:53:50Z)
• Introducing various Semantic Models for Amharic: Experimentation and Evaluation with multiple Tasks and Datasets [19.855120632909124]
  We introduce different semantic models for Amharic. Models are build using word2Vec embeddings, distributional thesaurus (DT), contextual embeddings, and DT embeddings. We find that newly trained models perform better than pre-trained multilingual models.
  arXiv  Detail & Related papers  (2020-11-02T17:48:25Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)
• Grounded Compositional Outputs for Adaptive Language Modeling [59.02706635250856]
  A language model's vocabulary$-$typically selected before training and permanently fixed later$-$affects its size. We propose a fully compositional output embedding layer for language models. To our knowledge, the result is the first word-level language model with a size that does not depend on the training vocabulary.
  arXiv  Detail & Related papers  (2020-09-24T07:21:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.