An Interpretable Graph-based Mapping of Trustworthy Machine Learning
Research
- URL: http://arxiv.org/abs/2105.06591v1
- Date: Thu, 13 May 2021 23:25:07 GMT
- Title: An Interpretable Graph-based Mapping of Trustworthy Machine Learning
Research
- Authors: Noemi Derzsy, Subhabrata Majumdar, Rajat Malik
- Abstract summary: We build a co-occurrence network of words using a web-scraped corpus of more than 7,000 peer-reviewed recent ML papers.
We use community detection to obtain semantic clusters of words in this network that can infer relative positions of TwML topics.
- Score: 3.222802562733787
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: There is an increasing interest in ensuring machine learning (ML) frameworks
behave in a socially responsible manner and are deemed trustworthy. Although
considerable progress has been made in the field of Trustworthy ML (TwML) in
the recent past, much of the current characterization of this progress is
qualitative. Consequently, decisions about how to address issues of
trustworthiness and future research goals are often left to the interested
researcher. In this paper, we present the first quantitative approach to
characterize the comprehension of TwML research. We build a co-occurrence
network of words using a web-scraped corpus of more than 7,000 peer-reviewed
recent ML papers -- consisting of papers both related and unrelated to TwML. We
use community detection to obtain semantic clusters of words in this network
that can infer relative positions of TwML topics. We propose an innovative
fingerprinting algorithm to obtain probabilistic similarity scores for
individual words, then combine them to give a paper-level relevance score. The
outcomes of our analysis inform a number of interesting insights on advancing
the field of TwML research.
Related papers
- Retrieval-Enhanced Machine Learning: Synthesis and Opportunities [60.34182805429511]
Retrieval-enhancement can be extended to a broader spectrum of machine learning (ML)
This work introduces a formal framework of this paradigm, Retrieval-Enhanced Machine Learning (REML), by synthesizing the literature in various domains in ML with consistent notations which is missing from the current literature.
The goal of this work is to equip researchers across various disciplines with a comprehensive, formally structured framework of retrieval-enhanced models, thereby fostering interdisciplinary future research.
arXiv Detail & Related papers (2024-07-17T20:01:21Z) - MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs [38.93090238335506]
Spurious bias, a tendency to use spurious correlations between non-essential input attributes and target variables for predictions, has revealed a severe pitfall in deep learning models trained on single modality data.
We introduce MM-SpuBench, a comprehensive visual question-answering (VQA) benchmark designed to evaluate MLLMs' reliance on nine distinct categories of spurious correlations.
Our findings illuminate the persistence of the reliance on spurious correlations from these models and underscore the urge for new methodologies to mitigate spurious biases.
arXiv Detail & Related papers (2024-06-24T20:29:16Z) - Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph [85.51252685938564]
Uncertainty quantification (UQ) is becoming increasingly recognized as a critical component of applications that rely on machine learning (ML)
As with other ML models, large language models (LLMs) are prone to make incorrect predictions, hallucinate'' by fabricating claims, or simply generate low-quality output for a given input.
We introduce a novel benchmark that implements a collection of state-of-the-art UQ baselines, and provides an environment for controllable and consistent evaluation of novel techniques.
arXiv Detail & Related papers (2024-06-21T20:06:31Z) - RelevAI-Reviewer: A Benchmark on AI Reviewers for Survey Paper Relevance [0.8089605035945486]
We propose RelevAI-Reviewer, an automatic system that conceptualizes the task of survey paper review as a classification problem.
We introduce a novel dataset comprised of 25,164 instances. Each instance contains one prompt and four candidate papers, each varying in relevance to the prompt.
We develop a machine learning (ML) model capable of determining the relevance of each paper and identifying the most pertinent one.
arXiv Detail & Related papers (2024-06-13T06:42:32Z) - Measuring Distributional Shifts in Text: The Advantage of Language
Model-Based Embeddings [11.393822909537796]
An essential part of monitoring machine learning models in production is measuring input and output data drift.
Recent advancements in large language models (LLMs) indicate their effectiveness in capturing semantic relationships.
We propose a clustering-based algorithm for measuring distributional shifts in text data by exploiting such embeddings.
arXiv Detail & Related papers (2023-12-04T20:46:48Z) - SEED-Bench-2: Benchmarking Multimodal Large Language Models [67.28089415198338]
Multimodal large language models (MLLMs) have recently demonstrated exceptional capabilities in generating not only texts but also images given interleaved multimodal inputs.
SEED-Bench-2 comprises 24K multiple-choice questions with accurate human annotations, which spans 27 dimensions.
We evaluate the performance of 23 prominent open-source MLLMs and summarize valuable observations.
arXiv Detail & Related papers (2023-11-28T05:53:55Z) - Controllable Multi-document Summarization: Coverage & Coherence
Intuitive Policy with Large Language Model Based Rewards [42.171703872560286]
Controllability is a matter of concern when it comes to text generation tasks with long inputs, such as multi-document summarization.
We train a controllable content extraction scheme to extract the text that will be refined by an LLM.
Our approach yields competitive results in the evaluation using ROUGE metrics and outperforms potential baselines in coherence.
arXiv Detail & Related papers (2023-10-05T11:29:09Z) - A Survey on Multimodal Large Language Models [71.63375558033364]
Multimodal Large Language Model (MLLM) represented by GPT-4V has been a new rising research hotspot.
This paper aims to trace and summarize the recent progress of MLLMs.
arXiv Detail & Related papers (2023-06-23T15:21:52Z) - Interpretability of Machine Learning: Recent Advances and Future
Prospects [21.68362950922772]
The proliferation of machine learning (ML) has drawn unprecedented interest in the study of various multimedia contents.
The black-box nature of contemporary ML, especially in deep neural networks (DNNs), has posed a primary challenge for ML-based representation learning.
This paper presents a survey on recent advances and future prospects on interpretability of ML.
arXiv Detail & Related papers (2023-04-30T17:31:29Z) - Investigating Fairness Disparities in Peer Review: A Language Model
Enhanced Approach [77.61131357420201]
We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs)
We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date.
We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
arXiv Detail & Related papers (2022-11-07T16:19:42Z) - Memory-Based Optimization Methods for Model-Agnostic Meta-Learning and
Personalized Federated Learning [56.17603785248675]
Model-agnostic meta-learning (MAML) has become a popular research area.
Existing MAML algorithms rely on the episode' idea by sampling a few tasks and data points to update the meta-model at each iteration.
This paper proposes memory-based algorithms for MAML that converge with vanishing error.
arXiv Detail & Related papers (2021-06-09T08:47:58Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.