Paradigms of AI Evaluation: Mapping Goals, Methodologies and Culture

• URL: http://arxiv.org/abs/2502.15620v1
• Date: Fri, 21 Feb 2025 17:44:05 GMT
• Title: Paradigms of AI Evaluation: Mapping Goals, Methodologies and Culture
• Abstract summary: We survey recent work in the AI evaluation landscape and identify six main paradigms.<n>We aim to increase awareness of the breadth of current evaluation approaches and foster cross-pollination between different paradigms.
• Score: 16.361352880545073
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Research in AI evaluation has grown increasingly complex and multidisciplinary, attracting researchers with diverse backgrounds and objectives. As a result, divergent evaluation paradigms have emerged, often developing in isolation, adopting conflicting terminologies, and overlooking each other's contributions. This fragmentation has led to insular research trajectories and communication barriers both among different paradigms and with the general public, contributing to unmet expectations for deployed AI systems. To help bridge this insularity, in this paper we survey recent work in the AI evaluation landscape and identify six main paradigms. We characterise major recent contributions within each paradigm across key dimensions related to their goals, methodologies and research cultures. By clarifying the unique combination of questions and approaches associated with each paradigm, we aim to increase awareness of the breadth of current evaluation approaches and foster cross-pollination between different paradigms. We also identify potential gaps in the field to inspire future research directions.

Related papers

• Monocular Depth Estimation from a Single Image: Progress and Opportunities [51.389814411876294]
  This survey traces the field's evolution from early learning-based methods to the emergence of transformative foundation models.<n>We highlight the integration of depth estimation into applications such as visual SLAM, content generation, and robot perception.
  arXiv  Detail & Related papers  (2026-09-01T12:48:10Z)
• Cross Domain Evaluation of Multimodal Chain-of-Thought Reasoning of different datasets into the Amazon CoT Framework [1.7842332554022695]
  This work presents a comprehensive analysis of Multimodal Chain-of-Thought (Multimodal-CoT) reasoning.<n>We evaluate its effectiveness on the A-OKVQA, OKVQA and ChartQA datasets.<n>Our findings reveal that while vision integration significantly reduces hallucination in rationale generation, the effectiveness of CoT reasoning varies substantially across question types.
  arXiv  Detail & Related papers  (2025-11-24T16:20:02Z)
• A Survey on Video Anomaly Detection via Deep Learning: Human, Vehicle, and Environment [2.3349787245442966]
  Video Anomaly Detection (VAD) has emerged as a pivotal task in computer vision, with broad relevance across multiple fields.<n>Recent advances in deep learning have driven significant progress in this area, yet the field remains fragmented across domains and learning paradigms.<n>This survey offers a comprehensive perspective on VAD, systematically organizing the literature across various supervision levels.
  arXiv  Detail & Related papers  (2025-08-19T18:50:49Z)
• Decoding the Multimodal Maze: A Systematic Review on the Adoption of Explainability in Multimodal Attention-based Models [0.0]
  This systematic literature review analyzes research published between January 2020 and early 2024 that focuses on the explainability of multimodal models.<n>We find that evaluation methods for XAI in multimodal settings are largely non-systematic, lacking consistency, robustness, and consideration for modality-specific cognitive and contextual factors.
  arXiv  Detail & Related papers  (2025-08-06T13:14:20Z)
• A Survey on Generative Model Unlearning: Fundamentals, Taxonomy, Evaluation, and Future Direction [21.966560704390716]
  We review current research on Generative Model Unlearning (GenMU)<n>We propose a unified analytical framework for categorizing unlearning objectives, methodological strategies, and evaluation metrics.<n>We highlight the potential practical value of unlearning techniques in real-world applications.
  arXiv  Detail & Related papers  (2025-07-26T09:49:57Z)
• Anomaly Detection and Generation with Diffusion Models: A Survey [51.61574868316922]
  Anomaly detection (AD) plays a pivotal role across diverse domains, including cybersecurity, finance, healthcare, and industrial manufacturing.<n>Recent advancements in deep learning, specifically diffusion models (DMs), have sparked significant interest.<n>This survey aims to guide researchers and practitioners in leveraging DMs for innovative AD solutions across diverse applications.
  arXiv  Detail & Related papers  (2025-06-11T03:29:18Z)
• Towards deployment-centric multimodal AI beyond vision and language [67.02589156099391]
  We advocate a deployment-centric workflow that incorporates deployment constraints early to reduce the likelihood of undeployable solutions. We identify common multimodal-AI-specific challenges shared across disciplines and examine three real-world use cases. By fostering multidisciplinary dialogue and open research practices, our community can accelerate deployment-centric development for broad societal impact.
  arXiv  Detail & Related papers  (2025-04-04T17:20:05Z)
• Bridging the Gap: Integrating Ethics and Environmental Sustainability in AI Research and Practice [57.94036023167952]
  We argue that the efforts aiming to study AI's ethical ramifications should be made in tandem with those evaluating its impacts on the environment. We propose best practices to better integrate AI ethics and sustainability in AI research and practice.
  arXiv  Detail & Related papers  (2025-04-01T13:53:11Z)
• Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey [124.23247710880008]
  multimodal CoT (MCoT) reasoning has recently garnered significant research attention. Existing MCoT studies design various methodologies to address the challenges of image, video, speech, audio, 3D, and structured data. We present the first systematic survey of MCoT reasoning, elucidating the relevant foundational concepts and definitions.
  arXiv  Detail & Related papers  (2025-03-16T18:39:13Z)
• On Generalization Across Environments In Multi-Objective Reinforcement Learning [6.686583184622338]
  We formalize the concept of generalization in Multi-Objective Reinforcement Learning (MORL) and how it can be evaluated. We contribute a novel benchmark featuring diverse multi-objective domains with parameterized environment configurations. Our baseline evaluations of state-of-the-art MORL algorithms on this benchmark reveals limited generalization capabilities, suggesting significant room for improvement.
  arXiv  Detail & Related papers  (2025-03-02T08:50:14Z)
• Survey on AI-Generated Media Detection: From Non-MLLM to MLLM [51.91311158085973]
  Methods for detecting AI-generated media have evolved rapidly.<n>General-purpose detectors based on MLLMs integrate authenticity verification, explainability, and localization capabilities.<n>Ethical and security considerations have emerged as critical global concerns.
  arXiv  Detail & Related papers  (2025-02-07T12:18:20Z)
• Multimodal Alignment and Fusion: A Survey [11.3029945633295]
  This survey provides a comprehensive overview of advances in multimodal alignment and fusion within the field of machine learning.<n>We systematically categorize and analyze key approaches to alignment and fusion through both structural perspectives.<n>This survey highlights critical challenges such as cross-modal misalignment, computational bottlenecks, data quality issues, and the modality gap.
  arXiv  Detail & Related papers  (2024-11-26T02:10:27Z)
• The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models [28.743404185915697]
  This paper provides a comprehensive overview of recent works on the evaluation of Attitudes, Opinions, Values (AOVs) in Large Language Models (LLMs) By doing so, we address the potential and challenges with respect to understanding the model, human-AI alignment, and downstream application in social sciences.
  arXiv  Detail & Related papers  (2024-06-16T22:59:18Z)
• Towards Bidirectional Human-AI Alignment: A Systematic Review for Clarifications, Framework, and Future Directions [101.67121669727354]
  Recent advancements in AI have highlighted the importance of guiding AI systems towards the intended goals, ethical principles, and values of individuals and groups, a concept broadly recognized as alignment. The lack of clarified definitions and scopes of human-AI alignment poses a significant obstacle, hampering collaborative efforts across research domains to achieve this alignment. We introduce a systematic review of over 400 papers published between 2019 and January 2024, spanning multiple domains such as Human-Computer Interaction (HCI), Natural Language Processing (NLP), Machine Learning (ML)
  arXiv  Detail & Related papers  (2024-06-13T16:03:25Z)
• Understanding the Application of Utility Theory in Robotics and Artificial Intelligence: A Survey [5.168741399695988]
  The utility is a unifying concept in economics, game theory, and operations research, even in the Robotics and AI field. This paper introduces a utility-orient needs paradigm to describe and evaluate inter and outer relationships among agents' interactions.
  arXiv  Detail & Related papers  (2023-06-15T18:55:48Z)
• Empathetic Conversational Systems: A Review of Current Advances, Gaps, and Opportunities [2.741266294612776]
  A growing number of studies have recognized the benefits of empathy and started to incorporate empathy in conversational systems. This paper examines this rapidly growing field using five review dimensions.
  arXiv  Detail & Related papers  (2022-05-09T05:19:48Z)
• Recent Advances in Monocular 2D and 3D Human Pose Estimation: A Deep Learning Perspective [69.44384540002358]
  We provide a comprehensive and holistic 2D-to-3D perspective to tackle this problem. We categorize the mainstream and milestone approaches since the year 2014 under unified frameworks. We also summarize the pose representation styles, benchmarks, evaluation metrics, and the quantitative performance of popular approaches.
  arXiv  Detail & Related papers  (2021-04-23T11:07:07Z)
• Transdisciplinary AI Observatory -- Retrospective Analyses and Future-Oriented Contradistinctions [22.968817032490996]
  This paper motivates the need for an inherently transdisciplinary AI observatory approach. Building on these AI observatory tools, we present near-term transdisciplinary guidelines for AI safety.
  arXiv  Detail & Related papers  (2020-11-26T16:01:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.