Loss Functions and Metrics in Deep Learning

• URL: http://arxiv.org/abs/2307.02694v5
• Date: Mon, 14 Apr 2025 00:48:47 GMT
• Title: Loss Functions and Metrics in Deep Learning
• Authors: Juan Terven, Diana M. Cordova-Esparza, Alfonso Ramirez-Pedraza, Edgar A. Chavez-Urbiola, Julio A. Romero-Gonzalez,
• Abstract summary: This paper presents a comprehensive review of loss functions and performance metrics in deep learning.<n>We show how different loss functions and evaluation metrics can be paired to address task-specific challenges.<n>We highlight best practices for selecting or combining losses and metrics based on empirical behaviors and domain constraints.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents a comprehensive review of loss functions and performance metrics in deep learning, highlighting key developments and practical insights across diverse application areas. We begin by outlining fundamental considerations in classic tasks such as regression and classification, then extend our analysis to specialized domains like computer vision and natural language processing including retrieval-augmented generation. In each setting, we systematically examine how different loss functions and evaluation metrics can be paired to address task-specific challenges such as class imbalance, outliers, and sequence-level optimization. Key contributions of this work include: (1) a unified framework for understanding how losses and metrics align with different learning objectives, (2) an in-depth discussion of multi-loss setups that balance competing goals, and (3) new insights into specialized metrics used to evaluate modern applications like retrieval-augmented generation, where faithfulness and context relevance are pivotal. Along the way, we highlight best practices for selecting or combining losses and metrics based on empirical behaviors and domain constraints. Finally, we identify open problems and promising directions, including the automation of loss-function search and the development of robust, interpretable evaluation measures for increasingly complex deep learning tasks. Our review aims to equip researchers and practitioners with clearer guidance in designing effective training pipelines and reliable model assessments for a wide spectrum of real-world applications.

Related papers

• Loss Functions in Deep Learning: A Comprehensive Review [3.8001666556614446]
  Loss functions are at the heart of deep learning, shaping how models learn and perform across diverse tasks. This paper presents a comprehensive review of loss functions, covering fundamental metrics like Mean Squared Error and Cross-Entropy to advanced functions such as Adversarial and Diffusion losses.
  arXiv  Detail & Related papers  (2025-04-05T18:07:20Z)
• Are We Solving a Well-Defined Problem? A Task-Centric Perspective on Recommendation Tasks [46.705107776194616]
  We analyze RecSys task formulations, emphasizing key components such as input-output structures, temporal dynamics, and candidate item selection. We explore the balance between task specificity and model generalizability, highlighting how well-defined task formulations serve as the foundation for robust evaluation and effective solution development.
  arXiv  Detail & Related papers  (2025-03-27T06:10:22Z)
• RESTOR: Knowledge Recovery through Machine Unlearning [71.75834077528305]
  Large language models trained on web-scale corpora can memorize undesirable datapoints. Many machine unlearning methods have been proposed that aim to 'erase' these datapoints from trained models. We propose the RESTOR framework for machine unlearning based on the following dimensions.
  arXiv  Detail & Related papers  (2024-10-31T20:54:35Z)
• Towards Effective Evaluations and Comparisons for LLM Unlearning Methods [97.2995389188179]
  This paper seeks to refine the evaluation of machine unlearning for large language models. It addresses two key challenges -- the robustness of evaluation metrics and the trade-offs between competing goals.
  arXiv  Detail & Related papers  (2024-06-13T14:41:00Z)
• Deep Learning-Based Object Pose Estimation: A Comprehensive Survey [73.74933379151419]
  We discuss the recent advances in deep learning-based object pose estimation. Our survey also covers multiple input data modalities, degrees-of-freedom of output poses, object properties, and downstream tasks.
  arXiv  Detail & Related papers  (2024-05-13T14:44:22Z)
• Fast and Efficient Local Search for Genetic Programming Based Loss Function Learning [12.581217671500887]
  We propose a new meta-learning framework for task and model-agnostic loss function learning via a hybrid search approach. Results show that the learned loss functions bring improved convergence, sample efficiency, and inference performance on tabulated, computer vision, and natural language processing problems.
  arXiv  Detail & Related papers  (2024-03-01T02:20:04Z)
• Loss Functions in the Era of Semantic Segmentation: A Survey and Outlook [11.119967679567587]
  Loss functions are crucial for shaping the development of deep learning-based segmentation algorithms. We provide a novel taxonomy and review of how these loss functions are customized and leveraged in image segmentation. We conclude this review by identifying current challenges and unveiling future research opportunities.
  arXiv  Detail & Related papers  (2023-12-08T22:06:05Z)
• Exploring Federated Unlearning: Analysis, Comparison, and Insights [101.64910079905566]
  federated unlearning enables the selective removal of data from models trained in federated systems. This paper examines existing federated unlearning approaches, examining their algorithmic efficiency, impact on model accuracy, and effectiveness in preserving privacy. We propose the OpenFederatedUnlearning framework, a unified benchmark for evaluating federated unlearning methods.
  arXiv  Detail & Related papers  (2023-10-30T01:34:33Z)
• Evaluating General-Purpose AI with Psychometrics [43.85432514910491]
  We discuss the need for a comprehensive and accurate evaluation of general-purpose AI systems such as large language models. Current evaluation methodology, mostly based on benchmarks of specific tasks, falls short of adequately assessing these versatile AI systems. To tackle these challenges, we suggest transitioning from task-oriented evaluation to construct-oriented evaluation.
  arXiv  Detail & Related papers  (2023-10-25T05:38:38Z)
• Generalization Performance of Transfer Learning: Overparameterized and Underparameterized Regimes [61.22448274621503]
  In real-world applications, tasks often exhibit partial similarity, where certain aspects are similar while others are different or irrelevant. Our study explores various types of transfer learning, encompassing two options for parameter transfer. We provide practical guidelines for determining the number of features in the common and task-specific parts for improved generalization performance.
  arXiv  Detail & Related papers  (2023-06-08T03:08:40Z)
• A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems [128.63953314853327]
  "Lifelong Learning" systems are capable of 1) Continuous Learning, 2) Transfer and Adaptation, and 3) Scalability. We show that this suite of metrics can inform the development of varied and complex Lifelong Learning systems.
  arXiv  Detail & Related papers  (2023-01-18T21:58:54Z)
• A survey and taxonomy of loss functions in machine learning [51.35995529962554]
  We present a comprehensive overview of the most widely used loss functions across key applications, including regression, classification, generative modeling, ranking, and energy-based modeling. We introduce 43 distinct loss functions, structured within an intuitive taxonomy that clarifies their theoretical foundations, properties, and optimal application contexts.
  arXiv  Detail & Related papers  (2023-01-13T14:38:24Z)
• L2Explorer: A Lifelong Reinforcement Learning Assessment Environment [49.40779372040652]
  Reinforcement learning solutions tend to generalize poorly when exposed to new tasks outside of the data distribution they are trained on. We introduce a framework for continual reinforcement-learning development and assessment using Lifelong Learning Explorer (L2Explorer) L2Explorer is a new, Unity-based, first-person 3D exploration environment that can be continuously reconfigured to generate a range of tasks and task variants structured into complex evaluation curricula.
  arXiv  Detail & Related papers  (2022-03-14T19:20:26Z)
• Meta-Learning with Task-Adaptive Loss Function for Few-Shot Learning [50.59295648948287]
  In few-shot learning scenarios, the challenge is to generalize and perform well on new unseen examples. We introduce a new meta-learning framework with a loss function that adapts to each task. Our proposed framework, named Meta-Learning with Task-Adaptive Loss Function (MeTAL), demonstrates the effectiveness and the flexibility across various domains.
  arXiv  Detail & Related papers  (2021-10-08T06:07:21Z)
• AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks [78.27036391638802]
  AutoLoss-Zero is the first framework for searching loss functions from scratch for generic tasks. A loss-rejection protocol and a gradient-equivalence-check strategy are developed so as to improve the search efficiency. Experiments on various computer vision tasks demonstrate that our searched loss functions are on par with or superior to existing loss functions.
  arXiv  Detail & Related papers  (2021-03-25T17:59:09Z)
• Auto Seg-Loss: Searching Metric Surrogates for Semantic Segmentation [56.343646789922545]
  We propose to automate the design of metric-specific loss functions by searching differentiable surrogate losses for each metric. Experiments on PASCAL VOC and Cityscapes demonstrate that the searched surrogate losses outperform the manually designed loss functions consistently.
  arXiv  Detail & Related papers  (2020-10-15T17:59:08Z)
• An analysis on the use of autoencoders for representation learning: fundamentals, learning task case studies, explainability and challenges [11.329636084818778]
  In many machine learning tasks, learning a good representation of the data can be the key to building a well-performant solution. We present a series of learning tasks: data embedding for visualization, image denoising, semantic hashing, detection of abnormal behaviors and instance generation. A solution is proposed for each task employing autoencoders as the only learning method.
  arXiv  Detail & Related papers  (2020-05-21T08:41:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.