ACCon: Angle-Compensated Contrastive Regularizer for Deep Regression

• URL: http://arxiv.org/abs/2501.07045v1
• Date: Mon, 13 Jan 2025 03:55:59 GMT
• Title: ACCon: Angle-Compensated Contrastive Regularizer for Deep Regression
• Authors: Botao Zhao, Xiaoyang Qu, Zuheng Kang, Junqing Peng, Jing Xiao, Jianzong Wang,
• Abstract summary: In deep regression, capturing the relationship among continuous labels in feature space is a fundamental challenge that has attracted increasing interest. Existing approaches often rely on order-aware representation learning or distance-based weighting. We propose an angle-compensated contrastive regularizer for deep regression, which adjusts the cosine distance between anchor and negative samples.
• Score: 28.491074229136014
• License:
• Abstract: In deep regression, capturing the relationship among continuous labels in feature space is a fundamental challenge that has attracted increasing interest. Addressing this issue can prevent models from converging to suboptimal solutions across various regression tasks, leading to improved performance, especially for imbalanced regression and under limited sample sizes. However, existing approaches often rely on order-aware representation learning or distance-based weighting. In this paper, we hypothesize a linear negative correlation between label distances and representation similarities in regression tasks. To implement this, we propose an angle-compensated contrastive regularizer for deep regression, which adjusts the cosine distance between anchor and negative samples within the contrastive learning framework. Our method offers a plug-and-play compatible solution that extends most existing contrastive learning methods for regression tasks. Extensive experiments and theoretical analysis demonstrate that our proposed angle-compensated contrastive regularizer not only achieves competitive regression performance but also excels in data efficiency and effectiveness on imbalanced datasets.

Related papers

• Distributed High-Dimensional Quantile Regression: Estimation Efficiency and Support Recovery [0.0]
  We focus on distributed estimation and support recovery for high-dimensional linear quantile regression. We transform the original quantile regression into the least-squares optimization. An efficient algorithm is developed, which enjoys high computation and communication efficiency.
  arXiv  Detail & Related papers  (2024-05-13T08:32:22Z)
• Gradient Aligned Regression via Pairwise Losses [40.676226700035585]
  Gradient Aligned Regression (GAR) is a competitive alternative method in label space. Running time experiments demonstrate the superior efficiency of the proposed GAR over existing methods. We limit our current scope as regression on the clean data setting without noises, outliers or distributional shifts.
  arXiv  Detail & Related papers  (2024-02-08T23:43:53Z)
• Deep Generative Symbolic Regression [83.04219479605801]
  Symbolic regression aims to discover concise closed-form mathematical equations from data. Existing methods, ranging from search to reinforcement learning, fail to scale with the number of input variables. We propose an instantiation of our framework, Deep Generative Symbolic Regression.
  arXiv  Detail & Related papers  (2023-12-30T17:05:31Z)
• Mixup Your Own Pairs [22.882694278940598]
  We argue that the potential of contrastive learning for regression has been overshadowed due to the neglect of two crucial aspects: ordinality-awareness and hardness. Specifically, we propose Supervised Contrastive Learning for Regression with Mixup (SupReMix) It takes anchor-inclusive mixtures (mixup of the anchor and a distinct negative sample) as hard negative pairs and anchor-exclusive mixtures (mixup of two distinct negative samples) as hard positive pairs at the embedding level.
  arXiv  Detail & Related papers  (2023-09-28T17:38:59Z)
• Linked shrinkage to improve estimation of interaction effects in regression models [0.0]
  We develop an estimator that adapts well to two-way interaction terms in a regression model. We evaluate the potential of the model for inference, which is notoriously hard for selection strategies. Our models can be very competitive to a more advanced machine learner, like random forest, even for fairly large sample sizes.
  arXiv  Detail & Related papers  (2023-09-25T10:03:39Z)
• Errors-in-variables Fr\'echet Regression with Low-rank Covariate Approximation [2.1756081703276]
  Fr'echet regression has emerged as a promising approach for regression analysis involving non-Euclidean response variables. Our proposed framework combines the concepts of global Fr'echet regression and principal component regression, aiming to improve the efficiency and accuracy of the regression estimator.
  arXiv  Detail & Related papers  (2023-05-16T08:37:54Z)
• Vector-Valued Least-Squares Regression under Output Regularity Assumptions [73.99064151691597]
  We propose and analyse a reduced-rank method for solving least-squares regression problems with infinite dimensional output. We derive learning bounds for our method, and study under which setting statistical performance is improved in comparison to full-rank method.
  arXiv  Detail & Related papers  (2022-11-16T15:07:00Z)
• Poseur: Direct Human Pose Regression with Transformers [119.79232258661995]
  We propose a direct, regression-based approach to 2D human pose estimation from single images. Our framework is end-to-end differentiable, and naturally learns to exploit the dependencies between keypoints. Ours is the first regression-based approach to perform favorably compared to the best heatmap-based pose estimation methods.
  arXiv  Detail & Related papers  (2022-01-19T04:31:57Z)
• Revisiting Consistency Regularization for Semi-Supervised Learning [80.28461584135967]
  We propose an improved consistency regularization framework by a simple yet effective technique, FeatDistLoss. Experimental results show that our model defines a new state of the art for various datasets and settings.
  arXiv  Detail & Related papers  (2021-12-10T20:46:13Z)
• Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates [68.09049111171862]
  This work focuses on quantifying, reducing and analyzing regression errors in the NLP model updates. We formulate the regression-free model updates into a constrained optimization problem. We empirically analyze how model ensemble reduces regression.
  arXiv  Detail & Related papers  (2021-05-07T03:33:00Z)
• A Hypergradient Approach to Robust Regression without Correspondence [85.49775273716503]
  We consider a variant of regression problem, where the correspondence between input and output data is not available. Most existing methods are only applicable when the sample size is small. We propose a new computational framework -- ROBOT -- for the shuffled regression problem.
  arXiv  Detail & Related papers  (2020-11-30T21:47:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.