CoCoA-Mix: Confusion-and-Confidence-Aware Mixture Model for Context Optimization

• URL: http://arxiv.org/abs/2506.07484v1
• Date: Mon, 09 Jun 2025 07:04:47 GMT
• Title: CoCoA-Mix: Confusion-and-Confidence-Aware Mixture Model for Context Optimization
• Authors: Dasol Hong, Wooju Lee, Hyun Myung,
• Abstract summary: We propose a confusion-aware loss (CoA-loss) that improves specialization by refining the decision boundaries between confusing classes.<n>We mathematically demonstrate that a mixture model can enhance generalization without compromising specialization.<n>CoCoA-Mix, a mixture model with CoA-loss and CoA-weights, outperforms state-of-the-art methods by enhancing specialization and generalization.
• Score: 9.888839721140231
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompt tuning, which adapts vision-language models by freezing model parameters and optimizing only the prompt, has proven effective for task-specific adaptations. The core challenge in prompt tuning is improving specialization for a specific task and generalization for unseen domains. However, frozen encoders often produce misaligned features, leading to confusion between classes and limiting specialization. To overcome this issue, we propose a confusion-aware loss (CoA-loss) that improves specialization by refining the decision boundaries between confusing classes. Additionally, we mathematically demonstrate that a mixture model can enhance generalization without compromising specialization. This is achieved using confidence-aware weights (CoA-weights), which adjust the weights of each prediction in the mixture model based on its confidence within the class domains. Extensive experiments show that CoCoA-Mix, a mixture model with CoA-loss and CoA-weights, outperforms state-of-the-art methods by enhancing specialization and generalization. Our code is publicly available at https://github.com/url-kaist/CoCoA-Mix.

Related papers

• ConCM: Consistency-Driven Calibration and Matching for Few-Shot Class-Incremental Learning [30.915755767447422]
  Few-Shot Class-Incremental Learning (FSCIL) requires models to adapt to novel classes with limited supervision while preserving learned knowledge.<n>We propose a Consistency-driven geometric and Matching Framework (ConCM) that systematically mitigates the knowledge conflict inherent in FSCIL.
  arXiv  Detail & Related papers  (2025-06-24T12:12:50Z)
• CoCoAFusE: Beyond Mixtures of Experts via Model Fusion [3.501882879116058]
  CoCoAFusE builds on the philosophy behind Mixtures of Experts (MoEs)<n>Our formulation extends that of a classical Mixture of Experts by contemplating the fusion of the experts' distributions.<n>This new approach is showcased extensively on a suite of motivating numerical examples and a collection of real-data ones.
  arXiv  Detail & Related papers  (2025-05-02T08:35:04Z)
• Semi-supervised Semantic Segmentation with Multi-Constraint Consistency Learning [81.02648336552421]
  We propose a Multi-Constraint Consistency Learning approach to facilitate the staged enhancement of the encoder and decoder.<n>Self-adaptive feature masking and noise injection are designed in an instance-specific manner to perturb the features for robust learning of the decoder.<n> Experimental results on Pascal VOC2012 and Cityscapes datasets demonstrate that our proposed MCCL achieves new state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-03-23T03:21:33Z)
• CNC: Cross-modal Normality Constraint for Unsupervised Multi-class Anomaly Detection [34.675120608542265]
  We propose a novel approach that leverages class-agnostic learnable prompts to guide the decoded features towards a normal textual representation.<n>Our method achieves competitive performance on the MVTec AD and VisA datasets, demonstrating its effectiveness.
  arXiv  Detail & Related papers  (2024-12-31T08:43:44Z)
• Gradient-free variational learning with conditional mixture networks [39.827869318925494]
  We introduce CAVI-CMN, a fast, gradient-free variational method for training conditional mixture networks (CMNs)<n>CAVI-CMN achieves competitive and often superior predictive accuracy compared to maximum likelihood estimation (MLE) with backpropagation.<n>As input size or the number of experts increases, computation time scales competitively with MLE.
  arXiv  Detail & Related papers  (2024-08-29T10:43:55Z)
• Model Inversion Attacks Through Target-Specific Conditional Diffusion Models [54.69008212790426]
  Model inversion attacks (MIAs) aim to reconstruct private images from a target classifier's training set, thereby raising privacy concerns in AI applications. Previous GAN-based MIAs tend to suffer from inferior generative fidelity due to GAN's inherent flaws and biased optimization within latent space. We propose Diffusion-based Model Inversion (Diff-MI) attacks to alleviate these issues.
  arXiv  Detail & Related papers  (2024-07-16T06:38:49Z)
• LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks [52.46420522934253]
  We introduce LoRA-Ensemble, a parameter-efficient ensembling method for self-attention networks.<n>The method not only outperforms state-of-the-art implicit techniques like BatchEnsemble, but even matches or exceeds the accuracy of an Explicit Ensemble.
  arXiv  Detail & Related papers  (2024-05-23T11:10:32Z)
• Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization [51.98792406392873]
  Mixture of Experts (MoE) provides a powerful way to decompose dense layers into smaller, modular computations. A major challenge lies in the computational cost of scaling the number of experts high enough to achieve fine-grained specialization. We propose the Multilinear Mixture of Experts ($mu$MoE) layer to address this, focusing on vision models.
  arXiv  Detail & Related papers  (2024-02-19T21:20:22Z)
• AE-RED: A Hyperspectral Unmixing Framework Powered by Deep Autoencoder and Regularization by Denoising [14.908906329456842]
  We propose a generic unmixing framework to integrate the autoencoder network with regularization by denoising (RED), named AE-RED. Experiment results on both synthetic and real data sets show the superiority of our proposed framework compared with state-of-the-art unmixing approaches.
  arXiv  Detail & Related papers  (2023-07-01T08:20:36Z)
• Precision-Recall Divergence Optimization for Generative Modeling with GANs and Normalizing Flows [54.050498411883495]
  We develop a novel training method for generative models, such as Generative Adversarial Networks and Normalizing Flows. We show that achieving a specified precision-recall trade-off corresponds to minimizing a unique $f$-divergence from a family we call the textitPR-divergences. Our approach improves the performance of existing state-of-the-art models like BigGAN in terms of either precision or recall when tested on datasets such as ImageNet.
  arXiv  Detail & Related papers  (2023-05-30T10:07:17Z)
• ER: Equivariance Regularizer for Knowledge Graph Completion [107.51609402963072]
  We propose a new regularizer, namely, Equivariance Regularizer (ER) ER can enhance the generalization ability of the model by employing the semantic equivariance between the head and tail entities. The experimental results indicate a clear and substantial improvement over the state-of-the-art relation prediction methods.
  arXiv  Detail & Related papers  (2022-06-24T08:18:05Z)
• Cauchy-Schwarz Regularized Autoencoder [68.80569889599434]
  Variational autoencoders (VAE) are a powerful and widely-used class of generative models. We introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs. Our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis.
  arXiv  Detail & Related papers  (2021-01-06T17:36:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.