An Exploration of Self-Supervised Mutual Information Alignment for Multi-Task Settings

• URL: http://arxiv.org/abs/2410.01704v1
• Date: Wed, 2 Oct 2024 16:15:04 GMT
• Title: An Exploration of Self-Supervised Mutual Information Alignment for Multi-Task Settings
• Authors: Soham Govande,
• Abstract summary: Self-Supervised Alignment with Mutual Information (SAMI) uses conditional mutual information to encourage the connection between behavioral preferences and model responses. We conduct two experiments exploring SAMI in multi-task settings. One iteration of SAMI has a 57% win rate against DPO, with significant variation in performance between task categories.
• Score: 0.0
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: There is a growing need for pluralistic alignment methods that can steer language models towards individual attributes and preferences. One such method, Self-Supervised Alignment with Mutual Information (SAMI), uses conditional mutual information to encourage the connection between behavioral preferences and model responses. We conduct two experiments exploring SAMI in multi-task settings. First, we compare SAMI to Direct Preference Optimization (DPO) on a multi-task benchmark (MT-Bench), using a stronger model to generate training data for a weaker one across diverse categories (humanities, STEM, extraction, coding, math, reasoning, and roleplay). Our results indicate that one iteration of SAMI has a 57% win rate against DPO, with significant variation in performance between task categories. Second, we examine SAMI's impact on mathematical accuracy (GSM-8K) relative to supervised fine-tuning (SFT). While SAMI increases zero-shot performance by 1.1%, SFT is more effective with a 3.2% boost. However, SAMI shows interesting scaling trends. When given 10 attempts, SAMI improves accuracy by 3.9%, while SFT achieves a 10.1% increase. Combining SAMI with SFT yields an additional improvement of 1.3% in multi-attempt settings, though single-attempt accuracy remains unchanged.

Related papers

• On Randomness in Agentic Evals [6.177270420667714]
  Agentic systems are evaluated on benchmarks where agents interact with environments to solve tasks.<n>Most papers report a pass@1 score computed from a single run per task, assuming this gives a reliable performance estimate.<n>We find substantial variance: single-run pass@1 estimates vary by 2.2 to 6.0 percentage points depending on which run is selected.
  arXiv  Detail & Related papers  (2026-02-06T19:49:13Z)
• SWE-RM: Execution-free Feedback For Software Engineering Agents [61.86380395896069]
  Execution-based feedback is widely used in the development of coding agents through test-time scaling (TTS) and reinforcement learning (RL)<n>In contrast, execution-free feedback from reward models can provide more fine-grained signals without depending on unit test cases.<n>We introduce SWE-RM, an accurate and robust reward model adopting a mixture-of-experts architecture with 30B total parameters and 3B activated during inference.
  arXiv  Detail & Related papers  (2025-12-26T08:26:18Z)
• CoT-Saliency: Unified Chain-of-Thought Reasoning for Heterogeneous Saliency Tasks [96.64597365827046]
  We present the first unified framework that jointly handles three operationally heterogeneous saliency tasks.<n>We introduce a Chain-of-Thought (CoT) reasoning process in a Vision-Language Model (VLM) to bridge task heterogeneity.<n>We show our model matches or outperforms specialized SOTA methods and strong closed-source VLMs across all tasks.
  arXiv  Detail & Related papers  (2025-11-01T04:37:01Z)
• MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources [113.33902847941941]
  Variance-Aware Sampling (VAS) is a data selection strategy guided by Variance Promotion Score (VPS)<n>We release large-scale, carefully curated resources containing 1.6M long CoT cold-start data and 15k RL QA pairs.<n> Experiments across mathematical reasoning benchmarks demonstrate the effectiveness of both the curated data and the proposed VAS.
  arXiv  Detail & Related papers  (2025-09-25T14:58:29Z)
• FedAPM: Federated Learning via ADMM with Partial Model Personalization [12.72555825043549]
  In federated learning (FL), the assumption that datasets from different devices are independent and identically distributed (i.i.d.) often does not hold due to user differences.<n>We propose an FL framework based on the alternating direction method of multipliers (ADMM), referred to as FedAPM, to mitigate client drift.
  arXiv  Detail & Related papers  (2025-06-05T06:38:29Z)
• Assessing Robustness to Spurious Correlations in Post-Training Language Models [0.9071144333827891]
  We evaluate three post-training algorithms --Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and KTO -- across a diverse set of synthetic tasks and spuriousness conditions.<n>Our results show that the models often but not always degrade under higher spuriousness.
  arXiv  Detail & Related papers  (2025-05-09T00:39:43Z)
• Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning [59.11519451499754]
  Direct Preference Optimization (DPO) has emerged as a de-facto approach for aligning language models with human preferences. Recent work has shown DPO's effectiveness relies on training data quality. We discover that reference model probability space naturally detects high-quality training samples.
  arXiv  Detail & Related papers  (2025-01-25T07:21:50Z)
• A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques [63.10251271444959]
  Large language models are first pre-trained on trillions of tokens and then instruction-tuned or aligned to specific preferences. We conduct an in-depth investigation of the impact of popular choices for three crucial axes. Our setup spanning over 300 experiments reveals consistent trends and unexpected findings.
  arXiv  Detail & Related papers  (2024-06-07T12:25:51Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• Triple Preference Optimization: Achieving Better Alignment with Less Data in a Single Step Optimization [35.36615140853107]
  Triple Preference Optimization (TPO) is designed to align large language models with three preferences without requiring a separate Supervised Fine-Tuned (SFT) model. We show that TPO achieves superior results compared to models aligned through other methods such as SFT, DPO, KTO, IPO, CPO, and ORPO.
  arXiv  Detail & Related papers  (2024-05-26T20:18:11Z)
• Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process [26.196705232699884]
  We introduce Intuitive Fine-Tuning (IFT) to integrate SFT and Preference Optimization into a single process. IFT performs comparably or even superiorly to sequential recipes of SFT and some typical Preference Optimization methods. An explainable Frozen Lake game further validates the effectiveness of IFT for getting competitive policy.
  arXiv  Detail & Related papers  (2024-05-20T08:23:28Z)
• Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks [35.36615140853107]
  This study evaluates Direct Preference Optimization (DPO) and its variants for aligning Large Language Models (LLMs) with human preferences. Our evaluation spans 13 benchmarks covering dialogue, reasoning, mathematical problem-solving, question answering, truthfulness, MT-Bench, Big Bench, and the Open LLM Leaderboard. We find that alignment methods often achieve near optimal performance even with smaller subsets of training data.
  arXiv  Detail & Related papers  (2024-04-23T03:55:01Z)
• FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Driving [20.037562671813]
  We present FusionAD, the first unified framework that fuse the information from most critical sensors, camera and LiDAR, goes beyond perception task. In constrast to camera-based end-to-end UniAD, we establish a method fusion aided modality-aware prediction status planning modules, dubbed FMS. We conduct extensive experiments on commonly used benchmark nu's dataset, our advantages state-of-the-art performance and surpassing baselines on average 15% on perception tasks like detection and tracking, 10% on occupancy prediction accuracy, reducing prediction error from 0.708 to 0.389, and reducing collision rate from 0.31%
  arXiv  Detail & Related papers  (2023-08-02T08:29:44Z)
• An Empirical Study of Multimodal Model Merging [148.48412442848795]
  Model merging is a technique that fuses multiple models trained on different tasks to generate a multi-task solution. We conduct our study for a novel goal where we can merge vision, language, and cross-modal transformers of a modality-specific architecture. We propose two metrics that assess the distance between weights to be merged and can serve as an indicator of the merging outcomes.
  arXiv  Detail & Related papers  (2023-04-28T15:43:21Z)
• Prompt-MIL: Boosting Multi-Instance Learning Schemes via Task-specific Prompt Tuning [31.0183821423397]
  Whole slide image (WSI) classification is a critical task in computational pathology. Current state of the art methods are based on multi-instance learning schemes (MIL), which usually rely on pretrained features to represent the instances. We propose Prompt-MIL, an MIL framework that integrates prompts into WSI classification.
  arXiv  Detail & Related papers  (2023-03-21T22:24:27Z)
• Adapted Multimodal BERT with Layer-wise Fusion for Sentiment Analysis [84.12658971655253]
  We propose Adapted Multimodal BERT, a BERT-based architecture for multimodal tasks. adapter adjusts the pretrained language model for the task at hand, while the fusion layers perform task-specific, layer-wise fusion of audio-visual information with textual BERT representations. In our ablations we see that this approach leads to efficient models, that can outperform their fine-tuned counterparts and are robust to input noise.
  arXiv  Detail & Related papers  (2022-12-01T17:31:42Z)
• PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptation [89.0074567748505]
  We propose a new metric to accurately predict the prompt transferability (regarding (i)), and a novel PoT approach (namely PANDA) Our proposed metric works well to predict the prompt transferability; 2) our PANDA consistently outperforms the vanilla PoT approach by 2.3% average score (up to 24.1%) among all tasks and model sizes; 3) with our PANDA approach, prompt-tuning can achieve competitive and even better performance than model-tuning in various PLM scales scenarios.
  arXiv  Detail & Related papers  (2022-08-22T09:14:14Z)
• Mutual-Information Based Few-Shot Classification [34.95314059362982]
  We introduce Transductive Infomation Maximization (TIM) for few-shot learning. Our method maximizes the mutual information between the query features and their label predictions for a given few-shot task. We propose a new alternating-direction solver, which speeds up transductive inference over gradient-based optimization.
  arXiv  Detail & Related papers  (2021-06-23T09:17:23Z)
• ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning [91.13797346047984]
  We introduce ADAHESSIAN, a second order optimization algorithm which dynamically incorporates the curvature of the loss function via ADAptive estimates. We show that ADAHESSIAN achieves new state-of-the-art results by a large margin as compared to other adaptive optimization methods.
  arXiv  Detail & Related papers  (2020-06-01T05:00:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.