Towards Scalable Automated Alignment of LLMs: A Survey

• URL: http://arxiv.org/abs/2406.01252v3
• Date: Tue, 3 Sep 2024 07:07:59 GMT
• Title: Towards Scalable Automated Alignment of LLMs: A Survey
• Authors: Boxi Cao, Keming Lu, Xinyu Lu, Jiawei Chen, Mengjie Ren, Hao Xiang, Peilin Liu, Yaojie Lu, Ben He, Xianpei Han, Le Sun, Hongyu Lin, Bowen Yu,
• Abstract summary: This paper systematically reviews the recently emerging methods of automated alignment. We categorize existing automated alignment methods into 4 major categories based on the sources of alignment signals. We discuss the essential factors that make automated alignment technologies feasible and effective from the fundamental role of alignment.
• Score: 54.820256625544225
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Alignment is the most critical step in building large language models (LLMs) that meet human needs. With the rapid development of LLMs gradually surpassing human capabilities, traditional alignment methods based on human-annotation are increasingly unable to meet the scalability demands. Therefore, there is an urgent need to explore new sources of automated alignment signals and technical approaches. In this paper, we systematically review the recently emerging methods of automated alignment, attempting to explore how to achieve effective, scalable, automated alignment once the capabilities of LLMs exceed those of humans. Specifically, we categorize existing automated alignment methods into 4 major categories based on the sources of alignment signals and discuss the current status and potential development of each category. Additionally, we explore the underlying mechanisms that enable automated alignment and discuss the essential factors that make automated alignment technologies feasible and effective from the fundamental role of alignment.

Related papers

• Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models [54.381650481255235]
  We introduce a new tuning-free approach for self-alignment, Dynamic Rewarding with Prompt Optimization (O) Our approach leverages a search-based optimization framework that allows LLMs to iteratively self-improve and craft the optimal alignment instructions. Empirical evaluations on eight recent LLMs, both open and closed-sourced, demonstrate that DRPO significantly enhances alignment performance.
  arXiv  Detail & Related papers  (2024-11-13T16:15:38Z)
• Position: A Call to Action for a Human-Centered AutoML Paradigm [83.78883610871867]
  Automated machine learning (AutoML) was formed around the fundamental objectives of automatically and efficiently configuring machine learning (ML) We argue that a key to unlocking AutoML's full potential lies in addressing the currently underexplored aspect of user interaction with AutoML systems.
  arXiv  Detail & Related papers  (2024-06-05T15:05:24Z)
• Human-Centered Automation [0.3626013617212666]
  The paper argues for the emerging area of Human-Centered Automation (HCA), which prioritizes user needs and preferences in the design and development of automation systems. The paper discusses the limitations of existing automation approaches, the challenges in integrating AI and RPA, and the benefits of human-centered automation for productivity, innovation, and democratizing access to these technologies.
  arXiv  Detail & Related papers  (2024-05-24T22:12:28Z)
• Combatting Human Trafficking in the Cyberspace: A Natural Language Processing-Based Methodology to Analyze the Language in Online Advertisements [55.2480439325792]
  This project tackles the pressing issue of human trafficking in online C2C marketplaces through advanced Natural Language Processing (NLP) techniques. We introduce a novel methodology for generating pseudo-labeled datasets with minimal supervision, serving as a rich resource for training state-of-the-art NLP models. A key contribution is the implementation of an interpretability framework using Integrated Gradients, providing explainable insights crucial for law enforcement.
  arXiv  Detail & Related papers  (2023-11-22T02:45:01Z)
• Prompts Matter: Insights and Strategies for Prompt Engineering in Automated Software Traceability [45.235173351109374]
  Large Language Models (LLMs) have the potential to revolutionize automated traceability. This paper explores the process of prompt engineering to extract link predictions from an LLM.
  arXiv  Detail & Related papers  (2023-08-01T01:56:22Z)
• Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision [84.31474052176343]
  Recent AI-assistant agents, such as ChatGPT, rely on supervised fine-tuning (SFT) with human annotations and reinforcement learning from human feedback to align the output with human intentions. This dependence can significantly constrain the true potential of AI-assistant agents due to the high cost of obtaining human supervision. We propose a novel approach called SELF-ALIGN, which combines principle-driven reasoning and the generative power of LLMs for the self-alignment of AI agents with minimal human supervision.
  arXiv  Detail & Related papers  (2023-05-04T17:59:28Z)
• OmniForce: On Human-Centered, Large Model Empowered and Cloud-Edge Collaborative AutoML System [85.8338446357469]
  We introduce OmniForce, a human-centered AutoML system that yields both human-assisted ML and ML-assisted human techniques. We show how OmniForce can put an AutoML system into practice and build adaptive AI in open-environment scenarios.
  arXiv  Detail & Related papers  (2023-03-01T13:35:22Z)
• The Roles and Modes of Human Interactions with Automated Machine Learning Systems [7.670270099306412]
  Automated machine learning (AutoML) systems continue to progress in both sophistication and performance. It becomes important to understand the how' and why' of human-computer interaction (HCI) within these frameworks. This review serves to identify key research directions aimed at better facilitating the roles and modes of human interactions with both current and future AutoML systems.
  arXiv  Detail & Related papers  (2022-05-09T09:28:43Z)
• AutonoML: Towards an Integrated Framework for Autonomous Machine Learning [9.356870107137095]
  Review seeks to motivate a more expansive perspective on what constitutes an automated/autonomous ML system. In doing so, we survey developments in the following research areas. We develop a conceptual framework throughout the review, augmented by each topic, to illustrate one possible way of fusing high-level mechanisms into an autonomous ML system.
  arXiv  Detail & Related papers  (2020-12-23T11:01:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.