Learn-to-learn on Arbitrary Textual Conditioning: A Hypernetwork-Driven Meta-Gated LLM

• URL: http://arxiv.org/abs/2605.01973v1
• Date: Sun, 03 May 2026 17:13:45 GMT
• Title: Learn-to-learn on Arbitrary Textual Conditioning: A Hypernetwork-Driven Meta-Gated LLM
• Abstract summary: In this paper, we activate the meta-signal of $$ within the SwiGLU blocks, resulting in a meta-gating mechanism that adaptively adjusts the nonlinearity of FFN.<n>By testing on different condition types such as task, domain, persona, and style, our method outperforms finetuning and meta-learning baselines.
• Score: 5.325313097784103
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Conventional LLMs may suffer from corpus heterogeneity and subtle condition changes. While finetuning can create the catastrophe forgetting issue, application of meta-learning on LLMs is also limited due to its complexity and scalability. In this paper, we activate the meta-signal of $β$ within the SwiGLU blocks, resulting in a meta-gating mechanism that adaptively adjusts the nonlinearity of FFN. A hypernetwork is employed which dynamically produces $β$ on textual conditions, providing meta-controllability on LLMs. By testing on different condition types such as task, domain, persona, and style, our method outperforms finetuning and meta-learning baselines, and can generalize reasonably on unseen tasks, condition types, or instructions. Our code can be found in https://github.com/AaronJi/MeGan.

Related papers

• Should LLM Agents Decide in Social Simulations? Comparing Finite-State and LLM-Based Decision Policies [44.05636851266385]
  This paper evaluates whether large language models preserve an interpretable reference policy in an online social network simulation.<n>Three open-weight LLMs are tested: LLaMA 3.1, GPT-OSS, and Mistral 24B.<n>Results show that LLMs can approximate the reference policy in some configurations, but do not preserve it reliably.
  arXiv  Detail & Related papers  (2026-06-10T17:35:32Z)
• OLLM: Options-based Large Language Models [1.4783646973333087]
  We introduce Options LLM (OLLM), a simple, general method that replaces the single next-token prediction of standard LLMs.<n>OLLM models variation explicitly: a small latent space parametrizes multiple plausible next-token options which can be selected or searched by a downstream policy.<n>Our results demonstrate that optionized next-token modeling enhances controllability, robustness, and efficiency in math reasoning.
  arXiv  Detail & Related papers  (2026-04-21T04:59:37Z)
• Introspection Adapters: Training LLMs to Report Their Learned Behaviors [8.097590361201535]
  We study a scalable approach to rapidly identify learned behaviors of many LLMs derived from a shared base LLM.<n>A single LoRA adapter jointly trained across the finetunes $M_i$ causes them to verbalize their implanted behaviors.<n>We find that this IA induces self-description of learned behaviors even in finetunes of $M$ that were trained in very different ways from the $M_i$.
  arXiv  Detail & Related papers  (2026-04-18T03:50:00Z)
• RIMRULE: Improving Tool-Using Language Agents via MDL-Guided Rule Learning [31.8598046524742]
  RIMRULE is a neuro-symbolic approach for large language models (LLMs) adaptation based on dynamic rule injection.<n> Compact, interpretable rules are distilled from failure traces and injected into the prompt during inference to improve task performance.
  arXiv  Detail & Related papers  (2025-12-31T19:40:10Z)
• Interactive and Expressive Code-Augmented Planning with Large Language Models [62.799579304821826]
  Large Language Models (LLMs) demonstrate strong abilities in common-sense reasoning and interactive decision-making. Recent techniques have sought to structure LLM outputs using control flow and other code-adjacent techniques to improve planning performance. We propose REPL-Plan, an LLM planning approach that is fully code-expressive and dynamic.
  arXiv  Detail & Related papers  (2024-11-21T04:23:17Z)
• Open-domain Implicit Format Control for Large Language Model Generation [52.83173553689678]
  We introduce a novel framework for controlled generation in large language models (LLMs) This study investigates LLMs' capabilities to follow open-domain, one-shot constraints and replicate the format of the example answers. We also develop a dataset collection methodology for supervised fine-tuning that enhances the open-domain format control of LLMs without degrading output quality.
  arXiv  Detail & Related papers  (2024-08-08T11:51:45Z)
• DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph [70.79413606968814]
  We introduce Dynamic Evaluation of LLMs via Adaptive Reasoning Graph Evolvement (DARG) to dynamically extend current benchmarks with controlled complexity and diversity. Specifically, we first extract the reasoning graphs of data points in current benchmarks and then perturb the reasoning graphs to generate novel testing data. Such newly generated test samples can have different levels of complexity while maintaining linguistic diversity similar to the original benchmarks.
  arXiv  Detail & Related papers  (2024-06-25T04:27:53Z)
• One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models [67.49462724595445]
  Retrieval-augmented generation (RAG) is a promising way to improve large language models (LLMs)<n>We propose a novel method that involves learning scalable and pluggable virtual tokens for RAG.
  arXiv  Detail & Related papers  (2024-05-30T03:44:54Z)
• From LLMs to Actions: Latent Codes as Bridges in Hierarchical Robot Control [58.72492647570062]
  We introduce our method -- Learnable Latent Codes as Bridges (LCB) -- as an alternate architecture to overcome limitations.<n>We find that methodoutperforms baselines that leverage pure language as the interface layer on tasks that require reasoning and multi-step behaviors.
  arXiv  Detail & Related papers  (2024-05-08T04:14:06Z)
• BOML: A Modularized Bilevel Optimization Library in Python for Meta Learning [52.90643948602659]
  BOML is a modularized optimization library that unifies several meta-learning algorithms into a common bilevel optimization framework. It provides a hierarchical optimization pipeline together with a variety of iteration modules, which can be used to solve the mainstream categories of meta-learning methods.
  arXiv  Detail & Related papers  (2020-09-28T14:21:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.