Search Beyond Queries: Training Smaller Language Models for Web Interactions via Reinforcement Learning

• URL: http://arxiv.org/abs/2404.10887v1
• Date: Tue, 16 Apr 2024 20:15:32 GMT
• Title: Search Beyond Queries: Training Smaller Language Models for Web Interactions via Reinforcement Learning
• Authors: Moghis Fereidouni, A. B. Siddique,
• Abstract summary: This work introduces a Grounded Language Agent for Intelligent Web Interactions, called GLAINTEL. Drawing upon advancements in language modeling and reinforcement learning, GLAINTEL investigates the efficacy of transformer-based models in enhancing the search capabilities of interactive web environments. This work focuses on training smaller language models as agents across various scenarios, systematically evaluating the impact of human demonstrations on the training process.
• Score: 2.2973978268630852
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Traditional search systems focus on query formulation for effective results but face challenges in scenarios such as product searches where crucial product details (e.g., size, color) remain concealed until users visit specific product pages. This highlights the need for intelligent web navigation agents capable of formulating queries and navigating web pages according to users' high-level intents. In response to this need, this work introduces a Grounded Language Agent for Intelligent Web Interactions, called GLAINTEL. Drawing upon advancements in language modeling and reinforcement learning, GLAINTEL investigates the efficacy of transformer-based models in enhancing the search capabilities of interactive web environments. Given the dynamic action space for each state in web navigation, GLAINTEL employs the Flan-T5 architecture and incorporates language modeling and value estimation heads. This work focuses on training smaller language models as agents across various scenarios, systematically evaluating the impact of human demonstrations on the training process. Specifically, we investigate scenarios where no human demonstrations are available and subsequently assess the effective utilization of such demonstrations. We also explore unsupervised domain adaptation for situations where demonstrations are confined to a specific domain. Experimental evaluations across diverse setups demonstrate the effectiveness of training agents in unsupervised settings, outperforming in-context learning-based approaches that employ larger models with up to 540 billion parameters. Surprisingly, behavioral cloning-based methods that straightforwardly use human demonstrations do not outperform unsupervised learning-based methods. Additionally, combining human demonstrations with Reinforcement Learning-based training yields results comparable to models utilizing GPT-4.

Related papers

• The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis [23.757767581876063]
  In-context learning is a popular inference strategy where large language models solve a task using only a few labeled demonstrations. We show that the effectiveness of demonstrations varies significantly across models, tasks, and languages. We also find that strong instruction-following models including Llama 2-Chat, GPT-3.5, and GPT-4 are largely insensitive to the quality of demonstrations.
  arXiv  Detail & Related papers  (2024-02-20T12:53:31Z)
• In-context Learning with Retrieved Demonstrations for Language Models: A Survey [23.24271704145876]
  Few-shot in-context learners (ICL) are adept at adapting to new tasks with just a few demonstrations in the input context. Instead of using a fixed set of demonstrations, one recent development is to retrieve demonstrations tailored to each input query. We discuss and compare different design choices for retrieval models, retrieval training procedures, and inference algorithms.
  arXiv  Detail & Related papers  (2024-01-21T23:34:42Z)
• Fine-Tune Language Models as Multi-Modal Differential Equation Solvers [14.181842691371935]
  We present a transformation of in-context operator learning into a multi-modal paradigm. In particular, we take inspiration from the recent success of large language models, and propose using "captions" to integrate human knowledge about the operator.
  arXiv  Detail & Related papers  (2023-08-09T16:44:25Z)
• Towards A Unified Agent with Foundation Models [18.558328028366816]
  We investigate how to embed and leverage such abilities in Reinforcement Learning (RL) agents. We design a framework that uses language as the core reasoning tool, exploring how this enables an agent to tackle a series of fundamental RL challenges. We demonstrate substantial performance improvements over baselines in exploration efficiency and ability to reuse data from offline datasets.
  arXiv  Detail & Related papers  (2023-07-18T22:37:30Z)
• SINC: Self-Supervised In-Context Learning for Vision-Language Tasks [64.44336003123102]
  We propose a framework to enable in-context learning in large language models. A meta-model can learn on self-supervised prompts consisting of tailored demonstrations. Experiments show that SINC outperforms gradient-based methods in various vision-language tasks.
  arXiv  Detail & Related papers  (2023-07-15T08:33:08Z)
• Adaptive action supervision in reinforcement learning from real-world multi-agent demonstrations [10.174009792409928]
  We propose a method for adaptive action supervision in RL from real-world demonstrations in multi-agent scenarios. In the experiments, using chase-and-escape and football tasks with the different dynamics between the unknown source and target environments, we show that our approach achieved a balance between the generalization and the generalization ability compared with the baselines.
  arXiv  Detail & Related papers  (2023-05-22T13:33:37Z)
• Multimodal Web Navigation with Instruction-Finetuned Foundation Models [99.14209521903854]
  We study data-driven offline training for web agents with vision-language foundation models. We propose an instruction-following multimodal agent, WebGUM, that observes both webpage screenshots and HTML pages. We empirically demonstrate this recipe improves the agent's ability of grounded multimodal perception, HTML comprehension, and multi-step reasoning.
  arXiv  Detail & Related papers  (2023-05-19T17:44:34Z)
• Foundation Models for Decision Making: Problems, Methods, and Opportunities [124.79381732197649]
  Foundation models pretrained on diverse data at scale have demonstrated extraordinary capabilities in a wide range of vision and language tasks. New paradigms are emerging for training foundation models to interact with other agents and perform long-term reasoning. Research at the intersection of foundation models and decision making holds tremendous promise for creating powerful new systems.
  arXiv  Detail & Related papers  (2023-03-07T18:44:07Z)
• Few-shot Prompting Towards Controllable Response Generation [49.479958672988566]
  We first explored the combination of prompting and reinforcement learning (RL) to steer models' generation without accessing any of the models' parameters. We apply multi-task learning to make the model learn to generalize to new tasks better. Experiment results show that our proposed method can successfully control several state-of-the-art (SOTA) dialogue models without accessing their parameters.
  arXiv  Detail & Related papers  (2022-06-08T14:48:06Z)
• Visual Adversarial Imitation Learning using Variational Models [60.69745540036375]
  Reward function specification remains a major impediment for learning behaviors through deep reinforcement learning. Visual demonstrations of desired behaviors often presents an easier and more natural way to teach agents. We develop a variational model-based adversarial imitation learning algorithm.
  arXiv  Detail & Related papers  (2021-07-16T00:15:18Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.