Learning Bidirectional Action-Language Translation with Limited Supervision and Incongruent Extra Input

• URL: http://arxiv.org/abs/2301.03353v1
• Date: Mon, 9 Jan 2023 14:09:09 GMT
• Title: Learning Bidirectional Action-Language Translation with Limited Supervision and Incongruent Extra Input
• Authors: Ozan \"Ozdemir, Matthias Kerzel, Cornelius Weber, Jae Hee Lee, Muhammad Burhan Hafez, Patrick Bruns, Stefan Wermter
• Abstract summary: We model a weakly supervised learning paradigm using our Paired Gated Autoencoders (PGAE) model. We introduce the Paired Transformed Autoencoders (PTAE) model, using Transformer-based crossmodal attention. PTAE achieves significantly higher accuracy in language-to-action and action-to-language translations.
• Score: 14.548576165754804
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Human infant learning happens during exploration of the environment, by interaction with objects, and by listening to and repeating utterances casually, which is analogous to unsupervised learning. Only occasionally, a learning infant would receive a matching verbal description of an action it is committing, which is similar to supervised learning. Such a learning mechanism can be mimicked with deep learning. We model this weakly supervised learning paradigm using our Paired Gated Autoencoders (PGAE) model, which combines an action and a language autoencoder. After observing a performance drop when reducing the proportion of supervised training, we introduce the Paired Transformed Autoencoders (PTAE) model, using Transformer-based crossmodal attention. PTAE achieves significantly higher accuracy in language-to-action and action-to-language translations, particularly in realistic but difficult cases when only few supervised training samples are available. We also test whether the trained model behaves realistically with conflicting multimodal input. In accordance with the concept of incongruence in psychology, conflict deteriorates the model output. Conflicting action input has a more severe impact than conflicting language input, and more conflicting features lead to larger interference. PTAE can be trained on mostly unlabelled data where labeled data is scarce, and it behaves plausibly when tested with incongruent input.

Related papers

• On the Shortcut Learning in Multilingual Neural Machine Translation [95.30470845501141]
  This study revisits the commonly-cited off-target issue in multilingual neural machine translation (MNMT) We attribute the off-target issue to the overfitting of the shortcuts of (non-centric, centric) language mappings. Analyses on learning dynamics show that the shortcut learning generally occurs in the later stage of model training.
  arXiv  Detail & Related papers  (2024-11-15T21:09:36Z)
• An iterated learning model of language change that mixes supervised and unsupervised learning [0.0]
  The iterated learning model is an agent-based model of language change in which language is transmitted from a tutor to a pupil. Previous models have implemented an agent's mapping from signals to meanings using an artificial neural network decoder. Here, a new model is presented in which both decoder and encoder are neural networks, trained separately through supervised learning, and trained together through unsupervised learning in the form of an autoencoder.
  arXiv  Detail & Related papers  (2024-05-31T14:14:01Z)
• Unsupervised Continual Anomaly Detection with Contrastively-learned Prompt [80.43623986759691]
  We introduce a novel Unsupervised Continual Anomaly Detection framework called UCAD. The framework equips the UAD with continual learning capability through contrastively-learned prompts. We conduct comprehensive experiments and set the benchmark on unsupervised continual anomaly detection and segmentation.
  arXiv  Detail & Related papers  (2024-01-02T03:37:11Z)
• Improving Language Models Meaning Understanding and Consistency by Learning Conceptual Roles from Dictionary [65.268245109828]
  Non-human-like behaviour of contemporary pre-trained language models (PLMs) is a leading cause undermining their trustworthiness. A striking phenomenon is the generation of inconsistent predictions, which produces contradictory results. We propose a practical approach that alleviates the inconsistent behaviour issue by improving PLM awareness.
  arXiv  Detail & Related papers  (2023-10-24T06:15:15Z)
• Explainable Verbal Deception Detection using Transformers [1.5104201344012347]
  This paper proposes and evaluates six deep-learning models, including combinations of BERT (and RoBERTa), MultiHead Attention, co-attentions, and transformers. The findings suggest that our transformer-based models can enhance automated deception detection performances (+2.11% in accuracy)
  arXiv  Detail & Related papers  (2022-10-06T17:36:00Z)
• Bridging the Gap Between Training and Inference of Bayesian Controllable Language Models [58.990214815032495]
  Large-scale pre-trained language models have achieved great success on natural language generation tasks. BCLMs have been shown to be efficient in controllable language generation. We propose a "Gemini Discriminator" for controllable language generation which alleviates the mismatch problem with a small computational cost.
  arXiv  Detail & Related papers  (2022-06-11T12:52:32Z)
• Language Model-Based Paired Variational Autoencoders for Robotic Language Learning [18.851256771007748]
  Similar to human infants, artificial agents can learn language while interacting with their environment. We present a neural model that bidirectionally binds robot actions and their language descriptions in a simple object manipulation scenario. Next, we introduce PVAE-BERT, which equips the model with a pretrained large-scale language model.
  arXiv  Detail & Related papers  (2022-01-17T10:05:26Z)
• Simple Contrastive Representation Adversarial Learning for NLP Tasks [17.12062566060011]
  Two novel frameworks, supervised contrastive adversarial learning (SCAL) and unsupervised SCAL (USCAL), are proposed. We employ it to Transformer-based models for natural language understanding, sentence semantic textual similarity and adversarial learning tasks. Experimental results on GLUE benchmark tasks show that our fine-tuned supervised method outperforms BERT$_base$ over 1.75%.
  arXiv  Detail & Related papers  (2021-11-26T03:16:09Z)
• An Explanation of In-context Learning as Implicit Bayesian Inference [117.19809377740188]
  We study the role of the pretraining distribution on the emergence of in-context learning. We prove that in-context learning occurs implicitly via Bayesian inference of the latent concept. We empirically find that scaling model size improves in-context accuracy even when the pretraining loss is the same.
  arXiv  Detail & Related papers  (2021-11-03T09:12:33Z)
• Adversarial Training with Contrastive Learning in NLP [0.0]
  We propose adversarial training with contrastive learning (ATCL) to adversarially train a language processing task. The core idea is to make linear perturbations in the embedding space of the input via fast gradient methods (FGM) and train the model to keep the original and perturbed representations close via contrastive learning. The results show not only an improvement in the quantitative (perplexity and BLEU) scores when compared to the baselines, but ATCL also achieves good qualitative results in the semantic level for both tasks.
  arXiv  Detail & Related papers  (2021-09-19T07:23:45Z)
• Exploring Unsupervised Pretraining Objectives for Machine Translation [99.5441395624651]
  Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT) Most approaches adapt masked-language modeling (MLM) to sequence-to-sequence architectures, by masking parts of the input and reconstructing them in the decoder. We compare masking with alternative objectives that produce inputs resembling real (full) sentences, by reordering and replacing words based on their context.
  arXiv  Detail & Related papers  (2021-06-10T10:18:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.