Inducing Meaningful Units from Character Sequences with Dynamic Capacity Slot Attention

• URL: http://arxiv.org/abs/2102.01223v3
• Date: Tue, 16 Jan 2024 11:50:49 GMT
• Title: Inducing Meaningful Units from Character Sequences with Dynamic Capacity Slot Attention
• Authors: Melika Behjati and James Henderson
• Abstract summary: We propose an unsupervised distributional method to learn the abstract meaningful units in a sequence of characters. Rather than segmenting the sequence, our Dynamic Capacity Slot Attention model discovers continuous representations of the objects in the sequence.
• Score: 12.25208417841772
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Characters do not convey meaning, but sequences of characters do. We propose an unsupervised distributional method to learn the abstract meaningful units in a sequence of characters. Rather than segmenting the sequence, our Dynamic Capacity Slot Attention model discovers continuous representations of the objects in the sequence, extending an architecture for object discovery in images. We train our model on different languages and evaluate the quality of the obtained representations with forward and reverse probing classifiers. These experiments show that our model succeeds in discovering units which are similar to those proposed previously in form, content and level of abstraction, and which show promise for capturing meaningful information at a higher level of abstraction.

Related papers

• Building, Reusing, and Generalizing Abstract Representations from Concrete Sequences [51.965994405124455]
  Humans excel at learning abstract patterns across different sequences, filtering out irrelevant details. Many sequence learning models lack the ability to abstract, which leads to memory inefficiency and poor transfer. We introduce a non-parametric hierarchical variable learning model (HVM) that learns chunks from sequences and abstracts contextually similar chunks as variables.
  arXiv  Detail & Related papers  (2024-10-27T18:13:07Z)
• CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling [52.404072802235234]
  We introduce Chunked Instruction-aware State Eviction (CItruS), a novel modeling technique that integrates the attention preferences useful for a downstream task into the eviction process of hidden states. Our training-free method exhibits superior performance on long sequence comprehension and retrieval tasks over several strong baselines under the same memory budget.
  arXiv  Detail & Related papers  (2024-06-17T18:34:58Z)
• How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval? [120.49126407479717]
  We propose a sketch-based image retrieval framework capable of handling sketch abstraction at varied levels. For granularity-level abstraction understanding, we dictate that the retrieval model should not treat all abstraction-levels equally. Our Acc.@q loss uniquely allows a sketch to narrow/broaden its focus in terms of how stringent the evaluation should be.
  arXiv  Detail & Related papers  (2024-03-11T23:08:29Z)
• Explicitly Disentangled Representations in Object-Centric Learning [0.0]
  We propose a novel architecture that biases object-centric models toward disentangling shape and texture components. In particular, we propose a novel architecture that biases object-centric models toward disentangling shape and texture components.
  arXiv  Detail & Related papers  (2024-01-18T17:22:11Z)
• Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter [47.29967666846132]
  generative text-to-image diffusion models are highly efficient open-vocabulary semantic segmenters. We introduce a novel training-free approach named DiffSegmenter to generate realistic objects that are semantically faithful to the input text. Extensive experiments on three benchmark datasets show that the proposed DiffSegmenter achieves impressive results for open-vocabulary semantic segmentation.
  arXiv  Detail & Related papers  (2023-09-06T06:31:08Z)
• Learning Debiased and Disentangled Representations for Semantic Segmentation [52.35766945827972]
  We propose a model-agnostic and training scheme for semantic segmentation. By randomly eliminating certain class information in each training iteration, we effectively reduce feature dependencies among classes. Models trained with our approach demonstrate strong results on multiple semantic segmentation benchmarks.
  arXiv  Detail & Related papers  (2021-10-31T16:15:09Z)
• Saliency-driven Class Impressions for Feature Visualization of Deep Neural Networks [55.11806035788036]
  It is advantageous to visualize the features considered to be essential for classification. Existing visualization methods develop high confidence images consisting of both background and foreground features. In this work, we propose a saliency-driven approach to visualize discriminative features that are considered most important for a given task.
  arXiv  Detail & Related papers  (2020-07-31T06:11:06Z)
• A Few-Shot Sequential Approach for Object Counting [63.82757025821265]
  We introduce a class attention mechanism that sequentially attends to objects in the image and extracts their relevant features. The proposed technique is trained on point-level annotations and uses a novel loss function that disentangles class-dependent and class-agnostic aspects of the model. We present our results on a variety of object-counting/detection datasets, including FSOD and MS COCO.
  arXiv  Detail & Related papers  (2020-07-03T18:23:39Z)
• Object-Centric Learning with Slot Attention [43.684193749891506]
  We present the Slot Attention module, an architectural component that interfaces with perceptual representations. Slot Attention produces task-dependent abstract representations which we call slots. We empirically demonstrate that Slot Attention can extract object-centric representations that enable generalization to unseen compositions.
  arXiv  Detail & Related papers  (2020-06-26T15:31:57Z)
• Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Reward [42.925345819778656]
  We present ASGARD, a novel framework for Abstractive Summarization with Graph-Augmentation and semantic-driven RewarD. We propose the use of dual encoders---a sequential document encoder and a graph-structured encoder---to maintain the global context and local characteristics of entities. Results show that our models produce significantly higher ROUGE scores than a variant without knowledge graph as input on both New York Times and CNN/Daily Mail datasets.
  arXiv  Detail & Related papers  (2020-05-03T18:23:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.