Semi-Structured Object Sequence Encoders
- URL: http://arxiv.org/abs/2301.01015v4
- Date: Tue, 23 May 2023 02:33:22 GMT
- Title: Semi-Structured Object Sequence Encoders
- Authors: Rudra Murthy V and Riyaz Bhat and Chulaka Gunasekara and Siva Sankalp
Patel and Hui Wan and Tejas Indulal Dhamecha and Danish Contractor and Marina
Danilevsky
- Abstract summary: We focus on the problem of developing a structure-aware input representation for semi-structured object sequences.
This type of data is often represented as a sequence of sets of key-value pairs over time.
We propose a two-part approach, which first considers each key independently and encodes a representation of its values over time.
- Score: 9.257633944317735
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: In this paper we explore the task of modeling semi-structured object
sequences; in particular, we focus our attention on the problem of developing a
structure-aware input representation for such sequences. Examples of such data
include user activity on websites, machine logs, and many others. This type of
data is often represented as a sequence of sets of key-value pairs over time
and can present modeling challenges due to an ever-increasing sequence length.
We propose a two-part approach, which first considers each key independently
and encodes a representation of its values over time; we then self-attend over
these value-aware key representations to accomplish a downstream task. This
allows us to operate on longer object sequences than existing methods. We
introduce a novel shared-attention-head architecture between the two modules
and present an innovative training schedule that interleaves the training of
both modules with shared weights for some attention heads. Our experiments on
multiple prediction tasks using real-world data demonstrate that our approach
outperforms a unified network with hierarchical encoding, as well as other
methods including a record-centric representation and a flattened
representation of the sequence.
Related papers
- Capturing Temporal Components for Time Series Classification [5.70772577110828]
This work introduces a textitcompositional representation learning approach trained on statistically coherent components extracted from sequential data.
Based on a multi-scale change space, an unsupervised approach is proposed to segment the sequential data into chunks with similar statistical properties.
A sequence-based encoder model is trained in a multi-task setting to learn compositional representations from these temporal components for time series classification.
arXiv Detail & Related papers (2024-06-20T16:15:21Z) - Advancing Time Series Classification with Multimodal Language Modeling [6.624754582682479]
We propose InstructTime, a novel attempt to reshape time series classification as a learning-to-generate paradigm.
The core idea is to formulate the classification of time series as a multimodal understanding task, in which both task-specific instructions and raw time series are treated as multimodal inputs.
Extensive experiments are conducted over benchmark datasets, whose results uncover the superior performance of InstructTime.
arXiv Detail & Related papers (2024-03-19T02:32:24Z) - TimeMAE: Self-Supervised Representations of Time Series with Decoupled
Masked Autoencoders [55.00904795497786]
We propose TimeMAE, a novel self-supervised paradigm for learning transferrable time series representations based on transformer networks.
The TimeMAE learns enriched contextual representations of time series with a bidirectional encoding scheme.
To solve the discrepancy issue incurred by newly injected masked embeddings, we design a decoupled autoencoder architecture.
arXiv Detail & Related papers (2023-03-01T08:33:16Z) - Real-World Compositional Generalization with Disentangled
Sequence-to-Sequence Learning [81.24269148865555]
A recently proposed Disentangled sequence-to-sequence model (Dangle) shows promising generalization capability.
We introduce two key modifications to this model which encourage more disentangled representations and improve its compute and memory efficiency.
Specifically, instead of adaptively re-encoding source keys and values at each time step, we disentangle their representations and only re-encode keys periodically.
arXiv Detail & Related papers (2022-12-12T15:40:30Z) - Complex-Valued Autoencoders for Object Discovery [62.26260974933819]
We propose a distributed approach to object-centric representations: the Complex AutoEncoder.
We show that this simple and efficient approach achieves better reconstruction performance than an equivalent real-valued autoencoder on simple multi-object datasets.
We also show that it achieves competitive unsupervised object discovery performance to a SlotAttention model on two datasets, and manages to disentangle objects in a third dataset where SlotAttention fails - all while being 7-70 times faster to train.
arXiv Detail & Related papers (2022-04-05T09:25:28Z) - Self-Attention Neural Bag-of-Features [103.70855797025689]
We build on the recently introduced 2D-Attention and reformulate the attention learning methodology.
We propose a joint feature-temporal attention mechanism that learns a joint 2D attention mask highlighting relevant information.
arXiv Detail & Related papers (2022-01-26T17:54:14Z) - ASFormer: Transformer for Action Segmentation [9.509416095106493]
We present an efficient Transformer-based model for action segmentation task, named ASFormer.
It constrains the hypothesis space within a reliable scope, and is beneficial for the action segmentation task to learn a proper target function with small training sets.
We apply a pre-defined hierarchical representation pattern that efficiently handles long input sequences.
arXiv Detail & Related papers (2021-10-16T13:07:20Z) - CoADNet: Collaborative Aggregation-and-Distribution Networks for
Co-Salient Object Detection [91.91911418421086]
Co-Salient Object Detection (CoSOD) aims at discovering salient objects that repeatedly appear in a given query group containing two or more relevant images.
One challenging issue is how to effectively capture co-saliency cues by modeling and exploiting inter-image relationships.
We present an end-to-end collaborative aggregation-and-distribution network (CoADNet) to capture both salient and repetitive visual patterns from multiple images.
arXiv Detail & Related papers (2020-11-10T04:28:11Z) - Keyphrase Extraction with Dynamic Graph Convolutional Networks and
Diversified Inference [50.768682650658384]
Keyphrase extraction (KE) aims to summarize a set of phrases that accurately express a concept or a topic covered in a given document.
Recent Sequence-to-Sequence (Seq2Seq) based generative framework is widely used in KE task, and it has obtained competitive performance on various benchmarks.
In this paper, we propose to adopt the Dynamic Graph Convolutional Networks (DGCN) to solve the above two problems simultaneously.
arXiv Detail & Related papers (2020-10-24T08:11:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.