FastRE: Towards Fast Relation Extraction with Convolutional Encoder and
Improved Cascade Binary Tagging Framework
- URL: http://arxiv.org/abs/2205.02490v1
- Date: Thu, 5 May 2022 07:59:51 GMT
- Title: FastRE: Towards Fast Relation Extraction with Convolutional Encoder and
Improved Cascade Binary Tagging Framework
- Authors: Guozheng Li and Xu Chen and Peng Wang and Jiafeng Xie and Qiqing Luo
- Abstract summary: We propose a fast relation extraction model (FastRE) based on convolutional encoder and improved cascade binary tagging framework.
FastRE achieves 3-10x training speed, 7-15x inference speed faster, and 1/100 parameters compared to the state-of-the-art models.
- Score: 13.4666880421568
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Recent work for extracting relations from texts has achieved excellent
performance. However, most existing methods pay less attention to the
efficiency, making it still challenging to quickly extract relations from
massive or streaming text data in realistic scenarios. The main efficiency
bottleneck is that these methods use a Transformer-based pre-trained language
model for encoding, which heavily affects the training speed and inference
speed. To address this issue, we propose a fast relation extraction model
(FastRE) based on convolutional encoder and improved cascade binary tagging
framework. Compared to previous work, FastRE employs several innovations to
improve efficiency while also keeping promising performance. Concretely, FastRE
adopts a novel convolutional encoder architecture combined with dilated
convolution, gated unit and residual connection, which significantly reduces
the computation cost of training and inference, while maintaining the
satisfactory performance. Moreover, to improve the cascade binary tagging
framework, FastRE first introduces a type-relation mapping mechanism to
accelerate tagging efficiency and alleviate relation redundancy, and then
utilizes a position-dependent adaptive thresholding strategy to obtain higher
tagging accuracy and better model generalization. Experimental results
demonstrate that FastRE is well balanced between efficiency and performance,
and achieves 3-10x training speed, 7-15x inference speed faster, and 1/100
parameters compared to the state-of-the-art models, while the performance is
still competitive.
Related papers
- MUSIC: Accelerated Convergence for Distributed Optimization With Inexact
and Exact Methods [6.800113478497425]
In this paper, we propose an accelerated framework named as MUSIC allowing each agent to perform multiple local updates and a single combination in each iteration.
We equip inexact and exact distributed optimization methods into this framework, thereby developing two new algorithms that exhibit accelerated linear convergence and high communication efficiency.
arXiv Detail & Related papers (2024-03-05T02:02:00Z) - Federated Learning of Large Language Models with Parameter-Efficient
Prompt Tuning and Adaptive Optimization [71.87335804334616]
Federated learning (FL) is a promising paradigm to enable collaborative model training with decentralized data.
The training process of Large Language Models (LLMs) generally incurs the update of significant parameters.
This paper proposes an efficient partial prompt tuning approach to improve performance and efficiency simultaneously.
arXiv Detail & Related papers (2023-10-23T16:37:59Z) - Fast as CHITA: Neural Network Pruning with Combinatorial Optimization [9.440450886684603]
We propose a novel optimization-based pruning framework that considers the combined effect of pruning (and updating) multiple weights subject to a sparsity constraint.
Our approach, CHITA, extends the classical Brain Surgeon framework and results in significant improvements in speed, memory, and performance.
arXiv Detail & Related papers (2023-02-28T15:03:18Z) - HEAT: Hardware-Efficient Automatic Tensor Decomposition for Transformer
Compression [69.36555801766762]
We propose a hardware-aware tensor decomposition framework, dubbed HEAT, that enables efficient exploration of the exponential space of possible decompositions.
We experimentally show that our hardware-aware factorized BERT variants reduce the energy-delay product by 5.7x with less than 1.1% accuracy loss.
arXiv Detail & Related papers (2022-11-30T05:31:45Z) - ECO-TR: Efficient Correspondences Finding Via Coarse-to-Fine Refinement [80.94378602238432]
We propose an efficient structure named Correspondence Efficient Transformer (ECO-TR) by finding correspondences in a coarse-to-fine manner.
To achieve this, multiple transformer blocks are stage-wisely connected to gradually refine the predicted coordinates.
Experiments on various sparse and dense matching tasks demonstrate the superiority of our method in both efficiency and effectiveness against existing state-of-the-arts.
arXiv Detail & Related papers (2022-09-25T13:05:33Z) - An Efficiency Study for SPLADE Models [5.725475501578801]
In this paper, we focus on improving the efficiency of the SPLADE model.
We propose several techniques including L1 regularization for queries, a separation of document/ encoders, a FLOPS-regularized middle-training, and the use of faster query encoders.
arXiv Detail & Related papers (2022-07-08T11:42:05Z) - Efficient Few-Shot Object Detection via Knowledge Inheritance [62.36414544915032]
Few-shot object detection (FSOD) aims at learning a generic detector that can adapt to unseen tasks with scarce training samples.
We present an efficient pretrain-transfer framework (PTF) baseline with no computational increment.
We also propose an adaptive length re-scaling (ALR) strategy to alleviate the vector length inconsistency between the predicted novel weights and the pretrained base weights.
arXiv Detail & Related papers (2022-03-23T06:24:31Z) - Cascaded Fast and Slow Models for Efficient Semantic Code Search [46.53530668938728]
We propose an efficient and accurate semantic code search framework with cascaded fast and slow models.
The proposed cascaded approach is not only efficient and scalable, but also achieves state-of-the-art results.
arXiv Detail & Related papers (2021-10-15T02:23:35Z) - Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for
Improved Cross-Modal Retrieval [80.35589927511667]
Current state-of-the-art approaches to cross-modal retrieval process text and visual input jointly, relying on Transformer-based architectures with cross-attention mechanisms that attend over all words and objects in an image.
We propose a novel fine-tuning framework which turns any pretrained text-image multi-modal model into an efficient retrieval model.
Our experiments on a series of standard cross-modal retrieval benchmarks in monolingual, multilingual, and zero-shot setups, demonstrate improved accuracy and huge efficiency benefits over the state-of-the-art cross-encoders.
arXiv Detail & Related papers (2021-03-22T15:08:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.