Heuristic Rank Selection with Progressively Searching Tensor Ring Network

• URL: http://arxiv.org/abs/2009.10580v2
• Date: Sun, 30 May 2021 08:44:25 GMT
• Title: Heuristic Rank Selection with Progressively Searching Tensor Ring Network
• Authors: Nannan Li, Yu Pan, Yaran Chen, Zixiang Ding, Dongbin Zhao, Zenglin Xu
• Abstract summary: Ring Networks (TRNs) have been applied in deep networks, achieving remarkable successes in compression ratio and accuracy. We propose a novel progressive genetic algorithm named Progressively Searching Ring Network Search (PSTRN), which has the ability to find optimal rank precisely and efficiently. Our method is validated on public benchmarks like MNIST, CIFAR10/100, UCF11 and HMDB51, achieving the state-of-the-art performance.
• Score: 25.003013285907524
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, Tensor Ring Networks (TRNs) have been applied in deep networks, achieving remarkable successes in compression ratio and accuracy. Although highly related to the performance of TRNs, rank selection is seldom studied in previous works and usually set to equal in experiments. Meanwhile, there is not any heuristic method to choose the rank, and an enumerating way to find appropriate rank is extremely time-consuming. Interestingly, we discover that part of the rank elements is sensitive and usually aggregate in a narrow region, namely an interest region. Therefore, based on the above phenomenon, we propose a novel progressive genetic algorithm named Progressively Searching Tensor Ring Network Search (PSTRN), which has the ability to find optimal rank precisely and efficiently. Through the evolutionary phase and progressive phase, PSTRN can converge to the interest region quickly and harvest good performance. Experimental results show that PSTRN can significantly reduce the complexity of seeking rank, compared with the enumerating method. Furthermore, our method is validated on public benchmarks like MNIST, CIFAR10/100, UCF11 and HMDB51, achieving the state-of-the-art performance.

Related papers

• Faster WIND: Accelerating Iterative Best-of-$N$ Distillation for LLM Alignment [81.84950252537618]
  This paper reveals a unified game-theoretic connection between iterative BOND and self-play alignment. We establish a novel framework, WIN rate Dominance (WIND), with a series of efficient algorithms for regularized win rate dominance optimization.
  arXiv  Detail & Related papers  (2024-10-28T04:47:39Z)
• BOND: Aligning LLMs with Best-of-N Distillation [63.254031574394965]
  We propose Best-of-N Distillation (BOND), a novel RLHF algorithm that seeks to emulate Best-of-N but without its significant computational overhead at inference time. Specifically, BOND is a distribution matching algorithm that forces the distribution of generations from the policy to get closer to the Best-of-N distribution. We demonstrate the effectiveness of our approach and several design choices through experiments on abstractive summarization and Gemma models.
  arXiv  Detail & Related papers  (2024-07-19T18:38:25Z)
• Random Search as a Baseline for Sparse Neural Network Architecture Search [0.0]
  Sparse neural networks have shown similar or better performance than their dense counterparts while having higher parameter efficiency. This has motivated a number of works to learn or search for high performing sparse networks. We propose Random Search as a baseline algorithm for finding good sparse configurations and study its performance. We observe that for this sparse architecture search task, sparse networks found by Random Search neither perform better nor converge more efficiently than their random counterparts.
  arXiv  Detail & Related papers  (2024-03-13T05:32:13Z)
• AANet: Aggregation and Alignment Network with Semi-hard Positive Sample Mining for Hierarchical Place Recognition [48.043749855085025]
  Visual place recognition (VPR) is one of the research hotspots in robotics, which uses visual information to locate robots. We present a unified network capable of extracting global features for retrieving candidates via an aggregation module. We also propose a Semi-hard Positive Sample Mining (ShPSM) strategy to select appropriate hard positive images for training more robust VPR networks.
  arXiv  Detail & Related papers  (2023-10-08T14:46:11Z)
• RankDNN: Learning to Rank for Few-shot Learning [70.49494297554537]
  This paper introduces a new few-shot learning pipeline that casts relevance ranking for image retrieval as binary ranking relation classification. It provides a new perspective on few-shot learning and is complementary to state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-11-28T13:59:31Z)
• Heuristic Search for Rank Aggregation with Application to Label Ranking [16.275063634853584]
  We propose an effective hybrid evolutionary ranking algorithm to solve the rank aggregation problem. The algorithm features a semantic crossover based on concordant pairs and a late acceptance local search reinforced by an efficient incremental evaluation technique. Experiments are conducted to assess the algorithm, indicating a highly competitive performance on benchmark instances.
  arXiv  Detail & Related papers  (2022-01-11T11:43:17Z)
• Back to Basics: Efficient Network Compression via IMP [22.586474627159287]
  Iterative Magnitude Pruning (IMP) is one of the most established approaches for network pruning. IMP is often argued that it reaches suboptimal states by not incorporating sparsification into the training phase. We find that IMP with SLR for retraining can outperform state-of-the-art pruning-during-training approaches.
  arXiv  Detail & Related papers  (2021-11-01T11:23:44Z)
• ZARTS: On Zero-order Optimization for Neural Architecture Search [94.41017048659664]
  Differentiable architecture search (DARTS) has been a popular one-shot paradigm for NAS due to its high efficiency. This work turns to zero-order optimization and proposes a novel NAS scheme, called ZARTS, to search without enforcing the above approximation. In particular, results on 12 benchmarks verify the outstanding robustness of ZARTS, where the performance of DARTS collapses due to its known instability issue.
  arXiv  Detail & Related papers  (2021-10-10T09:35:15Z)
• Temporal Action Localization Using Gated Recurrent Units [6.091096843566857]
  We propose a new network based on Gated Recurrent Unit (GRU) and two novel post-processing ideas for TAL task. Specifically, we propose a new design for the output layer of the GRU resulting in the so-called GRU-Splitted model. We evaluate the performance of the proposed method compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-08-07T06:25:29Z)
• Query-level Early Exit for Additive Learning-to-Rank Ensembles [14.240566571342924]
  Search engine ranking pipelines are commonly based on large ensembles of machine-learned decision trees. In this paper, we investigate the novel problem of textitquery-level early exiting We show that query-level early exiting achieves an overall gain of up to 7.5% in terms of NDCG@10 with a speedup of the scoring process of up to 2.2x.
  arXiv  Detail & Related papers  (2020-04-30T08:59:45Z)
• TRP: Trained Rank Pruning for Efficient Deep Neural Networks [69.06699632822514]
  We propose Trained Rank Pruning (TRP), which alternates between low rank approximation and training. A nuclear regularization optimized by sub-gradient descent is utilized to further promote low rank in TRP. The TRP trained network inherently has a low-rank structure, and is approximated with negligible performance loss.
  arXiv  Detail & Related papers  (2020-04-30T03:37:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.