Harnessing Input-Adaptive Inference for Efficient VLN

• URL: http://arxiv.org/abs/2508.09262v1
• Date: Tue, 12 Aug 2025 18:05:33 GMT
• Title: Harnessing Input-Adaptive Inference for Efficient VLN
• Authors: Dongwoo Kang, Akhil Perincherry, Zachary Coalson, Aiden Gabriel, Stefan Lee, Sanghyun Hong,
• Abstract summary: An emerging paradigm in vision-and-language navigation (VLN) is the use of history-aware multi-modal transformer models.<n>We propose a novel input-adaptive navigation method to enhance VLN model efficiency.
• Score: 13.847596428283861
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: An emerging paradigm in vision-and-language navigation (VLN) is the use of history-aware multi-modal transformer models. Given a language instruction, these models process observation and navigation history to predict the most appropriate action for an agent. While they have significantly improved performance, the scale of these models can be a bottleneck in practical settings with limited computational resources. In this work, we propose a novel input-adaptive navigation method to enhance VLN model efficiency. We first show that existing input-adaptive mechanisms fail to reduce computations without substantial performance degradation. To address this, we introduce three adaptive algorithms, each deployed at a different level: (1) To improve spatial efficiency, we selectively process panoramic views at each observation of an agent. (2) To improve intra-model efficiency, we propose importance-based adaptive thresholding for the early-exit methods. (3) To improve temporal efficiency, we implement a caching mechanism that prevents reprocessing of views previously seen by the agent. In evaluations on seven VLN benchmarks, we demonstrate over a 2$\times$ reduction in computation across three off-the-shelf agents in both standard and continuous environments. Our code is publicly available at https://github.com/secure-ai-systems-group/adaptive-vision-and-language-navigation.

Related papers

• ContextEvolve: Multi-Agent Context Compression for Systems Code Optimization [8.046059974853858]
  We introduce ContextEvolve, a multi-agent framework that achieves RL-level search efficiency under strict parameter-blind constraints.<n>On the ADRS benchmark, ContextEvolve outperforms state-of-the-art baselines by 33.3% while reducing token consumption by 29.0%.
  arXiv  Detail & Related papers  (2026-02-01T16:50:07Z)
• Efficient Onboard Vision-Language Inference in UAV-Enabled Low-Altitude Economy Networks via LLM-Enhanced Optimization [61.55616421408666]
  Low-Altitude Economy Networks (LAENets) have enabled a variety of applications, including aerial surveillance, environmental sensing, and semantic data collection.<n> onboard vision (VLMs) offer inference for real-time inference but limited onboard dynamic network conditions.<n>We propose a UAV-enabled LAENet system that improves communication efficiency under dynamic LAENet conditions.
  arXiv  Detail & Related papers  (2025-10-11T05:11:21Z)
• PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection [68.8373788348678]
  Visual instruction tuning adapts pre-trained Multimodal Large Language Models to follow human instructions.<n>PRISM is the first training-free framework for efficient visual instruction selection.<n>It reduces the end-to-end time for data selection and model tuning to just 30% of conventional pipelines.
  arXiv  Detail & Related papers  (2025-02-17T18:43:41Z)
• ALOcc: Adaptive Lifting-based 3D Semantic Occupancy and Cost Volume-based Flow Prediction [89.89610257714006]
  Existing methods prioritize higher accuracy to cater to the demands of these tasks. We introduce a series of targeted improvements for 3D semantic occupancy prediction and flow estimation. Our purelytemporalal architecture framework, named ALOcc, achieves an optimal tradeoff between speed and accuracy.
  arXiv  Detail & Related papers  (2024-11-12T11:32:56Z)
• Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate [105.86576388991713]
  We introduce a normalized gradient difference (NGDiff) algorithm, enabling us to have better control over the trade-off between the objectives.<n>We provide a theoretical analysis and empirically demonstrate the superior performance of NGDiff among state-of-the-art unlearning methods on the TOFU and MUSE datasets.
  arXiv  Detail & Related papers  (2024-10-29T14:41:44Z)
• AdaOcc: Adaptive Forward View Transformation and Flow Modeling for 3D Occupancy and Flow Prediction [56.72301849123049]
  We present our solution for the Vision-Centric 3D Occupancy and Flow Prediction track in the nuScenes Open-Occ dataset challenge at CVPR 2024. Our innovative approach involves a dual-stage framework that enhances 3D occupancy and flow predictions by incorporating adaptive forward view transformation and flow modeling. Our method combines regression with classification to address scale variations in different scenes, and leverages predicted flow to warp current voxel features to future frames, guided by future frame ground truth.
  arXiv  Detail & Related papers  (2024-07-01T16:32:15Z)
• REP: Resource-Efficient Prompting for Rehearsal-Free Continual Learning [23.92661395403251]
  Recent rehearsal-free methods, guided by prompts, excel in vision-related continual learning (CL) with drifting data but lack resource efficiency.<n>We introduce Resource-Efficient Prompting (REP), which improves the computational and memory efficiency of prompt-based rehearsal-free methods.<n>Our approach employs swift prompt selection to refine input data using a carefully provisioned model.
  arXiv  Detail & Related papers  (2024-06-07T09:17:33Z)
• VeCAF: Vision-language Collaborative Active Finetuning with Training Objective Awareness [56.87603097348203]
  VeCAF uses labels and natural language annotations to perform parametric data selection for PVM finetuning. VeCAF incorporates the finetuning objective to select significant data points that effectively guide the PVM towards faster convergence. On ImageNet, VeCAF uses up to 3.3x less training batches to reach the target performance compared to full finetuning.
  arXiv  Detail & Related papers  (2024-01-15T17:28:37Z)
• ReCoRe: Regularized Contrastive Representation Learning of World Model [21.29132219042405]
  We present a world model that learns invariant features using contrastive unsupervised learning and an intervention-invariant regularizer. Our method outperforms current state-of-the-art model-based and model-free RL methods and significantly improves on out-of-distribution point navigation tasks evaluated on the iGibson benchmark.
  arXiv  Detail & Related papers  (2023-12-14T15:53:07Z)
• E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning [55.50908600818483]
  Fine-tuning large-scale pretrained vision models for new tasks has become increasingly parameter-intensive. We propose an Effective and Efficient Visual Prompt Tuning (E2VPT) approach for large-scale transformer-based model adaptation. Our approach outperforms several state-of-the-art baselines on two benchmarks.
  arXiv  Detail & Related papers  (2023-07-25T19:03:21Z)
• Break a Lag: Triple Exponential Moving Average for Enhanced Optimization [2.0199251985015434]
  We introduce Fast Adaptive Moment Estimation (FAME), a novel optimization technique that leverages the power of Triple Exponential Moving Average.<n>FAME enhances responsiveness to data dynamics, mitigates trend identification lag, and optimize learning efficiency.<n>Our comprehensive evaluation encompasses different computer vision tasks including image classification, object detection, and semantic segmentation, integrating FAME into 30 distinct architectures.
  arXiv  Detail & Related papers  (2023-06-02T10:29:33Z)
• An Efficient and Scalable Collection of Fly-inspired Voting Units for Visual Place Recognition in Changing Environments [20.485491385050615]
  Low-overhead VPR techniques would enable platforms equipped with low-end, cheap hardware. Our goal is to provide an algorithm of extreme compactness and efficiency while achieving state-of-the-art robustness to appearance changes and small point-of-view variations.
  arXiv  Detail & Related papers  (2021-09-22T19:01:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.