pFedNavi: Structure-Aware Personalized Federated Vision-Language Navigation for Embodied AI

• URL: http://arxiv.org/abs/2602.14401v1
• Date: Mon, 16 Feb 2026 02:18:09 GMT
• Title: pFedNavi: Structure-Aware Personalized Federated Vision-Language Navigation for Embodied AI
• Abstract summary: Vision-Language Navigation VLN requires large-scale trajectory instruction data from private indoor environments.<n> Federated Learning FL mitigates this by keeping data on-device, but vanilla FL struggles under VLNs' extreme cross-client heterogeneity.<n>This paper proposes pFedNavi, a structure-aware and dynamically adaptive personalized federated learning framework tailored for VLN.
• Score: 27.929546538010516
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Navigation VLN requires large-scale trajectory instruction data from private indoor environments, raising significant privacy concerns. Federated Learning FL mitigates this by keeping data on-device, but vanilla FL struggles under VLNs' extreme cross-client heterogeneity in environments and instruction styles, making a single global model suboptimal. This paper proposes pFedNavi, a structure-aware and dynamically adaptive personalized federated learning framework tailored for VLN. Our key idea is to personalize where it matters: pFedNavi adaptively identifies client-specific layers via layer-wise mixing coefficients, and performs fine-grained parameter fusion on the selected components (e.g., the encoder-decoder projection and environment-sensitive decoder layers) to balance global knowledge sharing with local specialization. We evaluate pFedNavi on two standard VLN benchmarks, R2R and RxR, using both ResNet and CLIP visual representations. Across all metrics, pFedNavi consistently outperforms the FedAvg-based VLN baseline, achieving up to 7.5% improvement in navigation success rate and up to 7.8% gain in trajectory fidelity, while converging 1.38x faster under non-IID conditions.

Related papers

• Offline Vision-Language Navigation with Geometric Goal Localization for Outdoor Environments [0.0]
  Foundation-model-based vision-language navigation (VLN) has advanced autonomous robot navigation by enabling robots to interpret natural-language instructions.<n>Existing VLN systems rely heavily on cloud-hosted foundation models for language understanding and semantic grounding.<n>This paper makes three contributions toward fully onboard VLN for outdoor environments.
  arXiv  Detail & Related papers  (2026-07-24T11:49:25Z)
• Optimizing Server Placement for Vertical Federated Learning in Dynamic Edge/Fog Networks [49.91939413642465]
  We investigate the control and optimization of vertical federated learning (VFL) methods in which edge/fog devices contain separate data features.<n>In this setting, our proposed methodology, server controlled VFL in dynamic networks (SC-DN), first establishes the existence of a global first-order stationary point for every global round.<n>We show that our methodology demonstrates superior classification/regression performance and resource consumption savings than even greedy methodologies.
  arXiv  Detail & Related papers  (2026-05-10T23:30:18Z)
• TagaVLM: Topology-Aware Global Action Reasoning for Vision-Language Navigation [70.23578202012048]
  Vision-Language Navigation (VLN) presents a unique challenge for Large Vision-Language Models (VLMs) due to their inherent architectural mismatch.<n>We propose TagaVLM (Topology-Aware Global Action reasoning), an end-to-end framework that explicitly injects topological structures into the VLM backbone.<n>To enhance topological node information, an Interleaved Navigation Prompt strengthens node-level visual-text alignment.<n>With the embedded topological graph, the model is capable of global action reasoning, allowing for robust path correction.
  arXiv  Detail & Related papers  (2026-03-03T13:28:07Z)
• Adaptive Dual-Weighting Framework for Federated Learning via Out-of-Distribution Detection [53.45696787935487]
  Federated Learning (FL) enables collaborative model training across large-scale distributed service nodes.<n>In real-world service-oriented deployments, data generated by heterogeneous users, devices, and application scenarios are inherently non-IID.<n>We propose FLood, a novel FL framework inspired by out-of-distribution (OOD) detection.
  arXiv  Detail & Related papers  (2026-02-01T05:54:59Z)
• VLN-Zero: Rapid Exploration and Cache-Enabled Neurosymbolic Vision-Language Planning for Zero-Shot Transfer in Robot Navigation [52.00474922315126]
  We present VLN-Zero, a vision-language navigation framework for unseen environments.<n>We use vision-language models to efficiently construct symbolic scene graphs and enable zero-shot neurosymbolic navigation.<n>VLN-Zero achieves 2x higher success rate compared to state-of-the-art zero-shot models, outperforms most fine-tuned baselines, and reaches goal locations in half the time.
  arXiv  Detail & Related papers  (2025-09-23T03:23:03Z)
• GCN-TULHOR: Trajectory-User Linking Leveraging GCNs and Higher-Order Spatial Representations [3.704533038474922]
  Trajectory-user linking (TUL) aims to associate anonymized trajectories with the users who generated them.<n>In this paper, we introduce GCN-TULHOR, a method that transforms raw location data into higher-order mobility flow representations.<n>Our approach converts both sparse check-in and continuous GPS trajectory data into unified higher-order flow representations.
  arXiv  Detail & Related papers  (2025-09-14T05:14:09Z)
• Personalized Subgraph Federated Learning with Differentiable Auxiliary Projections [15.488985833084408]
  We introduce Federated learning with Auxiliary projections (FedAux)<n>FedAux is a personalized subgraph FL framework that learns to align, compare, and aggregate heterogeneously distributed local models without sharing raw data or node embeddings.<n> Empirical evaluations across diverse graph benchmarks demonstrate that FedAux substantially outperforms existing baselines in both accuracy and personalization performance.
  arXiv  Detail & Related papers  (2025-05-29T09:17:49Z)
• Generalized and Personalized Federated Learning with Foundation Models via Orthogonal Transformations [4.008780119020479]
  Federated Learning aims to train models across decentralized clients or devices holding local data without the need for centralized data collection.<n>We introduce FedOT, a novel approach that leverages black-box foundation models.<n>FedOT mitigates gradient conflicts across diverse clients, preserves semantic integrity, and achieves robust performance even in the presence of substantial data.
  arXiv  Detail & Related papers  (2025-05-26T12:18:24Z)
• UnitedVLN: Generalizable Gaussian Splatting for Continuous Vision-Language Navigation [71.97405667493477]
  We introduce a novel, generalizable 3DGS-based pre-training paradigm, called UnitedVLN.<n>It enables agents to better explore future environments by unitedly rendering high-fidelity 360 visual images and semantic features.<n>UnitedVLN outperforms state-of-the-art methods on existing VLN-CE benchmarks.
  arXiv  Detail & Related papers  (2024-11-25T02:44:59Z)
• NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning [97.88246428240872]
  Vision-and-Language Navigation (VLN), as a crucial research problem of Embodied AI, requires an embodied agent to navigate through complex 3D environments following natural language instructions.<n>Recent research has highlighted the promising capacity of large language models (LLMs) in VLN by improving navigational reasoning accuracy and interpretability.<n>This paper introduces a novel strategy called Navigational Chain-of-Thought (NavCoT), where we fulfill parameter-efficient in-domain training to enable self-guided navigational decision.
  arXiv  Detail & Related papers  (2024-03-12T07:27:02Z)
• Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning [49.72857433721424]
  Vision Transformers (ViT) and Visual Prompt Tuning (VPT) achieve state-of-the-art performance with improved efficiency in various computer vision tasks. We present a novel algorithm, SGPT, that integrates Generalized FL (GFL) and Personalized FL (PFL) approaches by employing a unique combination of both shared and group-specific prompts.
  arXiv  Detail & Related papers  (2023-10-27T17:22:09Z)
• Reinforced Structured State-Evolution for Vision-Language Navigation [42.46176089721314]
  Vision-and-language Navigation (VLN) task requires an embodied agent to navigate to a remote location following a natural language instruction. Previous methods usually adopt a sequence model (e.g., Transformer and LSTM) as the navigator. We propose a novel Structured state-Evolution (SEvol) model to effectively maintain the environment layout clues for VLN.
  arXiv  Detail & Related papers  (2022-04-20T07:51:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.