Lean-SAM2: Target-Anchored Memory and Encoder Acceleration for SAM2

• URL: http://arxiv.org/abs/2607.19811v1
• Date: Wed, 22 Jul 2026 06:42:27 GMT
• Title: Lean-SAM2: Target-Anchored Memory and Encoder Acceleration for SAM2
• Abstract summary: Lean-SAM2 is a holistic lightweight framework designed to address the above vulnerabilities while systematically eliminating computational redundancies.<n>Lean-SAM2 integrates three collaborative mechanisms: (1) Target-Anchored Memory Pruning (TAMP) safeguards target tokens against deceptive attention; (2) Temporal Condensation with Insurance Memory (TCIM) condenses historical context via a visibility-gated fusion; and (3) Target-Anchored Risk-Aware Routing (TARR) selectively activates the heavy image encoder for target-related windows based on anchor similarity.
• Score: 7.989538161016021
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Segment Anything Model 2 (SAM2) has advanced temporal promptable segmentation, yet its deployment remains hindered by heavy memory cross-attention overhead and redundant full-frame visual feature extraction. While recent methods explore efficiency via heuristic memory pruning and window-based sparse routing, they typically suffer from catastrophic performance degradation in complex segmentation scenarios replete with occlusions and distractors. To resolve these limitations, we propose \textbf{Lean-SAM2}, a holistic lightweight framework designed to address the above vulnerabilities while systematically eliminating computational redundancies. Specifically, Lean-SAM2 integrates three collaborative mechanisms: (1) Target-Anchored Memory Pruning (TAMP) safeguards target tokens against deceptive attention by modulating raw attention significance with semantic consistency against prompt-derived foreground anchors; (2) Temporal Condensation with Insurance Memory (TCIM) condenses historical context via a visibility-gated fusion while conditionally archiving high-confidence entries in a parallel insurance bank; and (3) Target-Anchored Risk-Aware Routing (TARR) selectively activates the heavy image encoder for target-related windows based on anchor similarity, utilizing a risk-aware fallback policy to trigger full-frame refreshes during volatile transitions. Extensive evaluations across multiple challenging benchmarks demonstrate that Lean-SAM2 establishes a superior balance between accuracy and efficiency. For example, on the LVOSv2 validation dataset, Lean-SAM2 achieves overall inference speedups of $1.412\times$ and $1.417\times$ on the SAM2.1-Large and SAM2.1-Base+, respectively, significantly outperforming Efficient-SAM2 while boosting the corresponding $\mathcal{J}\&\mathcal{F}$ scores by $5.0\%$ and $3.6\%$. Code is available at https://github.com/DeawhaleQwQ/Lean-SAM2.

Related papers

• Bifocal Diffusion Language Models: Asymmetric Bidirectional Context for Parallel Generation [39.51953946287085]
  This paper introduces Bifocal dLLMs, a new paradigm that resolves this dilemma through emphasymmetric bidirectional context<n>Experiments on continued pretraining of Qwen3-1.7B with 60B tokens demonstrate that R2LM achieves $2.4times$ to $12.9times$ higher throughput than bidirectional dLLMs.
  arXiv  Detail & Related papers  (2026-06-26T05:26:11Z)
• FLARE: Diffusion for Hybrid Language Model [72.60770374799634]
  FLARE is a systematic conversion framework for hybrid-attention large language models.<n>It combines a token-equal AR-and-diffusion objective, hardware-aware kernels, and unified inference, enabling one checkpoint to support both AR-style verified decoding and diffusion-style parallel denoising.<n>Our results suggest practical dLLMs are limited not only by decoding algorithms, but also by transfer data quality and the training inefficiency of current block-diffusion objectives.
  arXiv  Detail & Related papers  (2026-06-01T06:58:15Z)
• Efficient-SAM2: Accelerating SAM2 with Object-Aware Visual Encoding and Memory Retrieval [22.632907736085034]
  Segment Anything Model 2 (SAM2) shows excellent performance in video object segmentation tasks.<n>We propose Efficient-SAM2, which promotes SAM2 to adaptively focus on object regions while eliminating task-irrelevant computations.<n>With negligible additional parameters and minimal training overhead, Efficient-SAM2 delivers 1.68x speedup on SAM2.1-L model with only 1.0% accuracy drop on SA-V test set.
  arXiv  Detail & Related papers  (2026-02-09T02:58:33Z)
• SimpleMem: Efficient Lifelong Memory for LLM Agents [73.74399447715052]
  We introduce SimpleMem, an efficient memory framework based on semantic lossless compression.<n>We propose a three-stage pipeline designed to maximize information density and token utilization.<n> Experiments on benchmark datasets show that our method consistently outperforms baseline approaches in accuracy, retrieval efficiency, and inference cost.
  arXiv  Detail & Related papers  (2026-01-05T21:02:49Z)
• Vanish into Thin Air: Cross-prompt Universal Adversarial Attacks for SAM2 [48.71856814549096]
  We propose UAP-SAM2, the first cross-prompt universal adversarial attack against SAM2 driven by dual semantic deviation.<n>We show that UAP-SAM2 significantly outperforms state-of-the-art (SOTA) attacks by a large margin.
  arXiv  Detail & Related papers  (2025-10-28T08:59:11Z)
• Promptable Fire Segmentation: Unleashing SAM2's Potential for Real-Time Mobile Deployment with Strategic Bounding Box Guidance [0.0]
  This paper presents the first comprehensive evaluation of SAM2 variants for fire segmentation.<n>We systematically evaluate four SAM2.1 variants (tiny, small, base_plus, large) alongside mobile-oriented variants (TinySAM, MobileSAM) across three fire datasets.<n>Our experimental results demonstrate that bounding box prompts consistently outperform automatic and single point-based approaches.
  arXiv  Detail & Related papers  (2025-10-18T09:15:43Z)
• Seg2Track-SAM2: SAM2-based Multi-object Tracking and Segmentation for Zero-shot Generalization [3.108551551357326]
  Seg2Track-SAM2 is a framework that integrates pre-trained object detectors with SAM2 and a novel Seg2Track module.<n>Seg2Track-SAM2 achieves state-of-the-art (SOTA) performance, ranking fourth overall in both car and pedestrian classes on KITTI MOTS.<n>Results confirm that Seg2Track-SAM2 advances MOTS by combining robust zero-shot tracking, enhanced identity preservation, and efficient memory utilization.
  arXiv  Detail & Related papers  (2025-09-15T10:52:27Z)
• SAMITE: Position Prompted SAM2 with Calibrated Memory for Visual Object Tracking [58.35852822355312]
  Visual Object Tracking (VOT) is widely used in applications like autonomous driving to continuously track targets in videos.<n>To address these issues, some methods propose to adapt the video foundation model SAM2 for VOT, where the tracking results of each frame would be encoded as memory for conditioning the rest of frames in an autoregressive manner.<n>We present a SAMITE model, built upon SAM2 with additional modules, to tackle these challenges.
  arXiv  Detail & Related papers  (2025-07-29T12:11:56Z)
• SAMPa: Sharpness-aware Minimization Parallelized [51.668052890249726]
  Sharpness-aware (SAM) has been shown to improve the generalization of neural networks. Each SAM update requires emphsequentially computing two gradients, effectively doubling the per-iteration cost. We propose a simple modification of SAM, termed SAMPa, which allows us to fully parallelize the two gradient computations.
  arXiv  Detail & Related papers  (2024-10-14T16:21:23Z)
• FocSAM: Delving Deeply into Focused Objects in Segmenting Anything [58.042354516491024]
  The Segment Anything Model (SAM) marks a notable milestone in segmentation models. We propose FocSAM with a pipeline redesigned on two pivotal aspects. First, we propose Dynamic Window Multi-head Self-Attention (Dwin-MSA) to dynamically refocus SAM's image embeddings on the target object. Second, we propose Pixel-wise Dynamic ReLU (P-DyReLU) to enable sufficient integration of interactive information from a few initial clicks.
  arXiv  Detail & Related papers  (2024-05-29T02:34:13Z)
• Stable Segment Anything Model [79.9005670886038]
  The Segment Anything Model (SAM) achieves remarkable promptable segmentation given high-quality prompts. This paper presents the first comprehensive analysis on SAM's segmentation stability across a diverse spectrum of prompt qualities. Our solution, termed Stable-SAM, offers several advantages: 1) improved SAM's segmentation stability across a wide range of prompt qualities, while 2) retaining SAM's powerful promptable segmentation efficiency and generality.
  arXiv  Detail & Related papers  (2023-11-27T12:51:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.