A Deeper Analysis of Block-Sparse Featurizers

• URL: http://arxiv.org/abs/2608.27515v1
• Date: Thu, 27 Aug 2026 09:50:20 GMT
• Title: A Deeper Analysis of Block-Sparse Featurizers
• Abstract summary: Block-sparse featurizer (BSF) is similar to a sparse autoencoder (SAE), but its atomic unit is a small subspace (a block of directions) rather than a single direction.<n>This work studies the BSF's strengths and weaknesses, finding how it still somewhat suffers from classic SAE failure modes, like feature splitting and composition.
• Score: 49.5592634442128
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The recently introduced block-sparse featurizer (BSF; Fel et al., 2026) is similar to a sparse autoencoder (SAE), but its atomic unit is a small subspace (a block of directions) rather than a single direction. It is designed for features that live on low-dimensional manifolds, which are especially frequent in vision. This work studies the BSF's strengths and weaknesses, finding how it still somewhat suffers from classic SAE failure modes, like feature splitting and composition. We propose several architectural changes to the BSF, including a Tournament Top-K selection rule that significantly reduces feature splitting, and we also extend the block paradigm to the crosscoder.

Related papers

• Class-Based Heuristic Selection for Solving the Flying Block Puzzle [0.0]
  Heuristic search underlies planning in autonomous systems ranging from warehouse logistics to robotic navigation.<n>We study this problem through the two-column Flying Block Puzzle, a rigorously NP-complete spatial planning microworld.<n>We introduce the Class-Based Heuristic A* algorithm, which integrates a General Move Constraint to capture minimum displacement costs.
  arXiv  Detail & Related papers  (2026-08-20T09:18:21Z)
• Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation [61.19473093799777]
  Block attention can improve KV cache reuse in long-context scenarios such as Retrieval-Augmented Generation (RAG)<n>However, its broader application is hindered by two key challenges: the difficulty of segmenting input text into meaningful, self-contained blocks, and the inefficiency of existing block fine-tuning methods that risk degrading performance.<n>We propose block distillation, a training framework that is more efficient than block fine-tuning, which uses a frozen full-attention teacher model to guide the block-attention student.
  arXiv  Detail & Related papers  (2026-05-15T12:51:32Z)
• Rethinking Transferable Adversarial Attacks on Point Clouds from a Compact Subspace Perspective [55.919842734983156]
  CoSA is a transferable attack framework that operates within a shared low-dimensional semantic space.<n>CoSA consistently outperforms state-of-the-art transferable attacks.
  arXiv  Detail & Related papers  (2026-01-30T15:48:11Z)
• Block-Attention for Efficient Prefilling [6.916657784681611]
  We introduce Block-attention, an attention mechanism designed to address the increased inference latency and cost in Retrieval-Augmented Generation (RAG) scenarios.<n>By defining each passage as a block, Block-attention enables us to reuse the KV states of passages that have been seen before.<n>Block-attention significantly reduces the time to first token (TTFT) and floating point operations (FLOPs) to a very low level.
  arXiv  Detail & Related papers  (2024-09-14T02:34:26Z)
• Towards Universal Dense Blocking for Entity Resolution [49.06313308481536]
  We propose UniBlocker, a dense blocker that is pre-trained on a domain-independent, easily-obtainable corpus. By conducting domain-independent pre-training, UniBlocker can be adapted to various downstream blocking scenarios without requiring domain-specific fine-tuning. Our experiments show that the proposed UniBlocker, without any domain-specific learning, significantly outperforms previous self- and unsupervised dense blocking methods.
  arXiv  Detail & Related papers  (2024-04-23T08:39:29Z)
• Efficient Motion Modelling with Variable-sized blocks from Hierarchical Cuboidal Partitioning [24.100530697346155]
  Motion modelling with block-based architecture has been widely used in video coding where a frame is divided into fixed-sized blocks that are motion compensated independently. We have investigated the potential of cuboids in motion modelling against the fixed-sized blocks used in scalable video coding.
  arXiv  Detail & Related papers  (2022-08-28T04:13:58Z)
• Stage-Aware Feature Alignment Network for Real-Time Semantic Segmentation of Street Scenes [59.81228011432776]
  We present a novel Stage-aware Feature Alignment Network (SFANet) for real-time semantic segmentation of street scenes. By taking into account the unique role of each stage in the decoder, a novel stage-aware Feature Enhancement Block (FEB) is designed to enhance spatial details and contextual information of feature maps from the encoder. Experimental results show that the proposed SFANet exhibits a good balance between accuracy and speed for real-time semantic segmentation of street scenes.
  arXiv  Detail & Related papers  (2022-03-08T11:46:41Z)
• Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection [89.43987367139724]
  Face forgery detection is raising ever-increasing interest in computer vision. Recent works have reached sound achievements, but there are still unignorable problems. A novel frequency-aware discriminative feature learning framework is proposed in this paper.
  arXiv  Detail & Related papers  (2021-03-16T14:17:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.