Boolean-aware Attention for Dense Retrieval

• URL: http://arxiv.org/abs/2503.01753v1
• Date: Mon, 03 Mar 2025 17:23:08 GMT
• Title: Boolean-aware Attention for Dense Retrieval
• Authors: Quan Mai, Susan Gauch, Douglas Adams,
• Abstract summary: We present a novel attention mechanism that adjusts token focus based on Boolean operators (e.g., and, or, not)<n>Our model employs specialized Boolean experts, each tailored to amplify or suppress attention for operator-specific contexts.
• Score: 0.8192907805418583
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Boolean-aware attention, a novel attention mechanism that dynamically adjusts token focus based on Boolean operators (e.g., and, or, not). Our model employs specialized Boolean experts, each tailored to amplify or suppress attention for operator-specific contexts. A predefined gating mechanism activates the corresponding experts based on the detected Boolean type. Experiments on Boolean retrieval datasets demonstrate that integrating BoolAttn with BERT greatly enhances the model's capability to process Boolean queries.

Related papers

• Transformers Learn Faster with Semantic Focus [57.97235825738412]
  We study sparse transformers in terms of learnability and generalization.<n>We find that input-dependent sparse attention models appear to converge faster and generalize better than standard attention models.
  arXiv  Detail & Related papers  (2025-06-17T01:19:28Z)
• Practical Boolean Backpropagation [0.0]
  We present a practical method for purely Boolean backpropagation for networks based on a single specific gate we chose.<n>Initial experiments confirm its feasibility.
  arXiv  Detail & Related papers  (2025-05-01T12:50:02Z)
• CBW: Towards Dataset Ownership Verification for Speaker Verification via Clustering-based Backdoor Watermarking [85.68235482145091]
  Large-scale speech datasets have become valuable intellectual property. We propose a novel dataset ownership verification method. Our approach introduces a clustering-based backdoor watermark (CBW) We conduct extensive experiments on benchmark datasets, verifying the effectiveness and robustness of our method against potential adaptive attacks.
  arXiv  Detail & Related papers  (2025-03-02T02:02:57Z)
• BoolQuestions: Does Dense Retrieval Understand Boolean Logic in Language? [88.29075896295357]
  We first investigate whether current retrieval systems can comprehend the Boolean logic implied in language. Through extensive experimental results, we draw the conclusion that current dense retrieval systems do not fully understand Boolean logic in language. We propose a contrastive continual training method that serves as a strong baseline for the research community.
  arXiv  Detail & Related papers  (2024-11-19T05:19:53Z)
• Boolean Logic as an Error feedback mechanism [0.5439020425819]
  The notion of Boolean logic backpagation was introduced to build neural networks with weights and activations being Boolean numbers. Most of computations can be done with logic instead of real arithmetic during training and phases.
  arXiv  Detail & Related papers  (2024-01-29T18:56:21Z)
• Boolean Variation and Boolean Logic BackPropagation [0.0]
  The notion of variation is introduced for the Boolean set and based on which Boolean logic backpropagation principle is developed. Deep models can be built with weights and activations being Boolean numbers and operated with Boolean logic instead of real arithmetic.
  arXiv  Detail & Related papers  (2023-11-13T16:01:43Z)
• LF-checker: Machine Learning Acceleration of Bounded Model Checking for Concurrency Verification (Competition Contribution) [15.600922269515756]
  We describe and evaluate LF-checker, a metaverifier tool based on machine learning. It extracts multiple features of the program under test and predicts the optimal configuration of a bounded model checker with a decision tree.
  arXiv  Detail & Related papers  (2023-01-22T15:46:57Z)
• Oracle Operators for Non-Boolean Functions [0.0]
  We present a construction of a general oracle operator for a real-valued function on the Boolean cube. We use such operators in Shyamsundar's Non-Boolean Amplitude Amplification [arXiv:2102.04975] to solve binary optimization problems with a non-adiabatic algorithm.
  arXiv  Detail & Related papers  (2022-12-07T19:58:11Z)
• Bayesian Bellman Operators [55.959376449737405]
  We introduce a novel perspective on Bayesian reinforcement learning (RL) Our framework is motivated by the insight that when bootstrapping is introduced, model-free approaches actually infer a posterior over Bellman operators, not value functions.
  arXiv  Detail & Related papers  (2021-06-09T12:20:46Z)
• SparseBERT: Rethinking the Importance Analysis in Self-attention [107.68072039537311]
  Transformer-based models are popular for natural language processing (NLP) tasks due to its powerful capacity. Attention map visualization of a pre-trained model is one direct method for understanding self-attention mechanism. We propose a Differentiable Attention Mask (DAM) algorithm, which can be also applied in guidance of SparseBERT design.
  arXiv  Detail & Related papers  (2021-02-25T14:13:44Z)
• Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels [51.66448070984615]
  We show that maximizing training accuracy on sufficiently many noisy samples yields an approximately optimal classifier. For validation, we prove that a noisy validation set is reliable, addressing the critical demand of model selection. We show characterizations of models trained with noisy labels, motivated by our theoretical results, and verify the utility of a noisy validation set.
  arXiv  Detail & Related papers  (2020-12-08T03:37:47Z)
• SelfORE: Self-supervised Relational Feature Learning for Open Relation Extraction [60.08464995629325]
  Open-domain relation extraction is the task of extracting open-domain relation facts from natural language sentences. We proposed a self-supervised framework named SelfORE, which exploits weak, self-supervised signals. Experimental results on three datasets show the effectiveness and robustness of SelfORE.
  arXiv  Detail & Related papers  (2020-04-06T07:23:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.