AFABench: A Generic Framework for Benchmarking Active Feature Acquisition

• URL: http://arxiv.org/abs/2508.14734v1
• Date: Wed, 20 Aug 2025 14:29:16 GMT
• Title: AFABench: A Generic Framework for Benchmarking Active Feature Acquisition
• Authors: Valter Schütz, Han Wu, Reza Rezvan, Linus Aronsson, Morteza Haghir Chehreghani,
• Abstract summary: We introduce AFABench, the first benchmark framework for Active Feature Acquisition.<n>We implement and evaluate representative algorithms from all major categories, including static, greedy, and reinforcement learning-based approaches.<n>Our results highlight key trade-offs between different AFA strategies and provide actionable insights for future research.
• Score: 6.922744987645169
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In many real-world scenarios, acquiring all features of a data instance can be expensive or impractical due to monetary cost, latency, or privacy concerns. Active Feature Acquisition (AFA) addresses this challenge by dynamically selecting a subset of informative features for each data instance, trading predictive performance against acquisition cost. While numerous methods have been proposed for AFA, ranging from greedy information-theoretic strategies to non-myopic reinforcement learning approaches, fair and systematic evaluation of these methods has been hindered by the lack of standardized benchmarks. In this paper, we introduce AFABench, the first benchmark framework for AFA. Our benchmark includes a diverse set of synthetic and real-world datasets, supports a wide range of acquisition policies, and provides a modular design that enables easy integration of new methods and tasks. We implement and evaluate representative algorithms from all major categories, including static, greedy, and reinforcement learning-based approaches. To test the lookahead capabilities of AFA policies, we introduce a novel synthetic dataset, AFAContext, designed to expose the limitations of greedy selection. Our results highlight key trade-offs between different AFA strategies and provide actionable insights for future research. The benchmark code is available at: https://github.com/Linusaronsson/AFA-Benchmark.

Related papers

• Let the Barbarians In: How AI Can Accelerate Systems Performance Research [80.43506848683633]
  We term this iterative cycle of generation, evaluation, and refinement AI-Driven Research for Systems.<n>We demonstrate that ADRS-generated solutions can match or even outperform human state-of-the-art designs.
  arXiv  Detail & Related papers  (2025-12-16T18:51:23Z)
• How to Purchase Labels? A Cost-Effective Approach Using Active Learning Markets [2.7429630700600893]
  We introduce and analyse active learning markets as a way to purchase labels.<n>This comes in contrast to the many proposals that already exist to purchase features and examples.<n>We focus on a single-buyer-multiple-seller setup and propose the use of two active learning strategies.
  arXiv  Detail & Related papers  (2025-11-25T18:34:33Z)
• Information Templates: A New Paradigm for Intelligent Active Feature Acquisition [4.607145155913716]
  Active feature acquisition (AFA) is a computation-adaptive instance paradigm in which, at test time, a policy sequentially chooses which features to acquire.<n>We propose a non-greedy framework that learns a small library of feature templates and uses this library to guide the next feature acquisitions.
  arXiv  Detail & Related papers  (2025-08-25T18:15:11Z)
• Adapting Vision-Language Models Without Labels: A Comprehensive Survey [74.17944178027015]
  Vision-Language Models (VLMs) have demonstrated remarkable generalization capabilities across a wide range of tasks.<n>Recent research has increasingly focused on unsupervised adaptation methods that do not rely on labeled data.<n>We propose a taxonomy based on the availability and nature of unlabeled visual data, categorizing existing approaches into four key paradigms.
  arXiv  Detail & Related papers  (2025-08-07T16:27:37Z)
• Stochastic Encodings for Active Feature Acquisition [100.47043816019888]
  Active Feature Acquisition is an instance-wise, sequential decision making problem.<n>The aim is to dynamically select which feature to measure based on current observations, independently for each test instance.<n>Common approaches either use Reinforcement Learning, which experiences training difficulties, or greedily maximize the conditional mutual information of the label and unobserved features, which makes myopic.<n>We introduce a latent variable model, trained in a supervised manner. Acquisitions are made by reasoning about the features across many possible unobserved realizations in a latent space.
  arXiv  Detail & Related papers  (2025-08-03T23:48:46Z)
• Prompt Smart, Pay Less: Cost-Aware APO for Real-World Applications [1.3312007032203859]
  We introduce APE-OPRO, a novel hybrid framework that combines the complementary strengths of APE and OPRO.<n>We benchmark APE-OPRO alongside both gradient-free (APE, OPRO) and gradient-based (ProTeGi) methods on a dataset of 2,500 labeled products.<n>Our results highlight key trade-offs: ProTeGi offers the strongest absolute performance at lower API cost but higher computational time as noted inciteprotegi.
  arXiv  Detail & Related papers  (2025-07-18T21:46:15Z)
• Alice Benchmarks: Connecting Real World Re-Identification with the Synthetic [92.02220105679713]
  We introduce the Alice benchmarks, large-scale datasets providing benchmarks and evaluation protocols to the research community. Within the Alice benchmarks, two object re-ID tasks are offered: person and vehicle re-ID. As an important feature of our real target, the clusterability of its training set is not manually guaranteed to make it closer to a real domain adaptation test scenario.
  arXiv  Detail & Related papers  (2023-10-06T17:58:26Z)
• Domain-Expanded ASTE: Rethinking Generalization in Aspect Sentiment Triplet Extraction [67.54420015049732]
  Aspect Sentiment Triplet Extraction (ASTE) is a challenging task in sentiment analysis, aiming to provide fine-grained insights into human sentiments. Existing benchmarks are limited to two domains and do not evaluate model performance on unseen domains. We introduce a domain-expanded benchmark by annotating samples from diverse domains, enabling evaluation of models in both in-domain and out-of-domain settings.
  arXiv  Detail & Related papers  (2023-05-23T18:01:49Z)
• Acquisition Conditioned Oracle for Nongreedy Active Feature Acquisition [16.350351668269415]
  We develop methodology for active feature acquisition (AFA) We show that we can bypass many challenges with a novel, nonparametric oracle based approach.
  arXiv  Detail & Related papers  (2023-02-27T17:02:11Z)
• Deep Active Ensemble Sampling For Image Classification [8.31483061185317]
  Active learning frameworks aim to reduce the cost of data annotation by actively requesting the labeling for the most informative data points. Some proposed approaches include uncertainty-based techniques, geometric methods, implicit combination of uncertainty-based and geometric approaches. We present an innovative integration of recent progress in both uncertainty-based and geometric frameworks to enable an efficient exploration/exploitation trade-off in sample selection strategy. Our framework provides two advantages: (1) accurate posterior estimation, and (2) tune-able trade-off between computational overhead and higher accuracy.
  arXiv  Detail & Related papers  (2022-10-11T20:20:20Z)
• Single-Modal Entropy based Active Learning for Visual Question Answering [75.1682163844354]
  We address Active Learning in the multi-modal setting of Visual Question Answering (VQA) In light of the multi-modal inputs, image and question, we propose a novel method for effective sample acquisition. Our novel idea is simple to implement, cost-efficient, and readily adaptable to other multi-modal tasks.
  arXiv  Detail & Related papers  (2021-10-21T05:38:45Z)
• Towards Robust Active Feature Acquisition [14.785570635390744]
  Active feature acquisition (AFA) models deal with a small set of candidate features and have difficulty scaling to a large feature space. We propose several techniques to advance the current AFA approaches. Our framework can easily handle a large number of features using a hierarchical acquisition policy and is more robust to OOD inputs with the help of an OOD detector for partially observed data.
  arXiv  Detail & Related papers  (2021-07-09T01:06:13Z)
• Semi-Supervised Domain Generalization with Stochastic StyleMatch [90.98288822165482]
  In real-world applications, we might have only a few labels available from each source domain due to high annotation cost. In this work, we investigate semi-supervised domain generalization, a more realistic and practical setting. Our proposed approach, StyleMatch, is inspired by FixMatch, a state-of-the-art semi-supervised learning method based on pseudo-labeling.
  arXiv  Detail & Related papers  (2021-06-01T16:00:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.