HERO: A Heterogeneity-Aware Benchmark Library for Federated Continual Learning

• URL: http://arxiv.org/abs/2607.08784v1
• Date: Sat, 13 Jun 2026 10:32:34 GMT
• Title: HERO: A Heterogeneity-Aware Benchmark Library for Federated Continual Learning
• Abstract summary: HERO builds benchmark streams by separating three choices that are often coupled, namely the task split, the client data split, and the client task sequence.<n>We evaluate representative FCL methods on CIFAR-100 and TinyImageNet using final average accuracy, average forgetting, and bottom-10% client accuracy.
• Score: 7.256466895987238
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated continual learning (FCL) evaluates how distributed clients learn from changing data streams while retaining previously learned knowledge. Existing evaluations are difficult to compare because they often change datasets, task splits, client data splits, task orders, backbones, memory assumptions, and reporting rules simultaneously. We introduce \textbf{HERO}, a heterogeneity-aware benchmark library for FCL. HERO builds benchmark streams by separating three choices that are often coupled, namely the task split, the client data split, and the client task sequence. In HERO-Core, the main comparable benchmark, $α$ controls client data skew and $ρ$ controls task-order mismatch. We evaluate representative FCL methods on CIFAR-100 and TinyImageNet using final average accuracy, average forgetting, and bottom-10\% client accuracy. We also include a graph-based Domain-IL portability case study on OGB-MolPCBA, where scaffold-domain granularity changes the input distribution while the prediction task remains fixed. Our results show that method behavior changes across easy and heterogeneous settings, that average accuracy can hide weak bottom-client performance, that task-order mismatch favors different strategies from synchronized evaluation, and that the same HERO interface can expose domain-shift difficulty beyond image-based FCIL. HERO releases benchmark streams, configurations, method implementations, and reporting scripts to support reproducible and setting-aware FCL evaluation.

Related papers

• CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learning [3.2847672479922783]
  We propose a class-wise reliability-Aware Distillation (CRAD) for federated learning.<n>CRAD weights each teacher by its per-class reliability (precision, or inverse variance)<n>It consistently outperforms competing methods in global accuracy.
  arXiv  Detail & Related papers  (2026-08-31T22:36:30Z)
• Disagree to Explore, Agree to Commit: Routing-Guided Test-Time Scaling for Software Agents [60.650808962786364]
  We study whether native MoE router traces can guide steering and selection without an external judge or selection-time test execution.<n>Our analysis shows that routing provides a robust behavioral role signal; token-granular readouts and decision-matched comparison sets turn it into effective control.<n>We evaluate on SWE-bench Verified using open-weight sparse MoE agents across scales and reasoning-effort settings.
  arXiv  Detail & Related papers  (2026-08-23T03:07:03Z)
• Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining [13.114768589139308]
  Explicit skill libraries make computer-using agents easier to inspect, but it remains unclear whether such libraries can be mined from interaction data in a way that improves downstream policies.<n>We study this question through a three-stage pipeline that segments GUI trajectories, clusters segments into candidate skills, and trains a skill-aware policy from the resulting annotations.<n>We present the method as a diagnostic study: trajectory mining can expose inspectable skill structure, but the current boundary detector, orderless segment representation, and offline reward model are insufficient for reliable cross-domain policy improvement.
  arXiv  Detail & Related papers  (2026-06-18T15:25:42Z)
• Beyond the Reranker: Do RAG Retrieval Enhancements Help Once a Strong Reranker Is Present? [3.77661940418406]
  Methods for query expansion, hierarchical and cross-document summarization, graph-based expansion, per-query routing, rank fusion, and corrective re-retrieval are studied.<n>We show that a strong cross-encoder reranker accounts for most of the pipeline's quality.<n>The remaining reranking and pool-expansion methods in common use, among them hierarchical summarization, graph expansion, routing, and rank fusion, give no reliable gain once that reranker is present.
  arXiv  Detail & Related papers  (2026-06-14T20:50:41Z)
• Never Seen Before: Benchmarking Genuine Zero-Shot Composed Image Retrieval with Consistent Video-Sourced Datasets [61.420656457977195]
  Zero-Shot Composed Image Retrieval (ZS-CIR) aims to retrieve a target image based on a query composed of a reference image and a relative caption without training samples.<n>Existing ZS-CIR datasets often suffer from complete irrelevance between reference and target images due to noisy image sources.<n>We introduce ZeroSight, a novel benchmark for ZS-CIR.
  arXiv  Detail & Related papers  (2026-06-05T08:23:25Z)
• HASA: Subnet Allocation for Compute-Constrained Model-Heterogeneous Federated Learning [6.169964580735435]
  Edge services increasingly use federated learning to personalize on-device models while keeping sensitive data local.<n>In practice, deployments must handle in both client resources and local data distributions.<n>This paper proposes Heterogeneity-Aware Subnet Allocation (HASA), a train-only rule that assigns widths based on client heterogeneity scores.
  arXiv  Detail & Related papers  (2026-05-30T11:21:48Z)
• Scaling Laws for Agent Harnesses via Effective Feedback Compute [53.68149869349268]
  emphEffective Feedback Compute (EFC) is a trace-level scaling coordinate that credits feedback only when it is informative, valid, non-redundant, and retained for subsequent decisions.<n>EFC-based coordinates consistently predict failure rates better than raw-compute baselines.
  arXiv  Detail & Related papers  (2026-05-28T09:45:47Z)
• Exposing and Addressing Cross-Task Inconsistency in Unified Vision-Language Models [80.23791222509644]
  Inconsistent AI models are considered brittle and untrustworthy by human users. We find that state-of-the-art vision-language models suffer from a surprisingly high degree of inconsistent behavior across tasks. We propose a rank correlation-based auxiliary training objective, computed over large automatically created cross-task contrast sets.
  arXiv  Detail & Related papers  (2023-03-28T16:57:12Z)
• FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Data [8.677832361022809]
  Federated learning allows multiple clients to collaboratively train a model without exchanging their data. Common solutions involve an auxiliary loss to regularize weight divergence or feature inconsistency during local training. We propose a novel framework named Federated learning with Feature Anchors (FedFA)
  arXiv  Detail & Related papers  (2022-11-17T02:27:44Z)
• Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspaces [59.33965805898736]
  Clustered learning has been shown to produce promising results by grouping clients into clusters. Existing FL algorithms are essentially trying to group clients together with similar distributions. Prior FL algorithms attempt similarities indirectly during training.
  arXiv  Detail & Related papers  (2022-09-21T17:37:54Z)
• Federated Multi-Target Domain Adaptation [99.93375364579484]
  Federated learning methods enable us to train machine learning models on distributed user data while preserving its privacy. We consider a more practical scenario where the distributed client data is unlabeled, and a centralized labeled dataset is available on the server. We propose an effective DualAdapt method to address the new challenges.
  arXiv  Detail & Related papers  (2021-08-17T17:53:05Z)
• Learning Stable Classifiers by Transferring Unstable Features [59.06169363181417]
  We study transfer learning in the presence of spurious correlations. We experimentally demonstrate that directly transferring the stable feature extractor learned on the source task may not eliminate these biases for the target task. We hypothesize that the unstable features in the source task and those in the target task are directly related.
  arXiv  Detail & Related papers  (2021-06-15T02:41:12Z)
• Byzantine-Resilient High-Dimensional Federated Learning [10.965065178451104]
  We employ gradientCV descent (SGD) with local iterations in the presence of malicious/Byzantine clients. The clients update iteration by taking their own subset of gradient vectors then communicate with the net. We believe that our algorithm is the first Byzantine-resilient algorithm and analysis with local datasets.
  arXiv  Detail & Related papers  (2020-06-22T03:54:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.