When Is Prior Knowledge Helpful? Exploring the Evaluation and Selection of Unsupervised Pretext Tasks from a Neuro-Symbolic Perspective

• URL: http://arxiv.org/abs/2508.07299v1
• Date: Sun, 10 Aug 2025 11:23:36 GMT
• Title: When Is Prior Knowledge Helpful? Exploring the Evaluation and Selection of Unsupervised Pretext Tasks from a Neuro-Symbolic Perspective
• Authors: Lin-Han Jia, Si-Yu Han, Wen-Chao Hu, Jie-Jing Shao, Wen-Da Wei, Zhi Zhou, Lan-Zhe Guo, Yu-Feng Li,
• Abstract summary: We extend the Nesy theory based on reliable knowledge to the scenario of unreliable knowledge.<n>We propose schemes to operationalize these theoretical metrics, and thereby develop a method that can predict the effectiveness of pretext tasks in advance.
• Score: 45.419765404078724
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neuro-symbolic (Nesy) learning improves the target task performance of models by enabling them to satisfy knowledge, while semi/self-supervised learning (SSL) improves the target task performance by designing unsupervised pretext tasks for unlabeled data to make models satisfy corresponding assumptions. We extend the Nesy theory based on reliable knowledge to the scenario of unreliable knowledge (i.e., assumptions), thereby unifying the theoretical frameworks of SSL and Nesy. Through rigorous theoretical analysis, we demonstrate that, in theory, the impact of pretext tasks on target performance hinges on three factors: knowledge learnability with respect to the model, knowledge reliability with respect to the data, and knowledge completeness with respect to the target. We further propose schemes to operationalize these theoretical metrics, and thereby develop a method that can predict the effectiveness of pretext tasks in advance. This will change the current status quo in practical applications, where the selections of unsupervised tasks are heuristic-based rather than theory-based, and it is difficult to evaluate the rationality of unsupervised pretext task selection before testing the model on the target task. In experiments, we verify a high correlation between the predicted performance-estimated using minimal data-and the actual performance achieved after large-scale semi-supervised or self-supervised learning, thus confirming the validity of the theory and the effectiveness of the evaluation method.

Related papers

• Capabilities Ain't All You Need: Measuring Propensities in AI [32.960519634809145]
  We introduce the first formal framework for measuring AI propensities by using a bilogistic formulation for model success.<n>We find that we can measure how much the propensity is shifted and what effect this has on the tasks.<n>We obtain stronger predictive power when combining propensities and capabilities than either separately.
  arXiv  Detail & Related papers  (2026-02-20T12:40:18Z)
• Theory Trace Card: Theory-Driven Socio-Cognitive Evaluation of LLMs [2.98033672654447]
  We argue that many socio-cognitive evaluations proceed without an explicit theoretical specification of the target capability.<n>Without this theoretical grounding, benchmarks that exercise only narrow subsets of a capability are routinely misinterpreted as evidence of broad competence.<n>We introduce the Trace Card, a lightweight documentation artifact designed to accompany socio-cognitive evaluations.
  arXiv  Detail & Related papers  (2026-01-05T08:06:50Z)
• Safety Through Reasoning: An Empirical Study of Reasoning Guardrail Models [3.102576158218633]
  Reasoning-based language models have demonstrated strong performance across various domains.<n>Recent research has shown that reasoning also offers significant benefits for safety and guardrail applications.<n>Our study focuses on two key dimensions: data efficiency and inference efficiency.
  arXiv  Detail & Related papers  (2025-05-26T15:01:37Z)
• Spurious Forgetting in Continual Learning of Language Models [20.0936011355535]
  Recent advancements in large language models (LLMs) reveal a perplexing phenomenon in continual learning.<n>Despite extensive training, models experience significant performance declines.<n>This study proposes that such performance drops often reflect a decline in task alignment rather than true knowledge loss.
  arXiv  Detail & Related papers  (2025-01-23T08:09:54Z)
• Model Predictive Task Sampling for Efficient and Robust Adaptation [57.414812940406996]
  We introduce Model Predictive Task Sampling (MPTS), a framework that bridges the task space and adaptation risk distributions.<n>MPTS employs a generative model to characterize the episodic optimization process and predicts task-specific adaptation risk via posterior inference.<n>MPTS seamlessly integrates into zero-shot, few-shot, and supervised finetuning settings.
  arXiv  Detail & Related papers  (2025-01-19T13:14:53Z)
• Understanding Uncertainty-based Active Learning Under Model Mismatch [16.361254095103615]
  Uncertainty-based Active Learning (UAL) operates by querying the label(s) of pivotal samples from an unlabeled pool selected based on the prediction uncertainty. The efficacy of UAL depends on the model capacity as well as the adopted uncertainty-based acquisition function.
  arXiv  Detail & Related papers  (2024-08-24T23:37:08Z)
• Learning Latent Graph Structures and their Uncertainty [63.95971478893842]
  We show that minimizing point-prediction losses does not guarantee proper learning of latent relational information.<n>We propose a sampling-based method that solves this joint learning task.
  arXiv  Detail & Related papers  (2024-05-30T10:49:22Z)
• Uncertainty for Active Learning on Graphs [70.44714133412592]
  Uncertainty Sampling is an Active Learning strategy that aims to improve the data efficiency of machine learning models.<n>We benchmark Uncertainty Sampling beyond predictive uncertainty and highlight a significant performance gap to other Active Learning strategies.<n>We develop ground-truth Bayesian uncertainty estimates in terms of the data generating process and prove their effectiveness in guiding Uncertainty Sampling toward optimal queries.
  arXiv  Detail & Related papers  (2024-05-02T16:50:47Z)
• Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality [55.88910947643436]
  Self-supervised pre-training is essential for handling vast quantities of unlabeled data in practice. HiDe-Prompt is an innovative approach that explicitly optimize the hierarchical components with an ensemble of task-specific prompts and statistics. Our experiments demonstrate the superior performance of HiDe-Prompt and its robustness to pre-training paradigms in continual learning.
  arXiv  Detail & Related papers  (2023-10-11T06:51:46Z)
• Distortion Resilience for Goal-Oriented Semantic Communication [12.459448669650683]
  This work introduces an innovative approach that leverages the rate distortion theory to analyze distortions induced by communication and compression.<n>We can preemptively estimate the empirical accuracy of AI tasks, making the goal-oriented SemCom problem feasible.<n>The experimental results indicate that our proposed method enables accurate AI task performance while adhering to network constraints.
  arXiv  Detail & Related papers  (2023-09-26T00:26:29Z)
• Knowing when we do not know: Bayesian continual learning for sensing-based analysis tasks [8.201216572526302]
  We propose a Bayesian inference based framework to continually learn a set of real-world, sensing-based analysis tasks. Our experiments prove the robustness and reliability of the learned models to adapt to the changing sensing environment.
  arXiv  Detail & Related papers  (2021-06-06T13:45:06Z)
• Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering [80.60605604261416]
  We propose a novel neuro-symbolic framework for zero-shot question answering across commonsense tasks. We vary the set of language models, training regimes, knowledge sources, and data generation strategies, and measure their impact across tasks. We show that, while an individual knowledge graph is better suited for specific tasks, a global knowledge graph brings consistent gains across different tasks.
  arXiv  Detail & Related papers  (2020-11-07T22:52:21Z)
• Goal-Aware Prediction: Learning to Model What Matters [105.43098326577434]
  One of the fundamental challenges in using a learned forward dynamics model is the mismatch between the objective of the learned model and that of the downstream planner or policy. We propose to direct prediction towards task relevant information, enabling the model to be aware of the current task and encouraging it to only model relevant quantities of the state space. We find that our method more effectively models the relevant parts of the scene conditioned on the goal, and as a result outperforms standard task-agnostic dynamics models and model-free reinforcement learning.
  arXiv  Detail & Related papers  (2020-07-14T16:42:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.