Empower Nested Boolean Logic via Self-Supervised Curriculum Learning

• URL: http://arxiv.org/abs/2310.05450v2
• Date: Sun, 31 Dec 2023 04:20:00 GMT
• Title: Empower Nested Boolean Logic via Self-Supervised Curriculum Learning
• Authors: Hongqiu Wu, Linfeng Liu, Hai Zhao, Min Zhang
• Abstract summary: We find that any pre-trained language models even including large language models only behave like a random selector in the face of multi-nested logic. To empower language models with this fundamental capability, this paper proposes a new self-supervised learning method textitCurriculum Logical Reasoning (textscClr)
• Score: 67.46052028752327
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Beyond the great cognitive powers showcased by language models, it is crucial to scrutinize whether their reasoning capabilities stem from strong generalization or merely exposure to relevant data. As opposed to constructing increasingly complex logic, this paper probes into the boolean logic, the root capability of a logical reasoner. We find that any pre-trained language models even including large language models only behave like a random selector in the face of multi-nested boolean logic, a task that humans can handle with ease. To empower language models with this fundamental capability, this paper proposes a new self-supervised learning method \textit{Curriculum Logical Reasoning} (\textsc{Clr}), where we augment the training data with nested boolean logic chain step-by-step, and program the training from simpler logical patterns gradually to harder ones. This new training paradigm allows language models to effectively generalize to much harder and longer-hop logic, which can hardly be learned through naive training. Furthermore, we show that boolean logic is a great foundation for improving the subsequent general logical tasks.

Related papers

• BoolQuestions: Does Dense Retrieval Understand Boolean Logic in Language? [88.29075896295357]
  We first investigate whether current retrieval systems can comprehend the Boolean logic implied in language. Through extensive experimental results, we draw the conclusion that current dense retrieval systems do not fully understand Boolean logic in language. We propose a contrastive continual training method that serves as a strong baseline for the research community.
  arXiv  Detail & Related papers  (2024-11-19T05:19:53Z)
• Improving Complex Reasoning over Knowledge Graph with Logic-Aware Curriculum Tuning [89.89857766491475]
  We propose a complex reasoning schema over KG upon large language models (LLMs) We augment the arbitrary first-order logical queries via binary tree decomposition to stimulate the reasoning capability of LLMs. Experiments across widely used datasets demonstrate that LACT has substantial improvements(brings an average +5.5% MRR score) over advanced methods.
  arXiv  Detail & Related papers  (2024-05-02T18:12:08Z)
• Language Models can be Logical Solvers [99.40649402395725]
  We introduce LoGiPT, a novel language model that directly emulates the reasoning processes of logical solvers. LoGiPT is fine-tuned on a newly constructed instruction-tuning dataset derived from revealing and refining the invisible reasoning process of deductive solvers.
  arXiv  Detail & Related papers  (2023-11-10T16:23:50Z)
• Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logic [19.476840373850653]
  Large language models show hallucinations as their reasoning procedures are unconstrained by logical principles. We propose LoT (Logical Thoughts), a self-improvement prompting framework that leverages principles rooted in symbolic logic. Experimental evaluations conducted on language tasks in diverse domains, including arithmetic, commonsense, symbolic, causal inference, and social problems, demonstrate the efficacy of enhanced reasoning by logic.
  arXiv  Detail & Related papers  (2023-09-23T11:21:12Z)
• Planning with Logical Graph-based Language Model for Instruction Generation [9.70880913062245]
  We propose a graph-based language model, Logical-GLM, to infuse logic into language models. We generate logical skeletons to guide language model training, infusing domain knowledge into language models. Our approach can generate instructional texts with more correct logic owing to the internalized domain knowledge.
  arXiv  Detail & Related papers  (2023-08-26T06:28:14Z)
• Learning Language Representations with Logical Inductive Bias [19.842271716111153]
  We explore a new logical inductive bias for better language representation learning. We develop a novel neural architecture named FOLNet to encode this new inductive bias. We find that the self-attention module in transformers can be composed by two of our neural logic operators.
  arXiv  Detail & Related papers  (2023-02-19T02:21:32Z)
• APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning [73.3035118224719]
  We propose APOLLO, an adaptively pretrained language model that has improved logical reasoning abilities. APOLLO performs comparably on ReClor and outperforms baselines on LogiQA.
  arXiv  Detail & Related papers  (2022-12-19T07:40:02Z)
• LogiGAN: Learning Logical Reasoning via Adversarial Pre-training [58.11043285534766]
  We present LogiGAN, an unsupervised adversarial pre-training framework for improving logical reasoning abilities of language models. Inspired by the facilitation effect of reflective thinking in human learning, we simulate the learning-thinking process with an adversarial Generator-Verifier architecture. Both base and large size language models pre-trained with LogiGAN demonstrate obvious performance improvement on 12 datasets.
  arXiv  Detail & Related papers  (2022-05-18T08:46:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.