SEF-CLGC at SemEval-2026 Task 11: Logical Notation Impact on Language Model Performance

• URL: http://arxiv.org/abs/2606.09157v1
• Date: Mon, 08 Jun 2026 07:47:57 GMT
• Title: SEF-CLGC at SemEval-2026 Task 11: Logical Notation Impact on Language Model Performance
• Abstract summary: We combine formal logical notations with Small Language Models (SLMs) to evaluate reasoning performance on the SemEval-2026 Task 11 Subtask 1: Disentangling Content and Formal Reasoning in Large Language Models.<n>Our experiments show that by relying solely on SLMs, trained on a combination of natural and symbolic languages, our best model achieves a content score of 27.80% on the task while significantly lowering the content bias in reasoning.
• Score: 3.6047642906482142
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper revisits our pipeline called Syllogistic Evaluation Framework-Common Logic Grammar Construction (SEF-CLGC). We combine formal logical notations with Small Language Models (SLMs) to evaluate reasoning performance on the SemEval-2026 Task 11 Subtask 1: Disentangling Content and Formal Reasoning in Large Language Models. Our experiments show that by relying solely on SLMs, trained on a combination of natural and symbolic languages, our best model achieves a content score of 27.80% on the task while significantly lowering the content bias in reasoning.

Related papers

• Are you Talking Logic to Me? Assessing Language Models Syllogistic Reasoning Capabilities [3.6047642906482142]
  Language models (LMs) struggle with logical tasks like reasoning on syllogisms.<n>We study the impact of different formal KR notations on syllogistic reasoning.<n>We propose a syllogistic categorization method (SEF) and use it to enrich ZS prompts with logical definitions.
  arXiv  Detail & Related papers  (2026-07-22T08:29:56Z)
• HABIB_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective Optimization [0.0]
  We present our system for SemEval-2026 Task 11, which evaluates the ability of models to disentangle formal logic from content across 12 languages.<n>We address this challenge using mDeBERTa-v3 networks fine-tuned on a synthetic, rule-based dataset of syllogistic schemes.<n>Our training pipeline employs a multi-objective loss function combining Adaptive Group Distributionally Robust Optimization (DRO), a scheduled differentiable bias penalty, and KL-Divergence consistency regularization.
  arXiv  Detail & Related papers  (2026-07-15T20:24:26Z)
• LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening [69.1037790901185]
  We present LLMEval-Logic, a Chinese logical reasoning benchmark built from realistic situational scenarios.<n>Its pipeline forward-authors and expert-audits natural-language items together with their reference formalizations, verifies annotated answers with Z3, constructs expert rubrics for natural-to-formal grading, and hardens selected items through a closed-loop adversarial workflow.
  arXiv  Detail & Related papers  (2026-05-19T09:40:29Z)
• UFAL-CUNI at SemEval-2026 Task 11: An Efficient Modular Neuro-symbolic Method for Syllogistic Reasoning [2.622018851078979]
  This paper describes our system submitted to SemEval-2026 Task 11: Disentangling Content and Formal Reasoning in Large Language Models.<n>We present an efficient modular neuro-symbolic approach, combining a symbolic prover with small reasoning LLMs.
  arXiv  Detail & Related papers  (2026-05-06T14:10:06Z)
• ITLC at SemEval-2026 Task 11: Normalization and Deterministic Parsing for Formal Reasoning in LLMs [9.363838558599863]
  Large language models suffer from content effects in reasoning tasks, particularly in multi-lingual contexts.<n>We introduce a novel method that reduces these biases through explicit structural abstraction.<n>Our approach achieves top-5 rankings across all subtasks while substantially reducing content effects.
  arXiv  Detail & Related papers  (2026-03-03T07:02:45Z)
• DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models [58.439517684779936]
  This paper proposes a new classical logic benchmark DivLogicEval, consisting of natural sentences composed of diverse statements in a counterintuitive way.<n>To ensure a more reliable evaluation, we also introduce a new evaluation metric that mitigates the influence of bias and randomness inherent in Large Language Models.
  arXiv  Detail & Related papers  (2025-09-19T04:40:46Z)
• LTG at SemEval-2025 Task 10: Optimizing Context for Classification of Narrative Roles [0.3069335774032178]
  We show that a simple entity-oriented approach for context selection can enable text classification using models with limited context window.<n>Our context selection approach and the XLM-RoBERTa language model is on par with, or outperforms, Supervised Fine-Tuning with larger generative language models.
  arXiv  Detail & Related papers  (2025-06-06T10:58:45Z)
• Language Mixing in Reasoning Language Models: Patterns, Impact, and Internal Causes [54.96891982093408]
  Reasoning language models (RLMs) excel at complex tasks by leveraging a chain-of-thought process to generate structured intermediate steps.<n> Language mixing, i.e., reasoning steps containing tokens from languages other than the prompt, has been observed in their outputs and shown to affect performance.<n>We present the first systematic study of language mixing in RLMs, examining its patterns, impact, and internal causes across 15 languages.
  arXiv  Detail & Related papers  (2025-05-20T18:26:53Z)
• LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models [63.14196038655506]
  We introduce LogicAsker, a novel approach for evaluating and enhancing the logical reasoning capabilities of large language models (LLMs) Our methodology reveals significant gaps in LLMs' learning of logical rules, with identified reasoning failures ranging from 29% to 90% across different models. We leverage these findings to construct targeted demonstration examples and fine-tune data, notably enhancing logical reasoning in models like GPT-4o by up to 5%.
  arXiv  Detail & Related papers  (2024-01-01T13:53:53Z)
• CLOMO: Counterfactual Logical Modification with Large Language Models [109.60793869938534]
  We introduce a novel task, Counterfactual Logical Modification (CLOMO), and a high-quality human-annotated benchmark. In this task, LLMs must adeptly alter a given argumentative text to uphold a predetermined logical relationship. We propose an innovative evaluation metric, the Self-Evaluation Score (SES), to directly evaluate the natural language output of LLMs.
  arXiv  Detail & Related papers  (2023-11-29T08:29:54Z)
• Text Classification via Large Language Models [63.1874290788797]
  We introduce Clue And Reasoning Prompting (CARP) to address complex linguistic phenomena involved in text classification. Remarkably, CARP yields new SOTA performances on 4 out of 5 widely-used text-classification benchmarks. More importantly, we find that CARP delivers impressive abilities on low-resource and domain-adaptation setups.
  arXiv  Detail & Related papers  (2023-05-15T06:24:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.