Investigating More Explainable and Partition-Free Compositionality Estimation for LLMs: A Rule-Generation Perspective

• URL: http://arxiv.org/abs/2604.27340v1
• Date: Thu, 30 Apr 2026 02:33:13 GMT
• Title: Investigating More Explainable and Partition-Free Compositionality Estimation for LLMs: A Rule-Generation Perspective
• Abstract summary: We propose a novel rule-generation perspective for compositionality estimation for LLMs.<n>It requires LLMs to generate a program as rules for dataset mapping and provides estimates of the compositionality of LLMs using complexity-based theory.<n>We conduct experiments and analysis of existing advanced LLMs based on this perspective on a string-to-grid task, and find various compositionality characterizations and compositionality deficiencies exhibited by LLMs.
• Score: 27.14336518124242
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Compositional generalization tests are often used to estimate the compositionality of LLMs. However, such tests have the following limitations: (1) they only focus on the output results without considering LLMs' understanding of sample compositionality, resulting in explainability defects; (2) they rely on dataset partition to form the test set with combinations unseen in the training set, suffering from combination leakage issues. In this work, we propose a novel rule-generation perspective for compositionality estimation for LLMs. It requires LLMs to generate a program as rules for dataset mapping and provides estimates of the compositionality of LLMs using complexity-based theory. The perspective addresses the limitations of compositional generalization tests and provides a new way to analyze the compositionality characterization of LLMs. We conduct experiments and analysis of existing advanced LLMs based on this perspective on a string-to-grid task, and find various compositionality characterizations and compositionality deficiencies exhibited by LLMs.

Related papers

• Deliberate Evolution: Agentic Reasoning for Sample-Efficient Symbolic Regression with LLMs [52.89778838903305]
  Deliberate Evolution (DE) is an agentic framework that decouples symbolic generation from search control.<n>Experiments on LLM-SRBench show that DE consistently outperforms representative LLM-based SR baselines across diverse scientific domains.
  arXiv  Detail & Related papers  (2026-06-03T02:22:16Z)
• Iterative In-Context Learning to Enhance LLMs Abstract Reasoning: The Case-Study of Algebraic Tasks [40.48180253367968]
  We introduce an in-context learning methodology that improves the generalization capabilities of general purpose LLMs.<n>Our approach employs an iterative example selection strategy, which incrementally constructs a tailored set of few-shot examples.<n>Our experiments reveal that some LLMs achieve better generalization performances when prompted with simpler few-shot examples.
  arXiv  Detail & Related papers  (2025-09-01T08:54:45Z)
• An Empirical Study of Conformal Prediction in LLM with ASP Scaffolds for Robust Reasoning [52.29223403698673]
  This paper examines the use of Conformal Language Modelling (CLM) alongside Answer Set Programming (ASP)<n>We apply CLM to generate sets of ASP programs from an LLM, providing statistical guarantees on the correctness of the outputs.<n> Experimental results show that CLM significantly outperforms baseline models that use standard sampling methods.
  arXiv  Detail & Related papers  (2025-03-07T14:10:10Z)
• Can Frontier LLMs Replace Annotators in Biomedical Text Mining? Analyzing Challenges and Exploring Solutions [0.0]
  Multiple previous studies have reported suboptimal performance of LLMs in biomedical text mining.<n>By analyzing failure patterns in these evaluations, we identified three primary challenges for LLMs in biomedical corpora.<n>Our results show that frontier LLMs can approach or surpass the performance of SOTA BERT-based models.
  arXiv  Detail & Related papers  (2025-03-05T08:37:10Z)
• Investigating the (De)Composition Capabilities of Large Language Models in Natural-to-Formal Language Conversion [21.68354181391989]
  Large language models (LLMs) need to have strong capabilities of decomposition and composition in generalized and robust natural-to-formal language conversion (N2F)<n>We propose the DEDC framework, which performs sample and task construction, allowing evaluation of the set of decomposition and composition capabilities of LLMs in N2F.<n>Our work provides a new perspective for investigating the basic capabilities of decomposition and composition of LLMs in N2F.
  arXiv  Detail & Related papers  (2025-01-24T17:15:09Z)
• Beyond Binary: Towards Fine-Grained LLM-Generated Text Detection via Role Recognition and Involvement Measurement [51.601916604301685]
  Large language models (LLMs) generate content that can undermine trust in online discourse.<n>Current methods often focus on binary classification, failing to address the complexities of real-world scenarios like human-LLM collaboration.<n>To move beyond binary classification and address these challenges, we propose a new paradigm for detecting LLM-generated content.
  arXiv  Detail & Related papers  (2024-10-18T08:14:10Z)
• Benchmarking Complex Instruction-Following with Multiple Constraints Composition [72.82640456309821]
  How to evaluate the ability of complex instruction-following of large language models (LLMs) has become a critical research problem. Existing benchmarks mainly focus on modeling different types of constraints in human instructions while neglecting the composition of different constraints. We propose ComplexBench, a benchmark for comprehensively evaluating the ability of LLMs to follow complex instructions composed of multiple constraints.
  arXiv  Detail & Related papers  (2024-07-04T14:50:45Z)
• Large Language Models are Inconsistent and Biased Evaluators [2.136983452580014]
  We show that Large Language Models (LLMs) are biased evaluators as they exhibit familiarity bias and show skewed distributions of ratings. We also found that LLMs are inconsistent evaluators, showing low "inter-sample" agreement and sensitivity to prompt differences that are insignificant to human understanding of text quality.
  arXiv  Detail & Related papers  (2024-05-02T20:42:28Z)
• Evaluating LLMs' Mathematical Reasoning in Financial Document Question Answering [54.486757407849915]
  This study explores Large Language Models' mathematical reasoning on four financial question-answering datasets.<n>We focus on sensitivity to table complexity and performance variations with an increasing number of arithmetic reasoning steps.<n>We introduce a novel prompting technique tailored to semi-structured documents, matching or outperforming other baselines in performance.
  arXiv  Detail & Related papers  (2024-02-17T05:10:18Z)
• On Learning to Summarize with Large Language Models as References [101.79795027550959]
  Large language models (LLMs) are favored by human annotators over the original reference summaries in commonly used summarization datasets. We study an LLM-as-reference learning setting for smaller text summarization models to investigate whether their performance can be substantially improved.
  arXiv  Detail & Related papers  (2023-05-23T16:56:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.