Factual Inconsistency in Data-to-Text Generation Scales Exponentially with LLM Size: A Statistical Validation

• URL: http://arxiv.org/abs/2502.12372v1
• Date: Mon, 17 Feb 2025 23:24:00 GMT
• Title: Factual Inconsistency in Data-to-Text Generation Scales Exponentially with LLM Size: A Statistical Validation
• Authors: Joy Mahapatra, Soumyajit Roy, Utpal Garain,
• Abstract summary: This paper explores the impact of large language models (LLMs) size on factual inconsistency in data-to-text generation (D2T) We employ a statistical validation framework consisting of three key stages: predictive performance estimation, goodness-of-fit assessment, and comparative analysis. For a comprehensive empirical study, we analyze three popular LLM families across five D2T datasets, measuring factual inconsistency inversely using four state-of-the-art consistency metrics.
• Score: 1.6795461001108096
• License:
• Abstract: Monitoring factual inconsistency is essential for ensuring trustworthiness in data-to-text generation (D2T). While large language models (LLMs) have demonstrated exceptional performance across various D2T tasks, previous studies on scaling laws have primarily focused on generalization error through power law scaling to LLM size (i.e., the number of model parameters). However, no research has examined the impact of LLM size on factual inconsistency in D2T. In this paper, we investigate how factual inconsistency in D2T scales with LLM size by exploring two scaling laws: power law and exponential scaling. To rigorously evaluate and compare these scaling laws, we employ a statistical validation framework consisting of three key stages: predictive performance estimation, goodness-of-fit assessment, and comparative analysis. For a comprehensive empirical study, we analyze three popular LLM families across five D2T datasets, measuring factual inconsistency inversely using four state-of-the-art consistency metrics. Our findings, based on exhaustive empirical results and validated through our framework, reveal that, contrary to the widely assumed power law scaling, factual inconsistency in D2T follows an exponential scaling with LLM size.

Related papers

• Sloth: scaling laws for LLM skills to predict multi-benchmark performance across families [43.36524246307057]
  Scaling laws for large language models (LLMs) predict performance based on parameters like size and training data. We propose Skills Scaling Laws (SSLaws), a novel scaling law that leverages publicly available benchmark data. We present both theoretical results on parameter identification and empirical evaluations on 12 prominent benchmarks.
  arXiv  Detail & Related papers  (2024-12-09T14:51:26Z)
• Scaling Law for Language Models Training Considering Batch Size [17.09348741898811]
  Large language models (LLMs) have made remarkable advances in recent years, with scaling laws playing a critical role in this rapid progress. We empirically investigate how a critical hyper- parameter, i.e., the global batch size, influences the LLM training prdocess. We establish a basic scaling law on model size and training data amount. We then examine how varying batch sizes and learning rates affect the convergence and generalization of these models.
  arXiv  Detail & Related papers  (2024-12-02T13:58:35Z)
• An Extensive Evaluation of Factual Consistency in Large Language Models for Data-to-Text Generation [1.8876415010297893]
  Large Language Models (LLMs) have shown exceptional performance across various Data-to-Text Generation (DTG) tasks. generating factually consistent text in DTG remains challenging for LLMs. This paper provides an extensive evaluation of factual consistency in LLMs for DTG.
  arXiv  Detail & Related papers  (2024-11-28T15:23:12Z)
• Bayesian scaling laws for in-context learning [72.17734205418502]
  In-context learning (ICL) is a powerful technique for getting language models to perform complex tasks with no training updates. We show that ICL approximates a Bayesian learner and develop a family of novel Bayesian scaling laws for ICL.
  arXiv  Detail & Related papers  (2024-10-21T21:45:22Z)
• Performance Law of Large Language Models [58.32539851241063]
  Performance law can be used to guide the choice of LLM architecture and the effective allocation of computational resources. Performance law can be used to guide the choice of LLM architecture and the effective allocation of computational resources without extensive experiments.
  arXiv  Detail & Related papers  (2024-08-19T11:09:12Z)
• Impact of Model Size on Fine-tuned LLM Performance in Data-to-Text Generation: A State-of-the-Art Investigation [1.8876415010297893]
  Data-to-text (D2T) generation aims to generate human-readable text from semi-structured data, such as tables and graphs. No research has been conducted to illustrate the impact of model size on the performance of fine-tuned LLMs for D2T tasks. We aim to elucidate both the advantages and limitations of scaling model sizes across five widely used D2T datasets.
  arXiv  Detail & Related papers  (2024-07-19T07:54:30Z)
• Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoning [64.5243480989869]
  coding data is known to boost reasoning abilities during pretraining. Its role in activating internal reasoning capacities during IFT remains understudied. This paper investigates how coding data impact LLMs' reasoning capacities during IFT stage.
  arXiv  Detail & Related papers  (2024-05-30T23:20:25Z)
• Temporal Scaling Law for Large Language Models [24.12384260752973]
  We propose the novel concept of Temporal Scaling Law, studying how the test loss of an LLM evolves as the training steps scale up. In contrast to modeling the test loss as a whole in a coarse-grained manner, we break it down and dive into the fine-grained test loss of each token position. We derive the much more precise temporal scaling law by studying the temporal patterns of the parameters in the dynamic hyperbolic-law.
  arXiv  Detail & Related papers  (2024-04-27T05:49:11Z)
• When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method [56.571951345048355]
  Large language models (LLMs) often adopt finetuning to unlock their capabilities for downstream applications. We study whether and how different scaling factors, including LLM model size, pretraining data size, new finetuning parameter size and finetuning data size, affect the finetuning performance.
  arXiv  Detail & Related papers  (2024-02-27T04:18:49Z)
• Scaling Relationship on Learning Mathematical Reasoning with Large Language Models [75.29595679428105]
  We investigate how the pre-training loss, supervised data amount, and augmented data amount influence the reasoning performances of a supervised LLM. We find that rejection samples from multiple models push LLaMA-7B to an accuracy of 49.3% on GSM8K which outperforms the supervised fine-tuning (SFT) accuracy of 35.9% significantly.
  arXiv  Detail & Related papers  (2023-08-03T15:34:01Z)
• A Solvable Model of Neural Scaling Laws [72.8349503901712]
  Large language models with a huge number of parameters, when trained on near internet-sized number of tokens, have been empirically shown to obey neural scaling laws. We propose a statistical model -- a joint generative data model and random feature model -- that captures this neural scaling phenomenology. Key findings are the manner in which the power laws that occur in the statistics of natural datasets are extended by nonlinear random feature maps.
  arXiv  Detail & Related papers  (2022-10-30T15:13:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.