Statistical Analysis of Sentence Structures through ASCII, Lexical Alignment and PCA

• URL: http://arxiv.org/abs/2503.10470v1
• Date: Thu, 13 Mar 2025 15:42:44 GMT
• Title: Statistical Analysis of Sentence Structures through ASCII, Lexical Alignment and PCA
• Authors: Abhijeet Sahdev,
• Abstract summary: It proposes a novel statistical method that uses American Standard Code for Information Interchange (ASCII) codes to represent text of 11 text corpora.<n>It analyzes the results through histograms and normality tests such as Shapiro-Wilk and Anderson-Darling Tests.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: While utilizing syntactic tools such as parts-of-speech (POS) tagging has helped us understand sentence structures and their distribution across diverse corpora, it is quite complex and poses a challenge in natural language processing (NLP). This study focuses on understanding sentence structure balance - usages of nouns, verbs, determiners, etc - harmoniously without relying on such tools. It proposes a novel statistical method that uses American Standard Code for Information Interchange (ASCII) codes to represent text of 11 text corpora from various sources and their lexical category alignment after using their compressed versions through PCA, and analyzes the results through histograms and normality tests such as Shapiro-Wilk and Anderson-Darling Tests. By focusing on ASCII codes, this approach simplifies text processing, although not replacing any syntactic tools but complementing them by offering it as a resource-efficient tool for assessing text balance. The story generated by Grok shows near normality indicating balanced sentence structures in LLM outputs, whereas 4 out of the remaining 10 pass the normality tests. Further research could explore potential applications in text quality evaluation and style analysis with syntactic integration for more broader tasks.

Related papers

• Detecting Document-level Paraphrased Machine Generated Content: Mimicking Human Writing Style and Involving Discourse Features [57.34477506004105]
  Machine-generated content poses challenges such as academic plagiarism and the spread of misinformation.<n>We introduce novel methodologies and datasets to overcome these challenges.<n>We propose MhBART, an encoder-decoder model designed to emulate human writing style.<n>We also propose DTransformer, a model that integrates discourse analysis through PDTB preprocessing to encode structural features.
  arXiv  Detail & Related papers  (2024-12-17T08:47:41Z)
• SiTSE: Sinhala Text Simplification Dataset and Evaluation [1.7806363928929385]
  This paper presents a human curated sentence-level text simplification dataset for the Sinhala language.<n>We model the text simplification task as a zero-shot and zero resource sequence-to-sequence (seq-seq) task on the multilingual language models mT5 and mBART.<n>Our analysis shows that ITTL outperforms the previously proposed zero-resource methods for text simplification.
  arXiv  Detail & Related papers  (2024-12-02T09:08:06Z)
• Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective [50.261681681643076]
  We propose a novel metric called SemVarEffect and a benchmark named SemVarBench to evaluate the causality between semantic variations in inputs and outputs in text-to-image synthesis. Our work establishes an effective evaluation framework that advances the T2I synthesis community's exploration of human instruction understanding.
  arXiv  Detail & Related papers  (2024-10-14T08:45:35Z)
• Guess & Sketch: Language Model Guided Transpilation [59.02147255276078]
  Learned transpilation offers an alternative to manual re-writing and engineering efforts. Probabilistic neural language models (LMs) produce plausible outputs for every input, but do so at the cost of guaranteed correctness. Guess & Sketch extracts alignment and confidence information from features of the LM then passes it to a symbolic solver to resolve semantic equivalence.
  arXiv  Detail & Related papers  (2023-09-25T15:42:18Z)
• TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision [61.186488081379]
  We propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing.
  arXiv  Detail & Related papers  (2023-06-06T03:37:41Z)
• Evaluating Factual Consistency of Texts with Semantic Role Labeling [3.1776833268555134]
  We introduce SRLScore, a reference-free evaluation metric designed with text summarization in mind. A final factuality score is computed by an adjustable scoring mechanism. Correlation with human judgments on English summarization datasets shows that SRLScore is competitive with state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-22T17:59:42Z)
• A Case Study of Spanish Text Transformations for Twitter Sentiment Analysis [1.9694608733361543]
  Sentiment analysis is a text mining task that determines the polarity of a given text, i.e., its positiveness or negativeness. New forms of textual expressions present new challenges to analyze text given the use of slang, orthographic and grammatical errors.
  arXiv  Detail & Related papers  (2021-06-03T17:24:31Z)
• A Comparative Study on Structural and Semantic Properties of Sentence Embeddings [77.34726150561087]
  We propose a set of experiments using a widely-used large-scale data set for relation extraction. We show that different embedding spaces have different degrees of strength for the structural and semantic properties. These results provide useful information for developing embedding-based relation extraction methods.
  arXiv  Detail & Related papers  (2020-09-23T15:45:32Z)
• Extractive Summarization as Text Matching [123.09816729675838]
  This paper creates a paradigm shift with regard to the way we build neural extractive summarization systems. We formulate the extractive summarization task as a semantic text matching problem. We have driven the state-of-the-art extractive result on CNN/DailyMail to a new level (44.41 in ROUGE-1)
  arXiv  Detail & Related papers  (2020-04-19T08:27:57Z)
• A Multilingual Study of Multi-Sentence Compression using Word Vertex-Labeled Graphs and Integer Linear Programming [1.3922732150370447]
  Multi-Sentence Compression (MSC) aims to generate a short sentence with the key information from a cluster of similar sentences. This paper describes an Linear Programming method for MSC using a graph to select different keywords. Our system is of good quality and outperforms the state of the art for evaluations led on news datasets in three languages.
  arXiv  Detail & Related papers  (2020-04-09T10:35:16Z)
• CompLex: A New Corpus for Lexical Complexity Prediction from Likert Scale Data [13.224233182417636]
  This paper presents the first English dataset for continuous lexical complexity prediction. We use a 5-point Likert scale scheme to annotate complex words in texts from three sources/domains: the Bible, Europarl, and biomedical texts.
  arXiv  Detail & Related papers  (2020-03-16T03:54:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.