factgenie: A Framework for Span-based Evaluation of Generated Texts

• URL: http://arxiv.org/abs/2407.17863v1
• Date: Thu, 25 Jul 2024 08:33:23 GMT
• Title: factgenie: A Framework for Span-based Evaluation of Generated Texts
• Authors: Zdeněk Kasner, Ondřej Plátek, Patrícia Schmidtová, Simone Balloccu, Ondřej Dušek,
• Abstract summary: s can capture various span-based phenomena such as semantic inaccuracies or irrelevant text. Our framework consists of a web interface for data visualization and gathering text annotations.
• Score: 1.6864244598342872
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We present factgenie: a framework for annotating and visualizing word spans in textual model outputs. Annotations can capture various span-based phenomena such as semantic inaccuracies or irrelevant text. With factgenie, the annotations can be collected both from human crowdworkers and large language models. Our framework consists of a web interface for data visualization and gathering text annotations, powered by an easily extensible codebase.

Related papers

• Hypergraph based Understanding for Document Semantic Entity Recognition [65.84258776834524]
  We build a novel hypergraph attention document semantic entity recognition framework, HGA, which uses hypergraph attention to focus on entity boundaries and entity categories at the same time. Our results on FUNSD, CORD, XFUNDIE show that our method can effectively improve the performance of semantic entity recognition tasks.
  arXiv  Detail & Related papers  (2024-07-09T14:35:49Z)
• TAGA: Text-Attributed Graph Self-Supervised Learning by Synergizing Graph and Text Mutual Transformations [15.873944819608434]
  Text-Attributed Graphs (TAGs) enhance graph structures with natural language descriptions. This paper introduces a new self-supervised learning framework, Text-And-Graph Multi-View Alignment (TAGA), which integrates TAGs' structural and semantic dimensions. Our framework demonstrates strong performance in zero-shot and few-shot scenarios across eight real-world datasets.
  arXiv  Detail & Related papers  (2024-05-27T03:40:16Z)
• Learning Multiplex Representations on Text-Attributed Graphs with One Language Model Encoder [55.24276913049635]
  We propose METAG, a new framework for learning Multiplex rEpresentations on Text-Attributed Graphs. In contrast to existing methods, METAG uses one text encoder to model the shared knowledge across relations. We conduct experiments on nine downstream tasks in five graphs from both academic and e-commerce domains.
  arXiv  Detail & Related papers  (2023-10-10T14:59:22Z)
• Let the Chart Spark: Embedding Semantic Context into Chart with Text-to-Image Generative Model [7.587729429265939]
  Pictorial visualization seamlessly integrates data and semantic context into visual representation. We propose ChartSpark, a novel system that embeds semantic context into chart based on text-to-image generative model. We develop an interactive visual interface that integrates a text analyzer, editing module, and evaluation module to enable users to generate, modify, and assess pictorial visualizations.
  arXiv  Detail & Related papers  (2023-04-28T05:18:30Z)
• WordStylist: Styled Verbatim Handwritten Text Generation with Latent Diffusion Models [8.334487584550185]
  We present a latent diffusion-based method for styled text-to-text-content-image generation on word-level. Our proposed method is able to generate realistic word image samples from different writer styles. We show that the proposed model produces samples that are aesthetically pleasing, help boosting text recognition performance, and get similar writer retrieval score as real data.
  arXiv  Detail & Related papers  (2023-03-29T10:19:26Z)
• Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding [58.70423899829642]
  We present Pix2Struct, a pretrained image-to-text model for purely visual language understanding. We show that a single pretrained model can achieve state-of-the-art results in six out of nine tasks across four domains.
  arXiv  Detail & Related papers  (2022-10-07T06:42:06Z)
• Modelling the semantics of text in complex document layouts using graph transformer networks [0.0]
  We propose a model that approximates the human reading pattern of a document and outputs a unique semantic representation for every text span. We base our architecture on a graph representation of the structured text, and we demonstrate that not only can we retrieve semantically similar information across documents but also that the embedding space we generate captures useful semantic information.
  arXiv  Detail & Related papers  (2022-02-18T11:49:06Z)
• Sister Help: Data Augmentation for Frame-Semantic Role Labeling [9.62264668211579]
  We propose a data augmentation approach, which uses existing frame-specific annotation to automatically annotate other lexical units of the same frame which are unannotated. We present experiments on frame-semantic role labeling which demonstrate the importance of this data augmentation.
  arXiv  Detail & Related papers  (2021-09-16T05:15:29Z)
• Minimally-Supervised Structure-Rich Text Categorization via Learning on Text-Rich Networks [61.23408995934415]
  We propose a novel framework for minimally supervised categorization by learning from the text-rich network. Specifically, we jointly train two modules with different inductive biases -- a text analysis module for text understanding and a network learning module for class-discriminative, scalable network learning. Our experiments show that given only three seed documents per category, our framework can achieve an accuracy of about 92%.
  arXiv  Detail & Related papers  (2021-02-23T04:14:34Z)
• Improving Image Captioning with Better Use of Captions [65.39641077768488]
  We present a novel image captioning architecture to better explore semantics available in captions and leverage that to enhance both image representation and caption generation. Our models first construct caption-guided visual relationship graphs that introduce beneficial inductive bias using weakly supervised multi-instance learning. During generation, the model further incorporates visual relationships using multi-task learning for jointly predicting word and object/predicate tag sequences.
  arXiv  Detail & Related papers  (2020-06-21T14:10:47Z)
• Probing Contextual Language Models for Common Ground with Visual Representations [76.05769268286038]
  We design a probing model that evaluates how effective are text-only representations in distinguishing between matching and non-matching visual representations. Our findings show that language representations alone provide a strong signal for retrieving image patches from the correct object categories. Visually grounded language models slightly outperform text-only language models in instance retrieval, but greatly under-perform humans.
  arXiv  Detail & Related papers  (2020-05-01T21:28:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.