An Interdisciplinary Perspective on Evaluation and Experimental Design
for Visual Text Analytics: Position Paper
- URL: http://arxiv.org/abs/2209.11534v1
- Date: Fri, 23 Sep 2022 11:47:37 GMT
- Title: An Interdisciplinary Perspective on Evaluation and Experimental Design
for Visual Text Analytics: Position Paper
- Authors: Kostiantyn Kucher, Nicole Sultanum, Angel Daza, Vasiliki Simaki, Maria
Skeppstedt, Barbara Plank, Jean-Daniel Fekete, and Narges Mahyar
- Abstract summary: In this paper, we focus on the issues of evaluating visual text analytics approaches.
We identify four key groups of challenges for evaluating visual text analytics approaches.
- Score: 24.586485898038312
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Appropriate evaluation and experimental design are fundamental for empirical
sciences, particularly in data-driven fields. Due to the successes in
computational modeling of languages, for instance, research outcomes are having
an increasingly immediate impact on end users. As the gap in adoption by end
users decreases, the need increases to ensure that tools and models developed
by the research communities and practitioners are reliable, trustworthy, and
supportive of the users in their goals. In this position paper, we focus on the
issues of evaluating visual text analytics approaches. We take an
interdisciplinary perspective from the visualization and natural language
processing communities, as we argue that the design and validation of visual
text analytics include concerns beyond computational or visual/interactive
methods on their own. We identify four key groups of challenges for evaluating
visual text analytics approaches (data ambiguity, experimental design, user
trust, and "big picture'' concerns) and provide suggestions for research
opportunities from an interdisciplinary perspective.
Related papers
- Data Analysis in the Era of Generative AI [56.44807642944589]
This paper explores the potential of AI-powered tools to reshape data analysis, focusing on design considerations and challenges.
We explore how the emergence of large language and multimodal models offers new opportunities to enhance various stages of data analysis workflow.
We then examine human-centered design principles that facilitate intuitive interactions, build user trust, and streamline the AI-assisted analysis workflow across multiple apps.
arXiv Detail & Related papers (2024-09-27T06:31:03Z) - Good Idea or Not, Representation of LLM Could Tell [86.36317971482755]
We focus on idea assessment, which aims to leverage the knowledge of large language models to assess the merit of scientific ideas.
We release a benchmark dataset from nearly four thousand manuscript papers with full texts, meticulously designed to train and evaluate the performance of different approaches to this task.
Our findings suggest that the representations of large language models hold more potential in quantifying the value of ideas than their generative outputs.
arXiv Detail & Related papers (2024-09-07T02:07:22Z) - Combining Objective and Subjective Perspectives for Political News Understanding [5.741243797283764]
We introduce a text analysis framework which integrates both perspectives and provides a fine-grained processing of subjective aspects.
We illustrate its functioning with insights on news outlets, political orientations, topics, individual entities, and demographic segments.
arXiv Detail & Related papers (2024-08-20T20:13:19Z) - How Could AI Support Design Education? A Study Across Fields Fuels Situating Analytics [3.362956277221427]
We use the process and findings from a case study of design educators' practices of assessment and feedback to fuel theorizing.
We theorize a methodology, which we call situating analytics, because making AI support living human activity depends on aligning what analytics measure with situated practices.
arXiv Detail & Related papers (2024-04-26T13:06:52Z) - Research on the Laws of Multimodal Perception and Cognition from a
Cross-cultural Perspective -- Taking Overseas Chinese Gardens as an Example [5.749458457122218]
This study aims to explore the complex relationship between perceptual and cognitive interactions in multimodal data analysis.
It is found that evaluation content and images on social media can reflect individuals' concerns and sentiment responses.
arXiv Detail & Related papers (2023-12-29T15:13:23Z) - Multi-Dimensional Evaluation of Text Summarization with In-Context
Learning [79.02280189976562]
In this paper, we study the efficacy of large language models as multi-dimensional evaluators using in-context learning.
Our experiments show that in-context learning-based evaluators are competitive with learned evaluation frameworks for the task of text summarization.
We then analyze the effects of factors such as the selection and number of in-context examples on performance.
arXiv Detail & Related papers (2023-06-01T23:27:49Z) - Interactive Natural Language Processing [67.87925315773924]
Interactive Natural Language Processing (iNLP) has emerged as a novel paradigm within the field of NLP.
This paper offers a comprehensive survey of iNLP, starting by proposing a unified definition and framework of the concept.
arXiv Detail & Related papers (2023-05-22T17:18:29Z) - You Are What You Talk About: Inducing Evaluative Topics for Personality
Analysis [0.0]
evaluative language data has become more accessible with social media's rapid growth.
We introduce the notion of evaluative topics, obtained by applying topic models to pre-filtered evaluative text.
We then link evaluative topics to individual text authors to build their evaluative profiles.
arXiv Detail & Related papers (2023-02-01T15:04:04Z) - The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations [0.0]
Machine learning (ML) models are nowadays used in complex applications in various domains, such as medicine, bioinformatics, and other sciences.
Due to their black box nature, however, it may sometimes be hard to understand and trust the results they provide.
This has increased the demand for reliable visualization tools related to enhancing trust in ML models.
We present a State-of-the-Art Report (STAR) on enhancing trust in ML models with the use of interactive visualization.
arXiv Detail & Related papers (2022-12-22T14:29:43Z) - A Field Guide to Federated Optimization [161.3779046812383]
Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data.
This paper provides recommendations and guidelines on formulating, designing, evaluating and analyzing federated optimization algorithms.
arXiv Detail & Related papers (2021-07-14T18:09:08Z) - Survey on Visual Sentiment Analysis [87.20223213370004]
This paper reviews pertinent publications and tries to present an exhaustive overview of the field of Visual Sentiment Analysis.
The paper also describes principles of design of general Visual Sentiment Analysis systems from three main points of view.
A formalization of the problem is discussed, considering different levels of granularity, as well as the components that can affect the sentiment toward an image in different ways.
arXiv Detail & Related papers (2020-04-24T10:15:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.