An Interdisciplinary Perspective on Evaluation and Experimental Design
for Visual Text Analytics: Position Paper
- URL: http://arxiv.org/abs/2209.11534v1
- Date: Fri, 23 Sep 2022 11:47:37 GMT
- Title: An Interdisciplinary Perspective on Evaluation and Experimental Design
for Visual Text Analytics: Position Paper
- Authors: Kostiantyn Kucher, Nicole Sultanum, Angel Daza, Vasiliki Simaki, Maria
Skeppstedt, Barbara Plank, Jean-Daniel Fekete, and Narges Mahyar
- Abstract summary: In this paper, we focus on the issues of evaluating visual text analytics approaches.
We identify four key groups of challenges for evaluating visual text analytics approaches.
- Score: 24.586485898038312
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Appropriate evaluation and experimental design are fundamental for empirical
sciences, particularly in data-driven fields. Due to the successes in
computational modeling of languages, for instance, research outcomes are having
an increasingly immediate impact on end users. As the gap in adoption by end
users decreases, the need increases to ensure that tools and models developed
by the research communities and practitioners are reliable, trustworthy, and
supportive of the users in their goals. In this position paper, we focus on the
issues of evaluating visual text analytics approaches. We take an
interdisciplinary perspective from the visualization and natural language
processing communities, as we argue that the design and validation of visual
text analytics include concerns beyond computational or visual/interactive
methods on their own. We identify four key groups of challenges for evaluating
visual text analytics approaches (data ambiguity, experimental design, user
trust, and "big picture'' concerns) and provide suggestions for research
opportunities from an interdisciplinary perspective.
Related papers
- How Could AI Support Design Education? A Study Across Fields Fuels Situating Analytics [3.362956277221427]
We use the process and findings from a case study of design educators' practices of assessment and feedback to fuel theorizing.
We theorize a methodology, which we call situating analytics, because making AI support living human activity depends on aligning what analytics measure with situated practices.
arXiv Detail & Related papers (2024-04-26T13:06:52Z) - ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models [92.60282074937305]
We introduce ConTextual, a novel dataset featuring human-crafted instructions that require context-sensitive reasoning for text-rich images.
We conduct experiments to assess the performance of 14 foundation models and establish a human performance baseline.
We observe a significant performance gap of 30.8% between GPT-4V and human performance.
arXiv Detail & Related papers (2024-01-24T09:07:11Z) - Research on the Laws of Multimodal Perception and Cognition from a
Cross-cultural Perspective -- Taking Overseas Chinese Gardens as an Example [5.749458457122218]
This study aims to explore the complex relationship between perceptual and cognitive interactions in multimodal data analysis.
It is found that evaluation content and images on social media can reflect individuals' concerns and sentiment responses.
arXiv Detail & Related papers (2023-12-29T15:13:23Z) - Foundational Models Defining a New Era in Vision: A Survey and Outlook [151.49434496615427]
Vision systems to see and reason about the compositional nature of visual scenes are fundamental to understanding our world.
The models learned to bridge the gap between such modalities coupled with large-scale training data facilitate contextual reasoning, generalization, and prompt capabilities at test time.
The output of such models can be modified through human-provided prompts without retraining, e.g., segmenting a particular object by providing a bounding box, having interactive dialogues by asking questions about an image or video scene or manipulating the robot's behavior through language instructions.
arXiv Detail & Related papers (2023-07-25T17:59:18Z) - Multi-Dimensional Evaluation of Text Summarization with In-Context
Learning [79.02280189976562]
In this paper, we study the efficacy of large language models as multi-dimensional evaluators using in-context learning.
Our experiments show that in-context learning-based evaluators are competitive with learned evaluation frameworks for the task of text summarization.
We then analyze the effects of factors such as the selection and number of in-context examples on performance.
arXiv Detail & Related papers (2023-06-01T23:27:49Z) - Interactive Natural Language Processing [67.87925315773924]
Interactive Natural Language Processing (iNLP) has emerged as a novel paradigm within the field of NLP.
This paper offers a comprehensive survey of iNLP, starting by proposing a unified definition and framework of the concept.
arXiv Detail & Related papers (2023-05-22T17:18:29Z) - You Are What You Talk About: Inducing Evaluative Topics for Personality
Analysis [0.0]
evaluative language data has become more accessible with social media's rapid growth.
We introduce the notion of evaluative topics, obtained by applying topic models to pre-filtered evaluative text.
We then link evaluative topics to individual text authors to build their evaluative profiles.
arXiv Detail & Related papers (2023-02-01T15:04:04Z) - The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations [0.0]
Machine learning (ML) models are nowadays used in complex applications in various domains, such as medicine, bioinformatics, and other sciences.
Due to their black box nature, however, it may sometimes be hard to understand and trust the results they provide.
This has increased the demand for reliable visualization tools related to enhancing trust in ML models.
We present a State-of-the-Art Report (STAR) on enhancing trust in ML models with the use of interactive visualization.
arXiv Detail & Related papers (2022-12-22T14:29:43Z) - TraSE: Towards Tackling Authorial Style from a Cognitive Science
Perspective [4.123763595394021]
Authorship attribution experiments with over 27,000 authors and 1.4 million samples in a cross-domain scenario resulted in 90% attribution accuracy.
A qualitative analysis is performed on TraSE using physical human characteristics, like age, to validate its claim on capturing cognitive traits.
arXiv Detail & Related papers (2022-06-21T19:55:07Z) - A Field Guide to Federated Optimization [161.3779046812383]
Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data.
This paper provides recommendations and guidelines on formulating, designing, evaluating and analyzing federated optimization algorithms.
arXiv Detail & Related papers (2021-07-14T18:09:08Z) - Survey on Visual Sentiment Analysis [87.20223213370004]
This paper reviews pertinent publications and tries to present an exhaustive overview of the field of Visual Sentiment Analysis.
The paper also describes principles of design of general Visual Sentiment Analysis systems from three main points of view.
A formalization of the problem is discussed, considering different levels of granularity, as well as the components that can affect the sentiment toward an image in different ways.
arXiv Detail & Related papers (2020-04-24T10:15:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.