Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges

• URL: http://arxiv.org/abs/2203.10012v1
• Date: Fri, 18 Mar 2022 15:21:11 GMT
• Title: Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges
• Authors: Shikib Mehri, Jinho Choi, Luis Fernando D'Haro, Jan Deriu, Maxine Eskenazi, Milica Gasic, Kallirroi Georgila, Dilek Hakkani-Tur, Zekang Li, Verena Rieser, Samira Shaikh, David Traum, Yi-Ting Yeh, Zhou Yu, Yizhe Zhang, Chen Zhang
• Abstract summary: The workshop explored the current state of the art along with its limitations and suggested promising directions for future work in this important and very rapidly changing area of research.
• Score: 87.2207978732317
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This is a report on the NSF Future Directions Workshop on Automatic Evaluation of Dialog. The workshop explored the current state of the art along with its limitations and suggested promising directions for future work in this important and very rapidly changing area of research.

Related papers

• The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trends [64.99423243200296]
  Conversation Analysis (CA) strives to uncover and analyze critical information from conversation data. In this paper, we perform a thorough review and systematize CA task to summarize the existing related work. We derive four key steps of CA from conversation scene reconstruction, to in-depth attribution analysis, and then to performing targeted training, finally generating conversations.
  arXiv  Detail & Related papers  (2024-09-21T16:52:43Z)
• Target-Guided Open-Domain Conversation Planning [35.826722856941814]
  We propose the task of Target-Guided Open-Domain Conversation Planning task to evaluate whether neural conversational agents have goal-oriented conversation planning abilities. Using the TGCP task, we investigate the conversation planning abilities of existing retrieval models and recent strong generative models.
  arXiv  Detail & Related papers  (2022-09-20T14:22:33Z)
• GODEL: Large-Scale Pre-Training for Goal-Directed Dialog [119.1397031992088]
  We introduce GODEL, a large pre-trained language model for dialog. We show that GODEL outperforms state-of-the-art pre-trained dialog models in few-shot fine-tuning setups. A novel feature of our evaluation methodology is the introduction of a notion of utility that assesses the usefulness of responses.
  arXiv  Detail & Related papers  (2022-06-22T18:19:32Z)
• Automatic Evaluation and Moderation of Open-domain Dialogue Systems [59.305712262126264]
  A long standing challenge that bothers the researchers is the lack of effective automatic evaluation metrics. This paper describes the data, baselines and results obtained for the Track 5 at the Dialogue System Technology Challenge 10 (DSTC10)
  arXiv  Detail & Related papers  (2021-11-03T10:08:05Z)
• A Survey on Dialogue Summarization: Recent Advances and New Frontiers [19.07064815868915]
  We provide an overview of publicly available research datasets, summarize existing works according to the domain of input dialogue and organize leaderboards under unified metrics. We hope that this first survey of dialogue summarization can provide the community with a quick access and a general picture to this task and motivate future researches.
  arXiv  Detail & Related papers  (2021-07-07T12:11:14Z)
• Deep Learning & Software Engineering: State of Research and Future Directions [37.45171549466233]
  An NSF-sponsored community workshop was conducted in co-location with the 34th IEEE/ACM International Conference on Automated Software Engineering (ASE'19) in San Diego, California. The goal of this workshop was to outline high priority areas for cross-cutting research. This report provides a general summary of the research areas representing the areas of highest priority which were discussed at the workshop.
  arXiv  Detail & Related papers  (2020-09-17T20:46:08Z)
• Ideas for Improving the Field of Machine Learning: Summarizing Discussion from the NeurIPS 2019 Retrospectives Workshop [21.392095675840782]
  This report documents ideas for improving the field of machine learning, which arose from discussions at the ML Retrospectives workshop at NeurIPS 2019. The goal of the report is to disseminate these ideas more broadly, and in turn encourage continuing discussion about how the field could improve along these axes.
  arXiv  Detail & Related papers  (2020-07-21T01:17:29Z)
• A Survey on Dialog Management: Recent Advances and Challenges [72.52920723074638]
  Dialog management (DM) is a crucial component in a task-oriented dialog system. Recent advances and challenges within three critical topics for DM: (1) improving model scalability to facilitate dialog system modeling in new scenarios, (2) dealing with the data scarcity problem for dialog policy learning, and (3) enhancing the training efficiency to achieve better task-completion performance.
  arXiv  Detail & Related papers  (2020-05-05T14:31:24Z)
• Recent Advances and Challenges in Task-oriented Dialog System [63.82055978899631]
  Task-oriented dialog systems are attracting more and more attention in academic and industrial communities. We discuss three critical topics for task-oriented dialog systems: (1) improving data efficiency to facilitate dialog modeling in low-resource settings, (2) modeling multi-turn dynamics for dialog policy learning, and (3) integrating domain knowledge into the dialog model.
  arXiv  Detail & Related papers  (2020-03-17T01:34:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.