FrameBench:A Language Understanding Benchmark Based on Frame Semantics

• URL: http://arxiv.org/abs/2609.03370v1
• Date: Thu, 03 Sep 2026 04:59:13 GMT
• Title: FrameBench:A Language Understanding Benchmark Based on Frame Semantics
• Abstract summary: In frame semantics, sentence comprehension is assumed to proceed by relating lexical meaning to background knowledge called semantic frames.<n>Recent large language models (LLMs) have achieved strong performance across a wide range of downstream tasks.<n>We introduce FrameBench, a benchmark grounded in frame semantics.
• Score: 12.28123092144248
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In frame semantics, sentence comprehension is assumed to proceed by relating lexical meaning to background knowledge called semantic frames, thereby enabling readers to implicitly enrich the text with unstated information. Recent large language models (LLMs) have achieved strong performance across a wide range of downstream tasks. However, it remains unclear whether they can reproduce the kinds of implicit enrichment that humans naturally make during comprehension. To address this question, we introduce FrameBench, a benchmark grounded in frame semantics. FrameBench consists of multiple-choice questions that test whether models distinguish the frames evoked by the same verb across contexts. We construct the benchmark for English and Japanese using FrameNet-style resources and a generation-and-verification pipeline with native-speaker judgments. Our experiments on a diverse set of models reveal challenges for small models, while several large models surpass the human reference scores. We release the constructed FrameBench dataset and the code for dataset construction and evaluation at https://github.com/SasanoLab/FrameBench.

Related papers

• RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding [55.75126327155788]
  RIDGE is a frame selection framework that reads the frame-query similarity curve as a temporal signal.<n>It partitions the timeline into structural regions and applies region-specific selection to preserve event cores, transitions, buildup, aftermath, and contextual frames under a fixed budget.<n>It achieves the best performance in most settings and remains competitive in the others.
  arXiv  Detail & Related papers  (2026-08-30T18:40:51Z)
• StructGen: Disambiguating Multi-Reference Image Generation via Structured Context Modeling [112.49355973930555]
  Multi-reference image generation aims to synthesize images by integrating attributes from multiple reference images under textual instructions.<n>Existing approaches, which rely solely on natural language instruction, often fail to capture complex intentions precisely.<n>We propose StructGen, which employs a structured, dictionary-like format to encode multiple reference images.
  arXiv  Detail & Related papers  (2026-07-17T04:35:36Z)
• Do LLMs Encode Frame Semantics? Evidence from Frame Identification [4.508786802660182]
  We investigate whether large language models encode latent knowledge of frame semantics, focusing on frame identification.<n>We evaluate models under prompt-based inference and observe that they can perform frame identification effectively even without explicit supervision.
  arXiv  Detail & Related papers  (2025-09-23T20:09:32Z)
• Openstory++: A Large-scale Dataset and Benchmark for Instance-aware Open-domain Visual Storytelling [81.69474860607542]
  We present Openstory++, a large-scale dataset combining additional instance-level annotations with both images and text. We also present Cohere-Bench, a pioneering benchmark framework for evaluating the image generation tasks when long multimodal context is provided.
  arXiv  Detail & Related papers  (2024-08-07T11:20:37Z)
• Acquiring Frame Element Knowledge with Deep Metric Learning for Semantic Frame Induction [24.486546938073907]
  We propose a method that applies deep metric learning to semantic frame induction tasks. A pre-trained language model is fine-tuned to be suitable for distinguishing frame element roles. Experimental results on FrameNet demonstrate that our method achieves substantially better performance than existing methods.
  arXiv  Detail & Related papers  (2023-05-23T11:02:28Z)
• BenchCLAMP: A Benchmark for Evaluating Language Models on Syntactic and Semantic Parsing [55.058258437125524]
  We introduce BenchCLAMP, a Benchmark to evaluate Constrained LAnguage Model Parsing. We benchmark eight language models, including two GPT-3 variants available only through an API. Our experiments show that encoder-decoder pretrained language models can achieve similar performance or surpass state-of-the-art methods for syntactic and semantic parsing when the model output is constrained to be valid.
  arXiv  Detail & Related papers  (2022-06-21T18:34:11Z)
• A Double-Graph Based Framework for Frame Semantic Parsing [23.552054033442545]
  Frame semantic parsing is a fundamental NLP task, which consists of three subtasks: frame identification, argument identification and role classification. Most previous studies tend to neglect relations between different subtasks and arguments and pay little attention to ontological frame knowledge. In this paper, we propose a Knowledge-guided semanticPK with Double-graph (KID) Our experiments show KID outperforms the previous state-of-the-art method by up to 1.7 F1-score on two FrameNet datasets.
  arXiv  Detail & Related papers  (2022-06-18T09:39:38Z)
• Revealing Single Frame Bias for Video-and-Language Learning [115.01000652123882]
  We show that a single-frame trained model can achieve better performance than existing methods that use multiple frames for training. This result reveals the existence of a strong "static appearance bias" in popular video-and-language datasets. We propose two new retrieval tasks based on existing fine-grained action recognition datasets that encourage temporal modeling.
  arXiv  Detail & Related papers  (2022-06-07T16:28:30Z)
• DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting [91.56988987393483]
  We present a new framework for dense prediction by implicitly and explicitly leveraging the pre-trained knowledge from CLIP. Specifically, we convert the original image-text matching problem in CLIP to a pixel-text matching problem and use the pixel-text score maps to guide the learning of dense prediction models. Our method is model-agnostic, which can be applied to arbitrary dense prediction systems and various pre-trained visual backbones.
  arXiv  Detail & Related papers  (2021-12-02T18:59:32Z)
• Sister Help: Data Augmentation for Frame-Semantic Role Labeling [9.62264668211579]
  We propose a data augmentation approach, which uses existing frame-specific annotation to automatically annotate other lexical units of the same frame which are unannotated. We present experiments on frame-semantic role labeling which demonstrate the importance of this data augmentation.
  arXiv  Detail & Related papers  (2021-09-16T05:15:29Z)
• Efficient Semantic Video Segmentation with Per-frame Inference [117.97423110566963]
  In this work, we process efficient semantic video segmentation in a per-frame fashion during the inference process. We employ compact models for real-time execution. To narrow the performance gap between compact models and large models, new knowledge distillation methods are designed.
  arXiv  Detail & Related papers  (2020-02-26T12:24:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.