A Dynamic-Semantics Framework for Grounding Human Referring Expressions in Visual Perceptual Data

• URL: http://arxiv.org/abs/2608.08663v1
• Date: Sun, 09 Aug 2026 12:14:10 GMT
• Title: A Dynamic-Semantics Framework for Grounding Human Referring Expressions in Visual Perceptual Data
• Abstract summary: We present a framework that externalizes pact state into three explicit, inspectable sets of referent-object bindings.<n>We evaluate the framework on the Stanford Repeated Reference Game corpus.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humans converge on shared names for novel, hard-to-describe objects through repeated interaction, a process psycholinguists call lexical entrainment. Leading vision-language models fail at this: recent empirical work documents that they do not shorten references, reuse successful expressions, or maintain stable pact state across turns. We present a framework that addresses the gap by externalizing pact state into three explicit, inspectable sets of referent-object bindings ($Γ, Ξ, Ω$), updated by a dynamic-semantics context-change rule. The symbolic layer sits on top of a lightweight perceptual-alignment pipeline that grounds noisy human referring expressions in crowd-sourced imagery via SIFT homographies and the Universal Quality Index. Evaluated on the Stanford Repeated Reference Game corpus (over 15{,}000 director-matcher utterances on abstract tangram stimuli), the framework places the correct target in its top-5 hypothesis set 83.56% of the time from a single director utterance. Human matcher top-1 accuracy on the same corpus is approximately 77-80%. We also report results on a held-out condition in which obvious tangram-adjacent images are excluded from the retrieved set, which provides a more conservative measurement of the grounding signal. Ablations isolate the contribution of each component: SIFT alignment, UQI, query preprocessing, and image augmentation. The central contribution is the combination: a transparent, auditable symbolic layer that recovers the structure of lexical entrainment turn by turn, paired with a perceptual channel whose behavior can be examined ablation by ablation. We also discuss in detail what the framework does not do. It is not interactive, it does not close the loop with the director, and its retrieval-driven perceptual channel is vulnerable to a class of leakage effects that we quantify and bound rather than wave away.

Related papers

• Dual-Stream Cross-Anchor Correction Grounding Long-Form Captions and the Domain Limits of Object-Level Anchors [7.647944993041615]
  This paper proposes Dual-Stream Cross-Anchor Correction (DSCC)<n>Unlike work that post-processes decoding, DSCC injects object-level visual anchors into the language model itself during fine-tuning.<n> experiments span long-caption hallucination, object-existence discrimination and cross-domain generalisation.
  arXiv  Detail & Related papers  (2026-08-13T02:52:58Z)
• Mapping and Measuring the Behavioral Evolution of Large Language Models [28.603443859513153]
  Benchmark leaderboards summarize how well a language model performs, but not how its behavior relates to that of other models or changes across generations.<n>We characterize the output behavior of 32 models from six families using their responses to a shared bank of 10,000 prompts.
  arXiv  Detail & Related papers  (2026-08-11T15:07:28Z)
• Representation and Reference Selection in Training-Free Synthetic Image Attribution [20.117588342501758]
  Synthetic image attribution aims at identifying the generator responsible for a given AI-generated image.<n>We provide a controlled analysis of this interaction using references and off-the-shelf pretrained representations.<n>Our results show that attribution accuracy consistently peaks at intermediate representation levels.
  arXiv  Detail & Related papers  (2026-07-13T18:10:39Z)
• Cross-domain benchmarks reveal when coordinated AI agents improve scientific inference from partial evidence [1.1458853556386797]
  We evaluate when coordinated AI agents add value over simpler scientific provenance tasks.<n>Cases include mapping molecular structure into musical representations, detecting historical paradigm shifts in science, and vetting transiting-exoplanet candidates.<n>Results: When different disciplines each capture only part of the phenomenon, cross-channel composites improve over single-channel baselines.
  arXiv  Detail & Related papers  (2026-05-21T10:46:50Z)
• RelWitness: Open-Vocabulary 3D Scene Graph Generation with Visual-Geometric Relation Witnesses [1.3468350096927395]
  We propose a framework for open-vocabulary 3D scene graph generation from posed RGB-D sequences under incomplete relation supervision.<n>The key concept is a relation witness: a concrete visual-geometric cue that makes a relation observable in the captured scene.<n> RelWitness constructs relation witness records from RGB views, depth maps, reconstructed 3D geometry, role-sensitive text, object-prior null views, and multi-view consistency.
  arXiv  Detail & Related papers  (2026-05-20T07:18:56Z)
• DynaPURLS: Dynamic Refinement of Part-aware Representations for Skeleton-based Zero-Shot Action Recognition [51.80782323686666]
  We introduce textbfDynaPURLS, a unified framework that establishes robust, multi-scale visual-semantic correspondences.<n>Our framework leverages a large language model to generate hierarchical textual descriptions that encompass both global movements and local body-part dynamics.<n>Experiments on three large-scale benchmark datasets, including NTU RGB+D 60/120 and PKU-MMD, demonstrate that DynaPURLS significantly outperforms prior art.
  arXiv  Detail & Related papers  (2025-12-12T10:39:10Z)
• Learning by Neighbor-Aware Semantics, Deciding by Open-form Flows: Towards Robust Zero-Shot Skeleton Action Recognition [41.77490816513839]
  We propose a novel method for zero-shot skeleton action recognition, termed $texttt$textbfFlora$$.<n>Specifically, we attune textual semantics by incorporating direction-aware regional semantics, and a cross-modal consistency objective.<n>Experiments on three benchmark datasets validate the effectiveness of our method, showing particularly impressive performance even when trained with only 10% of the seen data.
  arXiv  Detail & Related papers  (2025-11-12T14:54:53Z)
• Generalized Visual Relation Detection with Diffusion Models [94.62313788626128]
  Visual relation detection (VRD) aims to identify relationships (or interactions) between object pairs in an image.<n>We propose to model visual relations as continuous embeddings, and design diffusion models to achieve generalized VRD in a conditional generative manner.<n>Our Diff-VRD is able to generate visual relations beyond the pre-defined category labels of datasets.
  arXiv  Detail & Related papers  (2025-04-16T14:03:24Z)
• Transitivity Recovering Decompositions: Interpretable and Robust Fine-Grained Relationships [69.04014445666142]
  Transitivity Recovering Decompositions (TRD) is a graph-space search algorithm that identifies interpretable equivalents of abstract emergent relationships. We show that TRD is provably robust to noisy views, with empirical evidence also supporting this finding.
  arXiv  Detail & Related papers  (2023-10-24T16:48:56Z)
• Noisy-Correspondence Learning for Text-to-Image Person Re-identification [50.07634676709067]
  We propose a novel Robust Dual Embedding method (RDE) to learn robust visual-semantic associations even with noisy correspondences. Our method achieves state-of-the-art results both with and without synthetic noisy correspondences on three datasets.
  arXiv  Detail & Related papers  (2023-08-19T05:34:13Z)
• Collaborative Group: Composed Image Retrieval via Consensus Learning from Noisy Annotations [67.92679668612858]
  We propose the Consensus Network (Css-Net), inspired by the psychological concept that groups outperform individuals. Css-Net comprises two core components: (1) a consensus module with four diverse compositors, each generating distinct image-text embeddings; and (2) a Kullback-Leibler divergence loss that encourages learning of inter-compositor interactions. On benchmark datasets, particularly FashionIQ, Css-Net demonstrates marked improvements. Notably, it achieves significant recall gains, with a 2.77% increase in R@10 and 6.67% boost in R@50, underscoring its
  arXiv  Detail & Related papers  (2023-06-03T11:50:44Z)
• High-Order Information Matters: Learning Relation and Topology for Occluded Person Re-Identification [84.43394420267794]
  We propose a novel framework by learning high-order relation and topology information for discriminative features and robust alignment. Our framework significantly outperforms state-of-the-art by6.5%mAP scores on Occluded-Duke dataset.
  arXiv  Detail & Related papers  (2020-03-18T12:18:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.