Bridging Coarse and Fine Recognition: A Hybrid Approach for Open-Ended Multi-Granularity Object Recognition in Interactive Educational Games

• URL: http://arxiv.org/abs/2604.16785v1
• Date: Sat, 18 Apr 2026 02:16:34 GMT
• Title: Bridging Coarse and Fine Recognition: A Hybrid Approach for Open-Ended Multi-Granularity Object Recognition in Interactive Educational Games
• Abstract summary: In HyMOR, the MLLM performs open-ended and coarse-grained object recognition, while the CLIP model specializes in fine-grained identification of domain-specific objects such as animals and plants.<n>This hybrid design enables accurate object understanding across multiple semantic granularities, serving as a robust perceptual foundation for downstream multi-modal content generation and interactive gameplay.
• Score: 11.804164932866192
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in Multimodal Large Language Models (MLLMs) have enabled open-ended object recognition, yet they struggle with fine-grained tasks. In contrast, CLIP-style models excel at fine-grained recognition but lack broad coverage of general object categories. To bridge this gap, we propose \textbf{HyMOR}, a \textbf{Hy}brid \textbf{M}ulti-granularity open-ended \textbf{O}bject \textbf{R}ecognition framework that integrates an MLLM with a CLIP model. In HyMOR, the MLLM performs open-ended and coarse-grained object recognition, while the CLIP model specializes in fine-grained identification of domain-specific objects such as animals and plants. This hybrid design enables accurate object understanding across multiple semantic granularities, serving as a robust perceptual foundation for downstream multi-modal content generation and interactive gameplay. To support evaluation in content-rich and educational scenarios, we introduce TBO (TextBook Objects), a dataset containing 20,942 images annotated with 8,816 object categories extracted from textbooks. Extensive experiments demonstrate that HyMOR narrows the fine-grained recognition gap with CLIP to 0.2\% while improving general object recognition by 2.5\% over a baseline MLLM, measured by average Sentence-BERT (SBert) similarity. Overall, HyMOR achieves a 23.2\% improvement in average SBert across all evaluated datasets, highlighting its effectiveness in enabling accurate perception for multi-modal game content generation and interactive learning applications.

Related papers

• HMGCLIP: Heterogeneous Multi-Granularity Contrastive Learning for E-commerce Representation Learning [8.325814892232943]
  HMGCLIP is a unified multimodal embedding framework.<n>We leverage hypergraph topology to mine structure-aware hard negatives and align multi-granular semantics at both relation and hyperedge levels.<n>This design enables a dual-granularity inference mechanism that dynamically fuses attribute evidence for both fine-grained and coarse-grained downstream tasks.
  arXiv  Detail & Related papers  (2026-08-25T12:15:05Z)
• GRACE: LLM-Grounded Semantic Metric Spaces for Scalable Mixed-Data Clustering [19.352298911016668]
  GRACE is a framework for scalable mixed-data clustering.<n>It maps heterogeneous values into knowledge-informed descriptions.<n>It achieves superior clustering accuracy and conceptual interpretability over 11 competing methods.
  arXiv  Detail & Related papers  (2026-08-08T03:14:50Z)
• WOW-Seg: A Word-free Open World Segmentation Model [54.80135620139728]
  We propose WOW-Seg, a Word-free Open World model for segmenting and recognizing objects from open-set categories.<n>Specifically, WOW-Seg introduces a novel visual prompt module, Mask2Token, which transforms image masks into visual tokens.<n>We also construct an open world region recognition test benchmark: the Region Recognition dataset.
  arXiv  Detail & Related papers  (2026-05-16T09:28:46Z)
• Modality-Agnostic Prompt Learning for Multi-Modal Camouflaged Object Detection [61.36976558603528]
  We propose a novel framework that generates modality-agnostic multi-modal prompts for the Segment Anything Model (SAM)<n>Specifically, we model multi-modal learning through interactions between a data-driven content domain and a knowledge-driven prompt domain.<n>We further introduce a lightweight Mask Refine Module to calibrate coarse predictions by incorporating fine-grained prompt cues.
  arXiv  Detail & Related papers  (2026-04-14T07:13:28Z)
• Adaptive Global and Fine-Grained Perceptual Fusion for MLLM Embeddings Compatible with Hard Negative Amplification [49.109117617514066]
  Multimodal embeddings serve as a bridge for aligning vision and language.<n>We propose Adaptive Global and Fine-grained perceptual Fusion for MLLM Embeddings.<n>AGFF-Embed comprehensively achieves state-of-the-art performance in both general and fine-grained understanding.
  arXiv  Detail & Related papers  (2026-02-05T14:52:35Z)
• Generalized Fine-Grained Category Discovery with Multi-Granularity Conceptual Experts [81.68203255687051]
  Generalized Category Discovery is an open-world problem that clusters unlabeled data by leveraging knowledge from partially labeled categories.<n>Existing approaches fail to exploit multi-granularity conceptual information in visual data.<n>We propose a Multi-Granularity Experts framework that integrates multi-granularity knowledge for accurate category discovery.
  arXiv  Detail & Related papers  (2025-09-30T13:25:11Z)
• MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs [61.56904387052982]
  This paper proposes a new visual grounding task called multi-context visual grounding.<n>It aims to localize instances of interest across multiple images based on open-ended text prompts.<n>We benchmark over 20 state-of-the-art MLLMs and foundation models with potential multi-context visual grounding capabilities.
  arXiv  Detail & Related papers  (2024-10-16T07:52:57Z)
• FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension [10.482908189805872]
  Referring Expression (REC) is a crucial cross-modal task that objectively evaluates the capabilities of language understanding, image comprehension, and language-to-image grounding.<n>We have established a new REC dataset characterized by two key features.<n>It includes negative text and images created through fine-grained editing and generation based on existing data.
  arXiv  Detail & Related papers  (2024-09-23T06:56:51Z)
• Towards Optimal Aggregation of Varying Range Dependencies in Haze Removal [17.29370328189668]
  Existing methods achieve notable success by specializing in either short-range dependencies to preserve local details or long-range dependencies to capture global context.<n>We propose DehazeMatic, which simultaneously and explicitly captures both short- and long-range dependencies through a dual-stream design.
  arXiv  Detail & Related papers  (2024-08-22T11:51:50Z)
• Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledge [76.45868419402265]
  multimodal large language models (MLLMs) have made significant strides by training on vast high-quality image-text datasets. However, the inherent difficulty in explicitly conveying fine-grained or spatially dense information in text, such as masks, poses a challenge for MLLMs. This paper proposes a new visual prompt approach to integrate fine-grained external knowledge, gleaned from specialized vision models, into MLLMs.
  arXiv  Detail & Related papers  (2024-07-05T17:43:30Z)
• RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognition [78.97487780589574]
  Multimodal Large Language Models (MLLMs) excel at classifying fine-grained categories. This paper introduces a Retrieving And Ranking augmented method for MLLMs. Our proposed approach not only addresses the inherent limitations in fine-grained recognition but also preserves the model's comprehensive knowledge base.
  arXiv  Detail & Related papers  (2024-03-20T17:59:55Z)
• SEED-Bench-2: Benchmarking Multimodal Large Language Models [67.28089415198338]
  Multimodal large language models (MLLMs) have recently demonstrated exceptional capabilities in generating not only texts but also images given interleaved multimodal inputs. SEED-Bench-2 comprises 24K multiple-choice questions with accurate human annotations, which spans 27 dimensions. We evaluate the performance of 23 prominent open-source MLLMs and summarize valuable observations.
  arXiv  Detail & Related papers  (2023-11-28T05:53:55Z)
• M$^3$Net: Multi-view Encoding, Matching, and Fusion for Few-shot Fine-grained Action Recognition [80.21796574234287]
  M$3$Net is a matching-based framework for few-shot fine-grained (FS-FG) action recognition. It incorporates textitmulti-view encoding, textitmulti-view matching, and textitmulti-view fusion to facilitate embedding encoding, similarity matching, and decision making. Explainable visualizations and experimental results demonstrate the superiority of M$3$Net in capturing fine-grained action details.
  arXiv  Detail & Related papers  (2023-08-06T09:15:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.