Separating Knowledge and Perception with Procedural Data

• URL: http://arxiv.org/abs/2508.11697v1
• Date: Tue, 12 Aug 2025 19:48:35 GMT
• Title: Separating Knowledge and Perception with Procedural Data
• Abstract summary: We train representation models with procedural data only, and apply them on visual similarity, classification, and semantic segmentation tasks without further training.<n>Unlike prior work on visual memory, our approach achieves full compartmentalization with respect to all real-world images.
• Score: 57.462522700631276
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We train representation models with procedural data only, and apply them on visual similarity, classification, and semantic segmentation tasks without further training by using visual memory -- an explicit database of reference image embeddings. Unlike prior work on visual memory, our approach achieves full compartmentalization with respect to all real-world images while retaining strong performance. Compared to a model trained on Places, our procedural model performs within $1\%$ on NIGHTS visual similarity, outperforms by $8\%$ and $15\%$ on CUB200 and Flowers102 fine-grained classification, and is within $10\%$ on ImageNet-1K classification. It also demonstrates strong zero-shot segmentation, achieving an $R^2$ on COCO within $10\%$ of the models trained on real data. Finally, we analyze procedural versus real data models, showing that parts of the same object have dissimilar representations in procedural models, resulting in incorrect searches in memory and explaining the remaining performance gap.

Related papers

• Learning Accurate Segmentation Purely from Self-Supervision [87.78965637247107]
  Selfment is a fully self-supervised framework that segments objects directly from raw images without human labels.<n>Selfment sets new state-of-the-art (SoTA) results across multiple benchmarks.
  arXiv  Detail & Related papers  (2026-02-27T07:36:32Z)
• SAMPO: Visual Preference Optimization for Intent-Aware Segmentation with Vision Foundation Models [5.3279948735247284]
  We introduce SAMPO, a novel framework that teaches visual foundation models to infer high-level categorical intent from sparse visual interactions.<n>Our work establishes a new paradigm for intent-aware alignment in visual foundation models, removing dependencies on auxiliary prompt generators or language-model-assisted preference learning.
  arXiv  Detail & Related papers  (2025-08-04T14:31:11Z)
• DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks [51.439283251703635]
  We create a good, generalist perception model that can tackle multiple tasks, within limits on computational resources and training data.<n>Our exhaustive evaluation metrics demonstrate that DICEPTION effectively tackles multiple perception tasks, achieving performance on par with state-of-the-art models.<n>We show that the strategy of assigning random colors to different instances is highly effective in both entity segmentation and semantic segmentation.
  arXiv  Detail & Related papers  (2025-02-24T13:51:06Z)
• Enhancing Fine-grained Image Classification through Attentive Batch Training [3.9677818965537983]
  We propose a novel module called Residual Relationship Attention (RRA) to integrate visual feature vectors of batch images.<n>We also design a novel framework called Relationship Position, which encodes the positions of relationships between original images in a batch.<n>Our proposed method demonstrates significant improvements in the accuracy of different fine-grained classifiers, with an average increase of $(+2.78%)$ and $(+3.83%)$ on the CUB200-2011 and Stanford Dog datasets.
  arXiv  Detail & Related papers  (2024-12-27T12:07:58Z)
• UnSeg: One Universal Unlearnable Example Generator is Enough against All Image Segmentation [64.01742988773745]
  An increasing privacy concern exists regarding training large-scale image segmentation models on unauthorized private data. We exploit the concept of unlearnable examples to make images unusable to model training by generating and adding unlearnable noise into the original images. We empirically verify the effectiveness of UnSeg across 6 mainstream image segmentation tasks, 10 widely used datasets, and 7 different network architectures.
  arXiv  Detail & Related papers  (2024-10-13T16:34:46Z)
• A Simple and Efficient Baseline for Data Attribution on Images [107.12337511216228]
  Current state-of-the-art approaches require a large ensemble of as many as 300,000 models to accurately attribute model predictions. In this work, we focus on a minimalist baseline, utilizing the feature space of a backbone pretrained via self-supervised learning to perform data attribution. Our method is model-agnostic and scales easily to large datasets.
  arXiv  Detail & Related papers  (2023-11-03T17:29:46Z)
• Exploring Open-Vocabulary Semantic Segmentation without Human Labels [76.15862573035565]
  We present ZeroSeg, a novel method that leverages the existing pretrained vision-language model (VL) to train semantic segmentation models. ZeroSeg overcomes this by distilling the visual concepts learned by VL models into a set of segment tokens, each summarizing a localized region of the target image. Our approach achieves state-of-the-art performance when compared to other zero-shot segmentation methods under the same training data.
  arXiv  Detail & Related papers  (2023-06-01T08:47:06Z)
• Exploring the Limits of Deep Image Clustering using Pretrained Models [1.1060425537315088]
  We present a methodology that learns to classify images without labels by leveraging pretrained feature extractors. We propose a novel objective that learns associations between image features by introducing a variant of pointwise mutual information together with instance weighting.
  arXiv  Detail & Related papers  (2023-03-31T08:56:29Z)
• Sanity checks and improvements for patch visualisation in prototype-based image classification [0.0]
  We perform an in-depth analysis of the visualisation methods implemented in two popular self-explaining models for visual classification based on prototypes. We first show that such methods do not correctly identify the regions of interest inside of the images, and therefore do not reflect the model behaviour. We discuss the implications of our findings for other prototype-based models sharing the same visualisation method.
  arXiv  Detail & Related papers  (2023-01-20T15:13:04Z)
• Jigsaw Clustering for Unsupervised Visual Representation Learning [68.09280490213399]
  We propose a new jigsaw clustering pretext task in this paper. Our method makes use of information from both intra- and inter-images. It is even comparable to the contrastive learning methods when only half of training batches are used.
  arXiv  Detail & Related papers  (2021-04-01T08:09:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.