Describing Textures using Natural Language

• URL: http://arxiv.org/abs/2008.01180v1
• Date: Mon, 3 Aug 2020 20:37:35 GMT
• Title: Describing Textures using Natural Language
• Authors: Chenyun Wu, Mikayla Timm, Subhransu Maji
• Abstract summary: Textures in natural images can be characterized by color, shape, periodicity of elements within them, and other attributes that can be described using natural language. We study the problem of describing visual attributes of texture on a novel dataset containing rich descriptions of textures. We present visualizations of several fine-grained domains and show that texture attributes learned on our dataset offer improvements over expert-designed attributes on the Caltech-UCSD Birds dataset.
• Score: 32.076605062485605
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Textures in natural images can be characterized by color, shape, periodicity of elements within them, and other attributes that can be described using natural language. In this paper, we study the problem of describing visual attributes of texture on a novel dataset containing rich descriptions of textures, and conduct a systematic study of current generative and discriminative models for grounding language to images on this dataset. We find that while these models capture some properties of texture, they fail to capture several compositional properties, such as the colors of dots. We provide critical analysis of existing models by generating synthetic but realistic textures with different descriptions. Our dataset also allows us to train interpretable models and generate language-based explanations of what discriminative features are learned by deep networks for fine-grained categorization where texture plays a key role. We present visualizations of several fine-grained domains and show that texture attributes learned on our dataset offer improvements over expert-designed attributes on the Caltech-UCSD Birds dataset.

Related papers

• On Synthetic Texture Datasets: Challenges, Creation, and Curation [1.9567015559455132]
  We create a dataset of 362,880 texture images that span 56 textures. During the process of generating images, we find that NSFW safety filters in image generation pipelines are highly sensitive to texture.
  arXiv  Detail & Related papers  (2024-09-16T14:02:18Z)
• Compositional Neural Textures [25.885557234297835]
  This work introduces a fully unsupervised approach for representing textures using a compositional neural model. We represent each texton as a 2D Gaussian function whose spatial support approximates its shape, and an associated feature that encodes its detailed appearance. This approach enables a wide range of applications, including transferring appearance from an image texture to another image, diversifying textures, revealing/modifying texture variations, edit propagation, texture animation, and direct texton manipulation.
  arXiv  Detail & Related papers  (2024-04-18T21:09:34Z)
• Explorations in Texture Learning [1.9567015559455132]
  We build texture-object associations that uncover new insights about the relationships between texture and object classes in CNNs. Our analysis demonstrates that investigations in texture learning enable new methods for interpretability and have the potential to uncover unexpected biases.
  arXiv  Detail & Related papers  (2024-03-14T16:30:52Z)
• TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusion [64.49276500129092]
  TextureDreamer is an image-guided texture synthesis method. It can transfer relightable textures from a small number of input images to target 3D shapes across arbitrary categories.
  arXiv  Detail & Related papers  (2024-01-17T18:55:49Z)
• Text2Scene: Text-driven Indoor Scene Stylization with Part-aware Details [12.660352353074012]
  We propose Text2Scene, a method to automatically create realistic textures for virtual scenes composed of multiple objects. Our pipeline adds detailed texture on labeled 3D geometries in the room such that the generated colors respect the hierarchical structure or semantic parts that are often composed of similar materials.
  arXiv  Detail & Related papers  (2023-08-31T17:37:23Z)
• Leveraging Textures in Zero-shot Understanding of Fine-Grained Domains [34.848408203825194]
  We study the effectiveness of large-scale language and vision models (e.g., CLIP) at recognizing texture attributes in natural images. We first conduct a systematic study of CLIP on texture datasets where we find that it has good coverage for a wide range of texture terms. We then show how these attributes allow for zero-shot fine-grained categorization on existing datasets.
  arXiv  Detail & Related papers  (2022-03-22T04:07:20Z)
• Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization [81.26077816854449]
  We first explore the use of constituency parse trees for encoding structured input. Second, we augment the structured input with commonsense information and study the impact of this external knowledge on the generation of visual story. Third, we incorporate visual structure via bounding boxes and dense captioning to provide feedback about the characters/objects in generated images.
  arXiv  Detail & Related papers  (2021-10-21T00:16:02Z)
• Artificial Text Detection via Examining the Topology of Attention Maps [58.46367297712477]
  We propose three novel types of interpretable topological features for this task based on Topological Data Analysis (TDA) We empirically show that the features derived from the BERT model outperform count- and neural-based baselines up to 10% on three common datasets. The probing analysis of the features reveals their sensitivity to the surface and syntactic properties.
  arXiv  Detail & Related papers  (2021-09-10T12:13:45Z)
• Texture Generation with Neural Cellular Automata [64.70093734012121]
  We learn a texture generator from a single template image. We make claims that the behaviour exhibited by the NCA model is a learned, distributed, local algorithm to generate a texture.
  arXiv  Detail & Related papers  (2021-05-15T22:05:46Z)
• Deep Texture-Aware Features for Camouflaged Object Detection [69.84122372541506]
  This paper formulates texture-aware refinement modules to learn the texture-aware features in a deep convolutional neural network. We evaluate our network on the benchmark dataset for camouflaged object detection both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2021-02-05T04:38:32Z)
• Reverse Engineering Configurations of Neural Text Generation Models [86.9479386959155]
  The study of artifacts that emerge in machine generated text as a result of modeling choices is a nascent research area. We conduct an extensive suite of diagnostic tests to observe whether modeling choices leave detectable artifacts in the text they generate. Our key finding, which is backed by a rigorous set of experiments, is that such artifacts are present and that different modeling choices can be inferred by observing the generated text alone.
  arXiv  Detail & Related papers  (2020-04-13T21:02:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.