Referential communication in heterogeneous communities of pre-trained
visual deep networks
- URL: http://arxiv.org/abs/2302.08913v4
- Date: Wed, 13 Mar 2024 16:04:03 GMT
- Title: Referential communication in heterogeneous communities of pre-trained
visual deep networks
- Authors: Mat\'eo Mahaut, Francesca Franzon, Roberto Dess\`i, Marco Baroni
- Abstract summary: Large pre-trained image-processing neural networks are being embedded in autonomous agents such as self-driving cars or robots.
We show that they can develop, in a self-supervised way, a shared protocol to refer to a target object among a set of candidates.
We also study, both qualitatively and quantitatively, the properties of the emergent protocol, providing some evidence that it is capturing high-level semantic features of objects.
- Score: 11.807640148536077
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: As large pre-trained image-processing neural networks are being embedded in
autonomous agents such as self-driving cars or robots, the question arises of
how such systems can communicate with each other about the surrounding world,
despite their different architectures and training regimes. As a first step in
this direction, we systematically explore the task of \textit{referential
communication} in a community of heterogeneous state-of-the-art pre-trained
visual networks, showing that they can develop, in a self-supervised way, a
shared protocol to refer to a target object among a set of candidates. This
shared protocol can also be used, to some extent, to communicate about
previously unseen object categories of different granularity. Moreover, a
visual network that was not initially part of an existing community can learn
the community's protocol with remarkable ease. Finally, we study, both
qualitatively and quantitatively, the properties of the emergent protocol,
providing some evidence that it is capturing high-level semantic features of
objects.
Related papers
- Identifying Sub-networks in Neural Networks via Functionally Similar Representations [41.028797971427124]
We take a step toward automating the understanding of the network by investigating the existence of distinct sub-networks.
Our approach offers meaningful insights into the behavior of neural networks with minimal human and computational cost.
arXiv Detail & Related papers (2024-10-21T20:19:00Z) - Visual-Geometric Collaborative Guidance for Affordance Learning [63.038406948791454]
We propose a visual-geometric collaborative guided affordance learning network that incorporates visual and geometric cues.
Our method outperforms the representative models regarding objective metrics and visual quality.
arXiv Detail & Related papers (2024-10-15T07:35:51Z) - Will 6G be Semantic Communications? Opportunities and Challenges from
Task Oriented and Secure Communications to Integrated Sensing [49.83882366499547]
This paper explores opportunities and challenges of task (goal)-oriented and semantic communications for next-generation (NextG) networks through the integration of multi-task learning.
We employ deep neural networks representing a dedicated encoder at the transmitter and multiple task-specific decoders at the receiver.
We scrutinize potential vulnerabilities stemming from adversarial attacks during both training and testing phases.
arXiv Detail & Related papers (2024-01-03T04:01:20Z) - Less Data, More Knowledge: Building Next Generation Semantic
Communication Networks [180.82142885410238]
We present the first rigorous vision of a scalable end-to-end semantic communication network.
We first discuss how the design of semantic communication networks requires a move from data-driven networks towards knowledge-driven ones.
By using semantic representation and languages, we show that the traditional transmitter and receiver now become a teacher and apprentice.
arXiv Detail & Related papers (2022-11-25T19:03:25Z) - Interpretable agent communication from scratch(with a generic visual
processor emerging on the side) [29.722833768572805]
We train two deep nets from scratch to perform realistic referent identification through unsupervised emergent communication.
We show that the largely interpretable emergent protocol allows the nets to successfully communicate even about object types they did not see at training time.
Our results provide concrete evidence of the viability of (interpretable) emergent deep net communication in a more realistic scenario than previously considered.
arXiv Detail & Related papers (2021-06-08T11:32:11Z) - Joint Learning of Neural Transfer and Architecture Adaptation for Image
Recognition [77.95361323613147]
Current state-of-the-art visual recognition systems rely on pretraining a neural network on a large-scale dataset and finetuning the network weights on a smaller dataset.
In this work, we prove that dynamically adapting network architectures tailored for each domain task along with weight finetuning benefits in both efficiency and effectiveness.
Our method can be easily generalized to an unsupervised paradigm by replacing supernet training with self-supervised learning in the source domain tasks and performing linear evaluation in the downstream tasks.
arXiv Detail & Related papers (2021-03-31T08:15:17Z) - Fast Concept Mapping: The Emergence of Human Abilities in Artificial
Neural Networks when Learning Embodied and Self-Supervised [0.0]
We introduce a setup in which an artificial agent first learns in a simulated world through self-supervised exploration.
We use a method we call fast concept mapping which uses correlated firing patterns of neurons to define and detect semantic concepts.
arXiv Detail & Related papers (2021-02-03T17:19:49Z) - Analyzing Representations inside Convolutional Neural Networks [8.803054559188048]
We propose a framework to categorize the concepts a network learns based on the way it clusters a set of input examples.
This framework is unsupervised and can work without any labels for input features.
We extensively evaluate the proposed method and demonstrate that it produces human-understandable and coherent concepts.
arXiv Detail & Related papers (2020-12-23T07:10:17Z) - Understanding the Role of Individual Units in a Deep Neural Network [85.23117441162772]
We present an analytic framework to systematically identify hidden units within image classification and image generation networks.
First, we analyze a convolutional neural network (CNN) trained on scene classification and discover units that match a diverse set of object concepts.
Second, we use a similar analytic method to analyze a generative adversarial network (GAN) model trained to generate scenes.
arXiv Detail & Related papers (2020-09-10T17:59:10Z) - Detecting Communities in Heterogeneous Multi-Relational Networks:A
Message Passing based Approach [89.19237792558687]
Community is a common characteristic of networks including social networks, biological networks, computer and information networks.
We propose an efficient message passing based algorithm to simultaneously detect communities for all homogeneous networks.
arXiv Detail & Related papers (2020-04-06T17:36:24Z) - Who2com: Collaborative Perception via Learnable Handshake Communication [34.29310680302486]
We propose the problem of collaborative perception, where robots can combine their local observations with those of neighboring agents in a learnable way to improve accuracy on a perception task.
Inspired by networking communication protocols, we propose a multi-stage handshake communication mechanism where the neural network can learn to compress relevant information needed for each stage.
We show that for the semantic segmentation task, our handshake communication method significantly improves accuracy by approximately 20% over decentralized baselines, and is comparable to centralized ones using a quarter of the bandwidth.
arXiv Detail & Related papers (2020-03-21T04:16:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.