Knowledge Graph and Hypergraph Transformers with Repository-Attention and Journey-Based Role Transport

• URL: http://arxiv.org/abs/2603.03304v1
• Date: Sat, 07 Feb 2026 22:44:37 GMT
• Title: Knowledge Graph and Hypergraph Transformers with Repository-Attention and Journey-Based Role Transport
• Abstract summary: We present a concise architecture for joint training on sentences and structured data while keeping knowledge and language representations separable.<n>The model treats knowledge graphs and hypergraphs as structured instances with role slots and encodes them into a key-value repository that a language transformer can attend over.<n>We outline a dual-stream architecture, hierarchical layer groups with instance-local, neighborhood, and global mixing attention, retrieval over a separate repository, and multi-task objectives spanning masked language modeling, link prediction, and role-consistency denoising.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We present a concise architecture for joint training on sentences and structured data while keeping knowledge and language representations separable. The model treats knowledge graphs and hypergraphs as structured instances with role slots and encodes them into a key-value repository that a language transformer can attend over. Attention is conditioned by journey-based role transport, which unifies edge-labeled KG traversal, hyperedge traversal, and sentence structure. We outline a dual-stream architecture, hierarchical layer groups with instance-local, neighborhood, and global mixing attention, retrieval over a separate repository, and multi-task objectives spanning masked language modeling, link prediction, and role-consistency denoising. The result is an explicit, inspectable separation between linguistic context and structured knowledge, while still enabling tight alignment through cross-attention.

Related papers

• FedLAB: Traceable Semantic Codebooks for Federated Multimodal Graph Foundation Learning [47.940785803361415]
  FedLAB is a traceable semantic codebook framework that organizes multimodal graph knowledge into typed codebooks for modality evidence, node semantics, and topology context.<n>Experiments show that FedLAB improves over state-of-the-art baselines by up to 7.53%, while preserving a native semantic trace interface.
  arXiv  Detail & Related papers  (2026-06-30T17:47:39Z)
• Learning Hierarchical Knowledge in Text-Rich Networks with Taxonomy-Informed Representation Learning [16.211492894357473]
  We propose TIER, which first constructs an implicit hierarchical taxonomy and then integrates it into the learned node representations.<n>By learning representations that encode both fine-grained and coarse-grained semantics, TIER enables more interpretable and structured modeling of real-world Text-Rich Networks (TRNs)<n>We demonstrate that our approach significantly outperforms existing methods on multiple datasets across diverse domains.
  arXiv  Detail & Related papers  (2026-03-09T09:40:18Z)
• Self-Supervised Learning as Discrete Communication [3.33876524834826]
  Most self-supervised learning (SSL) methods learn continuous visual representations by aligning different views of the same input.<n>We frame visual self-supervised learning as a discrete communication process between a teacher and a student network.<n>We show that periodically reinitializing the projection head strengthens this effect by encouraging embeddings that remain predictive.
  arXiv  Detail & Related papers  (2026-02-10T13:24:06Z)
• A representational framework for learning and encoding structurally enriched trajectories in complex agent environments [1.1470070927586018]
  The ability of artificial intelligence agents to make optimal decisions and generalise them to different domains and tasks is compromised in complex scenarios.<n>One way to address this issue has focused on learning efficient representations of the world and on how the actions of agents affect them in state-action transitions.<n>We propose to enhance the agent's ontology and extend the traditional conceptualisation of trajectories to provide a more nuanced view of task execution.
  arXiv  Detail & Related papers  (2025-03-17T14:04:27Z)
• Bridging Local Details and Global Context in Text-Attributed Graphs [62.522550655068336]
  GraphBridge is a framework that bridges local and global perspectives by leveraging contextual textual information. Our method achieves state-of-theart performance, while our graph-aware token reduction module significantly enhances efficiency and solves scalability issues.
  arXiv  Detail & Related papers  (2024-06-18T13:35:25Z)
• Contextualization Distillation from Large Language Model for Knowledge Graph Completion [51.126166442122546]
  We introduce the Contextualization Distillation strategy, a plug-in-and-play approach compatible with both discriminative and generative KGC frameworks. Our method begins by instructing large language models to transform compact, structural triplets into context-rich segments. Comprehensive evaluations across diverse datasets and KGC techniques highlight the efficacy and adaptability of our approach.
  arXiv  Detail & Related papers  (2024-01-28T08:56:49Z)
• SIM-Trans: Structure Information Modeling Transformer for Fine-grained Visual Categorization [59.732036564862796]
  We propose the Structure Information Modeling Transformer (SIM-Trans) to incorporate object structure information into transformer for enhancing discriminative representation learning. The proposed two modules are light-weighted and can be plugged into any transformer network and trained end-to-end easily. Experiments and analyses demonstrate that the proposed SIM-Trans achieves state-of-the-art performance on fine-grained visual categorization benchmarks.
  arXiv  Detail & Related papers  (2022-08-31T03:00:07Z)
• Compositional Generalization in Grounded Language Learning via Induced Model Sparsity [81.38804205212425]
  We consider simple language-conditioned navigation problems in a grid world environment with disentangled observations. We design an agent that encourages sparse correlations between words in the instruction and attributes of objects, composing them together to find the goal. Our agent maintains a high level of performance on goals containing novel combinations of properties even when learning from a handful of demonstrations.
  arXiv  Detail & Related papers  (2022-07-06T08:46:27Z)
• A Knowledge-Enhanced Adversarial Model for Cross-lingual Structured Sentiment Analysis [31.05169054736711]
  Cross-lingual structured sentiment analysis task aims to transfer the knowledge from source language to target one. We propose a Knowledge-Enhanced Adversarial Model (textttKEAM) with both implicit distributed and explicit structural knowledge. We conduct experiments on five datasets and compare textttKEAM with both the supervised and unsupervised methods.
  arXiv  Detail & Related papers  (2022-05-31T03:07:51Z)
• Multilingual Extraction and Categorization of Lexical Collocations with Graph-aware Transformers [86.64972552583941]
  We put forward a sequence tagging BERT-based model enhanced with a graph-aware transformer architecture, which we evaluate on the task of collocation recognition in context. Our results suggest that explicitly encoding syntactic dependencies in the model architecture is helpful, and provide insights on differences in collocation typification in English, Spanish and French.
  arXiv  Detail & Related papers  (2022-05-23T16:47:37Z)
• Improve Discourse Dependency Parsing with Contextualized Representations [28.916249926065273]
  We propose to take advantage of transformers to encode contextualized representations of units of different levels. Motivated by the observation of writing patterns commonly shared across articles, we propose a novel method that treats discourse relation identification as a sequence labelling task.
  arXiv  Detail & Related papers  (2022-05-04T14:35:38Z)
• Unsupervised Distillation of Syntactic Information from Contextualized Word Representations [62.230491683411536]
  We tackle the task of unsupervised disentanglement between semantics and structure in neural language representations. To this end, we automatically generate groups of sentences which are structurally similar but semantically different. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics.
  arXiv  Detail & Related papers  (2020-10-11T15:13:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.