SMART SLM: Structured Memory and Reasoning Transformer, A Small Language Model for Accurate Document Assistance

• URL: http://arxiv.org/abs/2512.21280v1
• Date: Wed, 24 Dec 2025 16:59:04 GMT
• Title: SMART SLM: Structured Memory and Reasoning Transformer, A Small Language Model for Accurate Document Assistance
• Abstract summary: SMART (Structured Memory and Reasoning Transformer) structures its processing by using a hierarchical approach.<n>The entire SMART model utilizes 45.51M parameters, which is 64% less than GPT-2 (124M) and 69% less than BERT (133M)
• Score: 0.34376560669160394
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The user of Engineering Manuals (EM) finds it difficult to read EM s because they are long, have a dense format which includes written documents, step by step procedures, and standard parameter lists for engineering equipment. Off the shelf transformers, especially compact ones, treat this material as a flat stream of tokens. This approach leads to confident but incorrect numeric answers and forces the models to memorize separate facts inefficiently. SMART (Structured Memory and Reasoning Transformer) offers a different and practical solution to the above problem. SMART structures its processing by using a hierarchical approach, and is based upon three main job categories (1) A syntax-aware Fact Extractor (Grammarian) Tree LSTM which extracts facts as subject relation object relations from EM sentences (2) A compact indexed memory MANN (Memory Augmented Neural Network) that indexes these Rational Subject Relation Objects as 384 dimensional vectors that are associated with the source of the information, and (3) A 6 layer Transformer that learns to fuse the previously retrieved facts into its generated response. The entire SMART model utilizes 45.51M parameters, which is 64% less than GPT-2 (124M) and 69% less than BERT (133M), and it achieves a 21.3% higher accuracy than GPT-2, indicating that SMART fits the data better with the least amount of processing requirements. SMART employs dual modes of inference an indexed fast path for known documents (sub-second answer times) and an indexed dynamic path assisted by RAGs for new uploads (FAISS Top 20 results with memory severed at 64 slots). In real world deployment, this framework leads to more well supported results with reduced hallucinations than comparable small transformer models.

Related papers

• Dual-Grained Agent Memory and Shapley Context Attribution for Multimodal Agentic Learner [55.80248281761638]
  multimodal large language models (MLLMs) deliver impressive perception yet still falter on scientific and mathematical reasoning.<n>We propose DG-Mem, a dual-grained agentic memory framework that augments a frozen MLLM with a non- externally stored memory built once from training-time rollouts and consulted read-only at test time.
  arXiv  Detail & Related papers  (2026-08-24T13:56:01Z)
• Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory [49.6908961629532]
  Agent Memory Distillation (AMD) is a training-free framework that transfers structured knowledge from a large teacher agent to a small student agent through hierarchical memory.<n>AMD constructs three complementary memory types from successful teacher trajectories: memory encodes task-level strategies, Subtask memory provides concrete behavioral examples, and Function memory captures per-function calling conventions.<n>We evaluate AMD on three tool-use benchmarks using four student models (4B-8B parameters) with GPT-5-mini as the teacher, achieving average accuracy gains of 27.2%p, 11.2%p, and 3.4%p on AppWorld, BFCL V
  arXiv  Detail & Related papers  (2026-08-07T12:43:00Z)
• Relatron: Automating Relational Machine Learning over Relational Databases [50.94254514286021]
  We present a study that unifies RDL and DFS in a shared design space and conducts architecture-centric searches across diverse RDB tasks.<n>Our analysis yields three key findings: (1) RDL does not consistently outperform DFS, with performance being highly task-dependent; (2) no single architecture dominates across tasks, underscoring the need for task-aware model selection; and accuracy is an unreliable guide for choice architecture.
  arXiv  Detail & Related papers  (2026-02-26T02:45:22Z)
• Lightweight Transformers for Zero-Shot and Fine-Tuned Text-to-SQL Generation Using Spider [2.1178416840822027]
  This study evaluates three lightweight transformer models - T5-Small, BART-Small, and GPT-2 - on the Spider dataset.<n>We developed a reusable, model-agnostic pipeline that tailors schema formatting to each model's architecture.
  arXiv  Detail & Related papers  (2025-08-06T16:49:13Z)
• Rethinking Memory in AI: Taxonomy, Operations, Topics, and Future Directions [55.19217798774033]
  Memory is a fundamental component of AI systems, underpinning large language models (LLMs)-based agents.<n>In this survey, we first categorize memory representations into parametric and contextual forms.<n>We then introduce six fundamental memory operations: Consolidation, Updating, Indexing, Forgetting, Retrieval, and Compression.
  arXiv  Detail & Related papers  (2025-05-01T17:31:33Z)
• Sliding Window Attention Training for Efficient Large Language Models [55.56483740523027]
  We introduce SWAT, which enables efficient long-context handling via Sliding Window Attention Training.<n>This paper first attributes the inefficiency of Transformers to the attention sink phenomenon.<n>We replace softmax with the sigmoid function and utilize a balanced ALiBi and Rotary Position Embedding for efficient information compression and retention.
  arXiv  Detail & Related papers  (2025-02-26T05:31:44Z)
• LM2: Large Memory Models [11.320069795732058]
  This paper introduces the Large Memory Model (LM2), a decoder-only Transformer architecture enhanced with an auxiliary memory module.<n> Experimental results on the BABILong benchmark demonstrate that the LM2model outperforms both the memory-augmented RMT model by 37.1% and the baseline Llama-3.2 model by 86.3% on average across tasks.
  arXiv  Detail & Related papers  (2025-02-09T22:11:42Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts.<n>Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• CMT: A Memory Compression Method for Continual Knowledge Learning of Large Language Models [22.93893181000535]
  Large Language Models (LLMs) need to adapt to the continuous changes in data, tasks, and user preferences.<n>To address these challenges, this paper proposes the Compression Memory Training (CMT) method.<n>CMT compresses and extracts information from new documents to be stored in a memory bank.<n>When answering to queries related to these new documents, the model aggregates these document memories from the memory bank to better answer user questions.
  arXiv  Detail & Related papers  (2024-12-10T10:35:19Z)
• UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memory [69.33445217944029]
  PETL is an effective strategy for adapting pre-trained models to downstream domains. Recent PETL works focus on the more valuable memory-efficient characteristic. We propose a new memory-efficient PETL strategy, Universal Parallel Tuning (UniPT)
  arXiv  Detail & Related papers  (2023-08-28T05:38:43Z)
• Mem-Rec: Memory Efficient Recommendation System using Alternative Representation [6.542635536704625]
  MEM-REC is a novel alternative representation approach for embedding tables. We show that MEM-REC can not only maintain the recommendation quality but can also improve the embedding latency.
  arXiv  Detail & Related papers  (2023-05-12T02:36:07Z)
• An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks [40.81306982129298]
  Parametric and retrieval-augmented models have complementary strengths in terms of computational efficiency and predictive accuracy. We propose the Efficient Memory-Augmented Transformer (EMAT) It encodes external knowledge into a key-value memory and exploits the fast maximum inner product search for memory querying.
  arXiv  Detail & Related papers  (2022-10-30T08:34:49Z)
• Parameter-Efficient Abstractive Question Answering over Tables or Text [60.86457030988444]
  A long-term ambition of information seeking QA systems is to reason over multi-modal contexts and generate natural answers to user queries. Memory intensive pre-trained language models are adapted to downstream tasks such as QA by fine-tuning the model on QA data in a specific modality like unstructured text or structured tables. To avoid training such memory-hungry models while utilizing a uniform architecture for each modality, parameter-efficient adapters add and train small task-specific bottle-neck layers between transformer layers.
  arXiv  Detail & Related papers  (2022-04-07T10:56:29Z)
• Transformer Memory as a Differentiable Search Index [102.41278496436948]
  We introduce the Differentiable Search Index (DSI), a new paradigm that learns a text-to-text model that maps string queries directly to relevant docids. We study variations in how documents and their identifiers are represented, variations in training procedures, and the interplay between models and corpus sizes.
  arXiv  Detail & Related papers  (2022-02-14T19:12:43Z)
• Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systems [99.13795374152997]
  We propose a neural network designed to distill an ensemble of large transformers into a single smaller model. An MHS model consists of two components: a stack of transformer layers that is used to encode inputs, and a set of ranking heads. Unlike traditional distillation techniques, our approach leverages individual models in ensemble as teachers in a way that preserves the diversity of the ensemble members.
  arXiv  Detail & Related papers  (2022-01-15T06:21:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.