Where A Small Language Model Helps in Invoice Categorisation, Understood Through Embedding Geometry

• URL: http://arxiv.org/abs/2608.18033v1
• Date: Tue, 18 Aug 2026 17:28:01 GMT
• Title: Where A Small Language Model Helps in Invoice Categorisation, Understood Through Embedding Geometry
• Abstract summary: In-house small language models (SLMs) can simultaneously reduce cost and improve data security, confidentiality, and interpretability.<n>We investigate this approach by first analysing the pre-trained embedding geometry of a small sentence transformer (SBERT) and classic SLM (DeBERTa)<n>SBERT fine-tuned on a single GPU reaches 0.96 accuracy on invoice classification, above both a zero-shot LLM and a vendor identity baseline.<n>For this important generalisation problem, SBERT reaches 0.9 F1 with roughly 100 client-specific invoices, showing that an in-house SLM implementation is promising.
• Score: 2.9184958249079975
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Categorising invoices into the correct General Ledger (GL) code underpins financial reporting and tax compliance. This is a skilled accounting judgement rather than a routine task: the correct category depends subtly on the nature of the purchasing business, the vendor and the invoice text. Whilst AI is increasingly being adopted across industries to automate tasks, including invoice categorisation, implementations built on in-house small language models (SLMs) can simultaneously reduce cost and improve data security, confidentiality, and interpretability. We investigate this approach by first analysing the pre-trained embedding geometry of a small sentence transformer (SBERT) and classic SLM (DeBERTa). The sentence-embedding space of this financial corpus is globally anisotropic but composed of locally isotropic clusters, extending prior token-level findings to sentence embeddings in a financial setting, and these clusters are strongly correlated with the vendor identity. SBERT fine-tuned on a single GPU reaches 0.96 accuracy on invoice classification, above both a zero-shot LLM and a vendor identity baseline, increasing performance for smaller, challenging categories and new clients. For this important generalisation problem, SBERT reaches 0.9 F1 with roughly 100 client-specific invoices, showing that an in-house SLM implementation is promising. Combining these results with geometric analysis shows that pre-trained embedding geometry is associated with classification performance and reveals a counterintuitive finding that a structured input that would help a human reader does not improve the SLM performance.

Related papers

• Concordia: Self-Improving Synthetic Tables for Federated LLMs [80.03837595689608]
  Federated learning enables training large language models (LLMs) without sharing raw data.<n>We propose Concordia, a framework that aligns synthetic data generation with federated validation utility.
  arXiv  Detail & Related papers  (2026-05-11T01:17:58Z)
• Understanding Structured Financial Data with LLMs: A Case Study on Fraud Detection [17.04809129025246]
  FinFRE-RAG is a two-stage approach that applies importance-guided feature reduction to serialize a compact subset of numeric/categorical attributes into natural language.<n>LLMs can produce human-readable explanations and facilitate feature analysis, potentially reducing the manual workload of fraud analysts.
  arXiv  Detail & Related papers  (2025-12-15T07:09:11Z)
• Enhancing Foundation Models in Transaction Understanding with LLM-based Sentence Embeddings [26.118375969968437]
  Large Language Models (LLMs) can address this limitation through superior semantic understanding.<n>We introduce a hybrid framework that uses LLM-generated embeddings as semantic initializations for lightweight transaction models.<n>Our approach employs multi-source data fusion to enrich merchant categorical fields and a one-word constraint principle for consistent embedding generation.
  arXiv  Detail & Related papers  (2025-12-01T23:30:17Z)
• FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering [57.18367828883773]
  FinAgentBench is a benchmark for evaluating agentic retrieval with multi-step reasoning in finance.<n>The benchmark consists of 26K expert-annotated examples on S&P-500 listed firms.<n>We evaluate a suite of state-of-the-art models and demonstrate how targeted fine-tuning can significantly improve agentic retrieval performance.
  arXiv  Detail & Related papers  (2025-08-07T22:15:22Z)
• FinTagging: Benchmarking LLMs for Extracting and Structuring Financial Information [47.37027539828975]
  FinTagging is the first comprehensive benchmark for structure-aware and full-scope.<n>tagging.<n> FinNI for numeric identification extracts numerical entities and their types from.<n>financial reports.<n>FinCL for concept linking maps each extracted entity to the corresponding concept in the full U.S. taxonomy.
  arXiv  Detail & Related papers  (2025-05-27T02:55:53Z)
• Structuring the Unstructured: A Multi-Agent System for Extracting and Querying Financial KPIs and Guidance [54.25184684077833]
  We propose an efficient and scalable method for extracting quantitative insights from unstructured financial documents.<n>Our proposed system consists of two specialized agents: the emphExtraction Agent and the emphText-to-Agent
  arXiv  Detail & Related papers  (2025-05-25T15:45:46Z)
• Harnessing Generative LLMs for Enhanced Financial Event Entity Extraction Performance [0.0]
  Financial event entity extraction is a crucial task for building financial knowledge graphs.<n>Traditional approaches often rely on sequence labeling models, which can struggle with long-range dependencies.<n>We propose a novel method that reframes financial event entity extraction as a text-to-structured generation task.
  arXiv  Detail & Related papers  (2025-04-20T14:23:31Z)
• Advancing Anomaly Detection: Non-Semantic Financial Data Encoding with LLMs [49.57641083688934]
  We introduce a novel approach to anomaly detection in financial data using Large Language Models (LLMs) embeddings. Our experiments demonstrate that LLMs contribute valuable information to anomaly detection as our models outperform the baselines.
  arXiv  Detail & Related papers  (2024-06-05T20:19:09Z)
• You can't pick your neighbors, or can you? When and how to rely on retrieval in the $k$NN-LM [65.74934004876914]
  Retrieval-enhanced language models (LMs) condition their predictions on text retrieved from large external datastores. One such approach, the $k$NN-LM, interpolates any existing LM's predictions with the output of a $k$-nearest neighbors model. We empirically measure the effectiveness of our approach on two English language modeling datasets.
  arXiv  Detail & Related papers  (2022-10-28T02:57:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.