HIRA: A Human-in-the-Loop Retrieval-Augmented Cascade for Document Classification in Regulated Industries

• URL: http://arxiv.org/abs/2608.21792v1
• Date: Sat, 22 Aug 2026 06:19:19 GMT
• Title: HIRA: A Human-in-the-Loop Retrieval-Augmented Cascade for Document Classification in Regulated Industries
• Abstract summary: HIRA is a training-free, on-premises retrieval-augmented cascade for document classification in regulated deployments.<n>Confident documents are classified directly by retrieval; uncertain or visually confusable documents are passed to a locally hosted verifier.<n>On a private 80-class trade-finance corpus, HIRA processes the full 30,233-document production stream while requesting human correction for only 1,945 documents.
• Score: 1.5965246343492343
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Document classification in regulated industries is constrained by data residency, limited cold-start labels, scarce review capacity, and costly model-governance procedures. We present HIRA, a training-free, on-premises retrieval-augmented cascade for document classification in regulated deployments that combines BM25 over OCR text, dense text embeddings, and image-level representations through validation-calibrated weighted reciprocal-rank fusion. Confident documents are classified directly by retrieval; uncertain or visually confusable documents are passed to a locally hosted LLM verifier, which receives the OCR text, retrieved exemplars, label descriptions, and confusion-specific terms. When the verifier remains uncertain, the document is sent to human review. Each correction is stored as a margin-weighted retrieval exemplar and updates a Dirichlet-smoothed confusion graph, letting the system improve without updating model weights. On a private 80-class trade-finance corpus, HIRA processes the full 30,233-document production stream while requesting human correction for only 1,945 documents (6.4%), improving Macro-F1 from 0.6218 to 0.8548. On the corrected Tobacco-3482 benchmark, HIRA reaches 0.9423 Macro-F1 with a locally hosted DeepSeek-R1-Distill-Qwen-32B verifier, 17.4 percentage points above the zero-shot LLM baseline, while invoking the verifier for only about 40% of documents and reducing LLM calls by approximately 60%. With 518 human corrections (24.8% of the pool), HIRA matches the fully labelled pool oracle, in which all 2,086 pool documents are indexed with their ground-truth labels. These results show that selective human feedback and retrieval-memory adaptation can be a practical alternative to repeated model retraining for long-tail document classification in regulated deployments.

Related papers

• Approval Integrity and Recovery in LLM Answer Publication [0.2864713389096699]
  LLM systems require binding approved content to its current authorization context.<n>We examine exact-content binding, authorization freshness and checkpoint recovery in Lightcap's publication enforcement mechanism.
  arXiv  Detail & Related papers  (2026-09-14T13:48:48Z)
• IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications [52.570108663867046]
  A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation.<n>We study the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions.<n>We introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances: 163 real-world gaps from reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready references.
  arXiv  Detail & Related papers  (2026-09-09T17:59:04Z)
• AtomCite: Verification and Correction of Supplied Page-Level Citations in Multi-Page Documents [21.842813905857863]
  AtomCite parses an answer into claims, checks each claim against the image of the cited page, and applies a deterministic repair policy.<n>It reaches around 93% binary verification accuracy on the injected benchmark, significantly outperforming every OCR-only condition.<n>Its repair policy lifts citation precision on the injected mix from a constructed 34% to 87-90% while retaining over 90% of correct claims.
  arXiv  Detail & Related papers  (2026-09-05T01:31:50Z)
• Pair-Level Essay-Scale Republication and Reuse from Fragmented Historical Text Reuse: A Workflow Study on Eighteenth-Century Books and Newspapers [0.40873486380569585]
  The study focuses on a candidate set centered on essays by eighteenth-century Scottish philosopher David Hume.<n>We formulate the task as pair-level evidence consolidation into plausible transmission relations.<n>On the full ECCO--ECCO candidate universe, direct LLM baselines flag up to 14,886 pairs as reprints compared to 771 for the final workflow.
  arXiv  Detail & Related papers  (2026-08-27T16:43:27Z)
• Novelty-Aware Agentic Retrieval: Comparing Research Contributions Through Structured Multi-Step Reasoning [0.0]
  Novelty-Aware Research Agent is a prototype agentic retrieval system.<n>It layers structured multi-step reasoning on a RAG pipeline through six typed-contract components.<n>It produces structured comparison artifacts: per-paper contribution records, paper-level overlaps, and a problem x method gap matrix.
  arXiv  Detail & Related papers  (2026-06-20T17:04:02Z)
• RealDocBench: A Benchmark for Field-Level QA and Layout Understanding on Real-World Regulated Documents [0.9003228139607131]
  Document parsing systems are increasingly deployed in high-stakes, regulated domains such as mortgage underwriting, financial reporting, supply-chain logistics, and clinical records.<n>Most public benchmarks evaluate adapters on academic layouts or synthetic prose, and report a single OCR or markdown-level similarity score.<n>We introduce RealDocBench, a two-track benchmark built from real regulated documents.
  arXiv  Detail & Related papers  (2026-06-05T15:41:34Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings [56.70440596502351]
  The past year has seen over 20 open document parsing models, yet benchmarks almost exclusively on OmniDocBench.<n>We present PureDocBench, a benchmark thatrenders document images from HTML/CSS, covering 10 domains, 66 pages, and 1,475 pages.
  arXiv  Detail & Related papers  (2026-05-08T09:30:31Z)
• Resolving the Robustness-Precision Trade-off in Financial RAG through Hybrid Document-Routed Retrieval [7.563079821809866]
  Hybrid Document-Routed Retrieval (HDRR) is a two-stage architecture that uses SFR as a document filter followed by chunk-based retrieval scoped to the identified document(s)<n> Experimental results demonstrate that HDRR achieves the best performance on every metric.
  arXiv  Detail & Related papers  (2026-03-26T18:05:38Z)
• DOCFORGE-BENCH: A Comprehensive 0-shot Benchmark for Document Forgery Detection and Analysis [6.329569532501952]
  We present DOCFORGE-BENCH, the first unified zero-shot benchmark for document forgery detection.<n>We evaluate 14 methods across eight datasets spanning text tampering, receipt forgery, and identity document manipulation.<n>Our central finding is a pervasive calibration failure invisible under single-threshold protocols.
  arXiv  Detail & Related papers  (2026-03-02T04:26:57Z)
• HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam [63.84155758655084]
  Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models.<n>We introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy.<n>We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points.
  arXiv  Detail & Related papers  (2026-02-15T02:50:15Z)
• Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Technique [36.96848724920411]
  We introduce a text-preserving watermarking framework that embeds invisible Unicode characters into documents.<n>We experimentally observe a failure rate of less than 0.1% when detecting a reply after fine-tuning with 50 marked documents.<n>No spurious reply was recovered in over 18,000 challenges, corresponding to a 100%TPR@0% FPR.
  arXiv  Detail & Related papers  (2025-10-07T08:34:08Z)
• LLMs Can Patch Up Missing Relevance Judgments in Evaluation [56.51461892988846]
  We use large language models (LLMs) to automatically label unjudged documents. We simulate scenarios with varying degrees of holes by randomly dropping relevant documents from the relevance judgment in TREC DL tracks. Our method achieves a Kendall tau correlation of 0.87 and 0.92 on an average for Vicuna-7B and GPT-3.5 Turbo respectively.
  arXiv  Detail & Related papers  (2024-05-08T00:32:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.