Synthetic Hallucinations, Real Gains: Hard Negatives from Frontier Models for FIM Hallucination Mitigation

• URL: http://arxiv.org/abs/2606.03130v1
• Date: Tue, 02 Jun 2026 04:15:50 GMT
• Title: Synthetic Hallucinations, Real Gains: Hard Negatives from Frontier Models for FIM Hallucination Mitigation
• Abstract summary: Small open-source code models that power autocomplete still emit Fill-in-the-Middle (FIM) completions.<n>We propose an execution-free alternative: use frontier code models to synthesise plausible-but-wrong completions as hard negatives.
• Score: 4.089259624354187
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Small open-source code models that power IDE autocomplete still emit hallucinated Fill-in-the-Middle (FIM) completions: syntactically natural calls to methods, parameters, variables, and imports that do not exist in the surrounding project. Existing mitigations either require per-language execution sandboxes that do not apply at mid-keystroke or preference-optimisation pipelines that need large human-labelled corpora. We propose an execution-free alternative: use frontier code models to synthesise plausible-but-wrong completions as hard negatives, then leverage the contrast between these synthetic hallucinations and the ground-truth developer edit as a supervised fine-tuning signal. Our pipeline scrapes multilingual FIM contexts from public GitHub across eight languages and asks a panel of three frontier generators to produce one hard negative per context for each of four hallucination types drawn from the Delulu taxonomy, a Docker-verified multilingual FIM hallucination benchmark, yielding a paired chosen/rejected dataset. Fine-tuning Qwen2.5-Coder-7B-Instruct on a 100K-row curated subset lifts Delulu exact match by +18.8 points and edit similarity by +0.22 on every language and every type, while also improving every HumanEval-Infilling split and every SAFIM subset. The same recipe at 3B lifts Delulu by +12.8 EM with a small, characterised general-FIM trade-off. Five-axis ablations (size, type mix, language coverage, base-model family, and a difficulty-aware fool rate) plus a head-to-head SFT vs. DPO/ORPO comparison map which design choices drive the gain. We release the full pipeline source code -- generation, fool-rate LLM judging, curation, and the FIM fine-tuning recipe -- so that the experiments in this paper can be reproduced end-to end on any permissively licensed corpus.

Related papers

• Selective Left-Shift: Turning Test-Time Compute and Difficulty-based Curation into Training Data for Low-Resource Code Generation [0.8602553195689513]
  Large Language Models achieve strong code generation for high resource languages like Python and Java but suffer sharp performance drops on Low-Resource Programming Languages(LRPLs) such as Julia.<n>We propose a three-phase pipeline that resolves this trilemma by decoupling syntax acquisition from algorithmic reasoning.<n>Applying to Qwen3-8B, our pipeline improves pass@1 by up to +7.6 points on MultiPL-E and +14.2 points on the Agnostics LiveCodeBench for Julia compared to SOTA results.
  arXiv  Detail & Related papers  (2026-07-08T09:18:56Z)
• Grounded Iterative Language Planning: How Parameterized World Models Reduce Hallucination Propagation in LLM Agents [2.571318992830046]
  An agent-based world model calls an API and reasons flexibly in language, but its errors appear as hallucinated state changes that are hard to score with ordinary regression losses.<n>A parameterized world model is a trained transition predictor; its errors are easier to measure with quantities such as NodeMSE, delta accuracy, and validity accuracy.<n>We compare these two families on four graph-structured planning benchmarks and introduce operational hallucination metrics for the agent-based case.<n>The comparison motivates Grounded Iterative Language Planning(GILP), which trains only a small parameterized backbone and combines it with API-based agent reasoning.
  arXiv  Detail & Related papers  (2026-06-26T07:45:15Z)
• MIPIAD: Multilingual Indirect Prompt Injection Attack Defense with Qwen -- TF-IDF Hybrid and Meta-Ensemble Learning [0.7161783472741748]
  MIPIAD is a defense framework evaluated on English and Bangla.<n>It combines a sequence fine-tuned from Qwen2.5-1.5B via LoRA (XLPID), TF-IDF lexical features, and validation-tuned ensembling.
  arXiv  Detail & Related papers  (2026-05-08T05:34:28Z)
• Delulu: A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Tasks [4.089259624354187]
  Large Language Models for code generation frequently produce hallucinations in Fillin-theMiddle (FIM) tasks.<n>We introduce Delulu, a verified multi-lingual benchmark of 1,951 FIM samples across 7 languages and 4 hallucination types.<n>We evaluate 11 openweight FIM models from five families spanning 0.5B-32B parameters.
  arXiv  Detail & Related papers  (2026-05-07T23:12:00Z)
• Hierarchical Policy Optimization for Simultaneous Translation of Unbounded Speech [50.45710815530982]
  Simultaneous speech translation (SST) generates translations while receiving partial speech input.<n>Recent advances show that large language models (LLMs) can substantially improve SST quality, but at the cost of high computational overhead.<n>We propose a Hierarchical Policy Optimization (HPO) approach that trains post-train models on imperfect SFT data.<n> Experiments on English to Chinese/German/Japanese demonstrate improvements of over +7 COMET score and +1.25 MetricX score at a latency of 1.5 seconds.
  arXiv  Detail & Related papers  (2026-04-22T19:43:51Z)
• Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization [55.543583937522804]
  Multimodal Large Language Models (MLLMs) emerge as a unified interface to address a multitude of tasks.<n>Despite showcasing state-of-the-art results in many benchmarks, a long-standing issue is the tendency of MLLMs to hallucinate.<n>In this paper, we address the problem of hallucinations as an alignment problem, seeking to steer the MLLM so that it prefers generating content without hallucinations.
  arXiv  Detail & Related papers  (2025-08-27T18:02:04Z)
• ETF: An Entity Tracing Framework for Hallucination Detection in Code Summaries [36.63923402713602]
  Large language models (LLMs) are prone to hallucination, outputs that stray from intended meanings.<n>We introduce a first-of-its-kind dataset, CodeSumEval, with 10K samples, curated specifically for hallucination detection in code summarisation.
  arXiv  Detail & Related papers  (2024-10-17T19:38:55Z)
• Nearest Neighbor Speculative Decoding for LLM Generation and Attribution [87.3259169631789]
  Nearest Speculative Decoding (NEST) is capable of incorporating real-world text spans of arbitrary length into the LM generations and providing attribution to their sources.<n>NEST significantly enhances the generation quality and attribution rate of the base LM across a variety of knowledge-intensive tasks.<n>In addition, NEST substantially improves the generation speed, achieving a 1.8x speedup in inference time when applied to Llama-2-Chat 70B.
  arXiv  Detail & Related papers  (2024-05-29T17:55:03Z)
• Aligning Modalities in Vision Large Language Models via Preference Fine-tuning [67.62925151837675]
  In this work, we frame the hallucination problem as an alignment issue, tackle it with preference tuning. Specifically, we propose POVID to generate feedback data with AI models. We use ground-truth instructions as the preferred response and a two-stage approach to generate dispreferred data. In experiments across broad benchmarks, we show that we can not only reduce hallucinations, but improve model performance across standard benchmarks, outperforming prior approaches.
  arXiv  Detail & Related papers  (2024-02-18T00:56:16Z)
• OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation [124.9008419182485]
  We present OPERA, a novel MLLM decoding method grounded in an Over-trust Penalty and a Retrospection-Allocation strategy. Our approach begins with an interesting observation that, most hallucinations are closely tied to the knowledge aggregation patterns in the self-attention matrix. Based on the observation, OPERA introduces a penalty term on the model logits during the beam-search decoding to mitigate the over-trust issue.
  arXiv  Detail & Related papers  (2023-11-29T18:57:07Z)
• Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information [72.2412707779571]
  mRASP is an approach to pre-train a universal multilingual neural machine translation model. We carry out experiments on 42 translation directions across a diverse setting, including low, medium, rich resource, and as well as transferring to exotic language pairs.
  arXiv  Detail & Related papers  (2020-10-07T03:57:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.