SynH-Rank: Quality-Aware Code Search via Diverse Data Synthesis and Hierarchical Ranking Training

• URL: http://arxiv.org/abs/2607.17139v1
• Date: Sun, 19 Jul 2026 08:52:36 GMT
• Title: SynH-Rank: Quality-Aware Code Search via Diverse Data Synthesis and Hierarchical Ranking Training
• Abstract summary: Current code search systems often use a retrieve-then-rerank pipeline, where rerankers focus on modeling semantic relevance between queries and code.<n>We propose SynH-Rank, a quality-aware code reranking framework that combines LLM-driven diverse data synthesis with hierarchical ranking training.<n>We introduce a new benchmark with 4,209 pairs and two novel metrics: Quality Preference Accuracy (QPA) for assessing prioritization of high-quality code and Multi-Condition Accuracy (MCA) for evaluating performance under complex constraints.
• Score: 10.063134179303118
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Code search enhances developer productivity by enabling efficient code reuse. Current code search systems often use a retrieve-then-rerank pipeline, where rerankers focus on modeling semantic relevance between queries and code. However, these rerankers overlook critical non-functional qualities like execution speed, memory usage, and maintainability, which are essential for practical software development. Studies reveal developers expect results to maintain high coding standards and satisfy specific needs, such as resource optimization, highlighting the importance of quality-aware code search. Achieving quality-aware code search faces two major challenges: the scarcity of quality-annotated datasets for effective training and the limitations of standard contrastive learning objectives, which fail to capture the ordinal relationships among high-quality, low-quality, and irrelevant code. Although contrastive learning excels in distinguishing relevant from irrelevant code, its binary objective does not support nuanced quality distinctions.To address these challenges, we propose SynH-Rank, a quality-aware code reranking framework that combines LLM-driven diverse data synthesis with hierarchical ranking training. SynH-Rank employs a three-level labeling scheme to explicitly model the hierarchy: high-quality relevant > low-quality relevant > irrelevant. Additionally, we introduce a new benchmark with 4,209 pairs and two novel metrics: Quality Preference Accuracy (QPA) for assessing prioritization of high-quality code and Multi-Condition Accuracy (MCA) for evaluating performance under complex constraints.Experimental results show SynH-Rank improves QPA by 20.15\% over backbone models and outperforms standard relevance-only contrastive training by 15.80\%, while simultaneously enhancing traditional relevance metrics and multi-condition generalizability.

Related papers

• ClarifyCodeBench: Evaluating LLMs on Clarifying Ambiguous Requirements for Code Generation [54.788849448970154]
  We introduce ClarifyCodeBench, a novel interactive benchmark for evaluating Large Language Models' capability in resolving requirement ambiguity.<n>We conduct a systematic evaluation of six state-of-the-art LLMs using ClarifyCodeBench.<n>Our empirical results yield three critical insights: 1) Capability Decoupling: Strong code generation performance does not inherently translate to effective requirement clarification; 2) The Reasoning Paradox: While increased computational thinking enhances code correctness, it yields marginal gains in identifying ambiguities.
  arXiv  Detail & Related papers  (2026-07-01T09:58:59Z)
• AXIOM: Benchmarking LLM-as-a-Judge for Code via Rule-Based Perturbation and Multisource Quality Calibration [28.117814524373667]
  AXIOM is a novel perturbation-based framework for synthesizing code evaluation benchmarks at scale.<n>It reframes program scores as the refinement effort needed for deployment.
  arXiv  Detail & Related papers  (2025-12-23T08:39:22Z)
• Compressed Video Quality Enhancement: Classifying and Benchmarking over Standards [7.991926266678548]
  This paper introduces a novel taxonomy classifying CVQE methods across architectural paradigms, coding standards, and compressed-domain feature utilization.<n>Second, it proposes a unified benchmarking framework integrating modern compression protocols and standard test sequences for fair multi-criteria evaluation.<n>Third, it provides a systematic analysis of the critical trade-offs between reconstruction performance and computational complexity observed in state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-09-12T16:58:20Z)
• Beyond the Surface: A Solution-Aware Retrieval Model for Competition-level Code Generation [12.805684772054684]
  We propose SolveRank, a solution-aware ranking model empowered by synthetic data for competitive programming tasks.<n>Specifically, we leverage the DeepSeek-R1 model to generate equivalent but differently phrased new problems, verified by GPT-4o for solution consistency.<n>During inference, SolveRank retrieves relevant problems and corresponding code from the corpus to assist a downstream code generator.
  arXiv  Detail & Related papers  (2025-09-01T04:47:51Z)
• COMPASS: A Multi-Dimensional Benchmark for Evaluating Code Generation in Large Language Models [0.0]
  We introduce a comprehensive evaluation framework that assesses code generation across three dimensions: correctness, efficiency, and quality.<n>Our evaluation of three leading reasoning-enhanced models, Anthropic Claude Opus 4, Google Gemini 2.5 Pro, and OpenAI O4-Mini-High, reveals that models achieving high correctness scores do not necessarily produce efficient algorithms or maintainable code.
  arXiv  Detail & Related papers  (2025-08-19T11:55:07Z)
• ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability [83.16850534680505]
  We propose an automated reasoning-intensive training data synthesis framework.<n>A self-consistency data filtering mechanism is designed to ensure the data quality.<n>Our trained reasoning-intensive reranker textbfReasonRank achieves state-of-the-art (SOTA) performance 40.6 on the BRIGHT leaderboard.
  arXiv  Detail & Related papers  (2025-08-09T17:26:18Z)
• CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval [44.39462689088926]
  CoQuIR is the first large-scale, multilingual benchmark designed to evaluate quality-aware code retrieval.<n>CoQuIR provides fine-grained quality annotations for 42,725 queries and 134,907 code snippets in 11 programming languages.
  arXiv  Detail & Related papers  (2025-05-31T13:00:17Z)
• CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programming [58.48683464644606]
  We introduce CPRet, a retrieval-oriented benchmark suite for competitive programming.<n>Our contribution includes both high-quality training data and temporally separated test sets for reliable evaluation.<n>We develop two task-specialized retrievers based on this dataset: CPRetriever-Code, trained with a novel Group-InfoNCE loss for problem-code alignment, and CPRetriever-Prob, fine-tuned for identifying problem-level similarity.
  arXiv  Detail & Related papers  (2025-05-19T10:07:51Z)
• Multi-Level Aware Preference Learning: Enhancing RLHF for Complex Multi-Instruction Tasks [81.44256822500257]
  RLHF has emerged as a predominant approach for aligning artificial intelligence systems with human preferences.<n> RLHF exhibits insufficient compliance capabilities when confronted with complex multi-instruction tasks.<n>We propose a novel Multi-level Aware Preference Learning (MAPL) framework, capable of enhancing multi-instruction capabilities.
  arXiv  Detail & Related papers  (2025-05-19T08:33:11Z)
• IQPFR: An Image Quality Prior for Blind Face Restoration and Beyond [56.99331967165238]
  Blind Face Restoration (BFR) addresses the challenge of reconstructing degraded low-quality (LQ) facial images into high-quality (HQ) outputs.<n>We propose a novel framework that incorporates an Image Quality Prior (IQP) derived from No-Reference Image Quality Assessment (NR-IQA) models.<n>Our method outperforms state-of-the-art techniques across multiple benchmarks.
  arXiv  Detail & Related papers  (2025-03-12T11:39:51Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.