Think Inside the JSON: Reinforcement Strategy for Strict LLM Schema Adherence

• URL: http://arxiv.org/abs/2502.14905v1
• Date: Tue, 18 Feb 2025 16:44:55 GMT
• Title: Think Inside the JSON: Reinforcement Strategy for Strict LLM Schema Adherence
• Authors: Bhavik Agarwal, Ishan Joshi, Viktoria Rojkova,
• Abstract summary: We address the challenge of enforcing strict schema adherence in large language model (LLM) generation by leveraging reasoning capabilities.<n>Our approach trains structured reasoning skills of a 1.5B parameter model through a novel pipeline.<n>We compare our ThinkJSON approach against the original DeepSeek R1 (671B), distilled versions of DeepSeek R1 (Qwen-1.5B and Qwen-7B), and Gemini 2.0 Flash (70B)
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we address the challenge of enforcing strict schema adherence in large language model (LLM) generation by leveraging LLM reasoning capabilities. Building on the DeepSeek R1 reinforcement learning framework, our approach trains structured reasoning skills of a 1.5B parameter model through a novel pipeline that combines synthetic reasoning dataset construction with custom reward functions under Group Relative Policy Optimization (GRPO). Specifically, we first perform R1 reinforcement learning on a 20K sample unstructured-to-structured dataset, mirroring the original DeepSeek R1 methods, to establish core reasoning abilities. Subsequently, we performed supervised fine-tuning on a separate 10K reasoning sample dataset, focusing on refining schema adherence for downstream tasks. Despite the relatively modest training scope, requiring approximately 20 hours on an 8xH100 GPU cluster for GRPO training and 3 hours on 1xA100 for SFT, our model demonstrates robust performance in enforcing schema consistency. We compare our ThinkJSON approach against the original DeepSeek R1 (671B), distilled versions of DeepSeek R1 (Qwen-1.5B and Qwen-7B), and Gemini 2.0 Flash (70B), showcasing its effectiveness in real-world applications. Our results underscore the practical utility of a resource-efficient framework for schema-constrained text generation.

Related papers

• SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM [18.275547804539016]
  Two-Staged history-Resampling Policy optimization surpasses the performance of DeepSeek-R1-Zero-32B on the AIME24 and LiveCodeBench benchmarks. We introduce two key methodological innovations: (1) a two-stage cross-domain training paradigm designed to balance the development of mathematical reasoning and coding proficiency, and (2) History Resampling (HR), a technique to address ineffective samples.
  arXiv  Detail & Related papers  (2025-04-19T13:06:03Z)
• Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model [47.108822717757945]
  We introduce Open-Reasoner-Zero, the first open source implementation of large-scale reasoning-oriented RL training. In the spirit of open source, we release our source code, parameter settings, training data, and model weights across various sizes.
  arXiv  Detail & Related papers  (2025-03-31T16:36:05Z)
• Reinforced Model Merging [53.84354455400038]
  We present an innovative framework termed Reinforced Model Merging (RMM), which encompasses an environment and agent tailored for merging tasks. By utilizing data subsets during the evaluation process, we addressed the bottleneck in the reward feedback phase, thereby accelerating RMM by up to 100 times.
  arXiv  Detail & Related papers  (2025-03-27T08:52:41Z)
• Understanding R1-Zero-Like Training: A Critical Perspective [38.515771096651356]
  We critically examine R1-Zero-like training by analyzing its two core components: base models and RL. We investigate a wide range of base models, including DeepSeek-V3-Base, to understand how pretraining characteristics influence RL performance. We present a minimalist R1-Zero recipe that achieves 43.3% accuracy on AIME 2024 with a 7B base model.
  arXiv  Detail & Related papers  (2025-03-26T17:59:14Z)
• Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models [24.45348222168512]
  We propose the reasoning MLLM, Vision-R1, to improve multimodal reasoning capability. Our model achieves an average improvement of $sim$6% across various multimodal math reasoning benchmarks. Vision-R1-7B achieves a 73.5% accuracy on the widely used MathVista benchmark, which is only 0.4% lower than the leading reasoning model, OpenAI O1.
  arXiv  Detail & Related papers  (2025-03-09T20:06:45Z)
• Zeroth-order Informed Fine-Tuning for Diffusion Model: A Recursive Likelihood Ratio Optimizer [9.153197757307762]
  probabilistic diffusion model (DM) is a powerful framework for visual generation.<n>How to efficiently align the foundation DM is a crucial task.<n>We propose the Recursive Likelihood Ratio (RLR), a zeroth-order informed fine-tuning paradigm for DM.
  arXiv  Detail & Related papers  (2025-02-02T03:00:26Z)
• Efficient Neural Theorem Proving via Fine-grained Proof Structure Analysis [50.020850767257095]
  We propose ProofAug, a novel theorem proving method that enjoys superior sample efficiency.<n>Our method is validated on the miniF2F-test benchmark using the open-source deepseek-math-7b-base model and the Isabelle proof assistant.
  arXiv  Detail & Related papers  (2025-01-30T12:37:06Z)
• Chain-of-Retrieval Augmented Generation [72.06205327186069]
  This paper introduces an approach for training o1-like RAG models that retrieve and reason over relevant information step by step before generating the final answer.<n>Our proposed method, CoRAG, allows the model to dynamically reformulate the query based on the evolving state.
  arXiv  Detail & Related papers  (2025-01-24T09:12:52Z)
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning [147.16121855209246]
  We introduce our first-generation reasoning models, DeepSeek-R1-Zero and DeepSeek-R1.<n>DeepSeek-R1-Zero is trained via large-scale reinforcement learning.<n>DeepSeek-R1 incorporates multi-stage training and cold-start data before RL.
  arXiv  Detail & Related papers  (2025-01-22T15:19:35Z)
• Deep Companion Learning: Enhancing Generalization Through Historical Consistency [35.5237083057451]
  We propose a novel training method for Deep Neural Networks (DNNs) that enhances generalization by penalizing inconsistent model predictions. We train a deep-companion model (DCM) by using previous versions of the model to provide forecasts on new inputs. This companion model deciphers a meaningful latent semantic structure within the data, thereby providing targeted supervision.
  arXiv  Detail & Related papers  (2024-07-26T15:31:13Z)
• REBEL: Reinforcement Learning via Regressing Relative Rewards [59.68420022466047]
  We propose REBEL, a minimalist RL algorithm for the era of generative models.<n>In theory, we prove that fundamental RL algorithms like Natural Policy Gradient can be seen as variants of REBEL.<n>We find that REBEL provides a unified approach to language modeling and image generation with stronger or similar performance as PPO and DPO.
  arXiv  Detail & Related papers  (2024-04-25T17:20:45Z)
• Interpolation for Robust Learning: Data Augmentation on Wasserstein Geodesics [38.81209454516577]
  We propose to study and promote the robustness of a model as per its performance through the categories of training data distributions. Specifically, we augment the data by finding the worst-case Wasserstein barycenter on the geodesic connecting subpopulation distributions. We regularize the model for smoother performance on the continuous geodesic path connecting subpopulation distributions.
  arXiv  Detail & Related papers  (2023-02-04T04:52:22Z)
• Unifying Language Learning Paradigms [96.35981503087567]
  We present a unified framework for pre-training models that are universally effective across datasets and setups. We show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization.
  arXiv  Detail & Related papers  (2022-05-10T19:32:20Z)
• LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval [55.097573036580066]
  Experimental results show that LaPraDoR achieves state-of-the-art performance compared with supervised dense retrieval models. Compared to re-ranking, our lexicon-enhanced approach can be run in milliseconds (22.5x faster) while achieving superior performance.
  arXiv  Detail & Related papers  (2022-03-11T18:53:12Z)
• SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval [11.38022203865326]
  SPLADE model provides highly sparse representations and competitive results with respect to state-of-the-art dense and sparse approaches. We modify the pooling mechanism, benchmark a model solely based on document expansion, and introduce models trained with distillation. Overall, SPLADE is considerably improved with more than $9$% gains on NDCG@10 on TREC DL 2019, leading to state-of-the-art results on the BEIR benchmark.
  arXiv  Detail & Related papers  (2021-09-21T10:43:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.