PERSA: Reinforcement Learning for Professor-Style Personalized Feedback with LLMs

• URL: http://arxiv.org/abs/2605.01123v1
• Date: Fri, 01 May 2026 21:49:20 GMT
• Title: PERSA: Reinforcement Learning for Professor-Style Personalized Feedback with LLMs
• Abstract summary: We study how Reinforcement Learning from Human Feedback can adapt a transformer-based LLM to generate programming feedback that matches a professors grading voice.<n>We introduce PERSA, an RLHF pipeline that combines supervised fine-tuning on professor demonstrations, reward modeling from pairwise preferences, and Proximal Policy Optimization.<n>We evaluate our proposed approach on three code-feedback benchmarks (APPS, PyFiXV, and CodeReviewQA) using complementary metrics for style alignment and fidelity.
• Score: 1.8986796884429726
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) can provide automated feedback in educational settings, but aligning an LLMs style with a specific instructors tone while maintaining diagnostic correctness remains challenging. We ask how can we update an LLM for automated feedback generation to align with a target instructors style without sacrificing core knowledge? We study how Reinforcement Learning from Human Feedback (RLHF) can adapt a transformer-based LLM to generate programming feedback that matches a professors grading voice. We introduce PERSA, an RLHF pipeline that combines supervised fine-tuning on professor demonstrations, reward modeling from pairwise preferences, and Proximal Policy Optimization (PPO), while deliberately constraining learning to style-bearing components. Motivated by analyses of transformer internals, PERSA applies parameter efficient fine-tuning. It updates only the top transformer blocks and their feed-forward projections, minimizing global parameter drift while increasing stylistic controllability. We evaluate our proposed approach on three code-feedback benchmarks (APPS, PyFiXV, and CodeReviewQA) using complementary metrics for style alignment and fidelity. Across both Llama-3 and Gemma-2 backbones, PERSA delivers the strongest professor-style transfer while retaining correctness, for example on APPS, it boosts Style Alignment Score (SAC) to 96.2% (from 34.8% for Base) with Correctness Accuracy (CA) up to 100% on Llama-3, and Gemma-2. Overall, PERSA offers a practical route to personalized educational feedback by aligning both what it says (content correctness) and, crucially, how it says it (instructor-like tone and structure).

Related papers

• Listen, Correct, and Feed Back: Spoken Pedagogical Feedback Generation [25.967830220467633]
  We introduce textbfSPFG (textbfSpoken textbfPedagogical textbfFeedback textbfGeneration), a dataset built based on the Speak & Improve Challenge 2025 corpus.<n>We study a transcript-based Grammatical Error Correction (SGEC) setting and evaluate three instruction-tuned LLMs (Qwen2.5, Llama-3.1, and GLM-4)<n>Results show that SFT provides the most consistent improvements, while preference-based alignment (using DPO and KTO
  arXiv  Detail & Related papers  (2026-03-28T07:47:29Z)
• Expanding the Capabilities of Reinforcement Learning via Text Feedback [49.561885700139676]
  We formalize a multi-turn RL setup, RL from Text Feedback (RLTF), where text feedback is available during training but not at inference.<n>To do this, we propose two methods: Self Distillation (RLTF-SD), which trains the single-turn policy to match its own feedback-conditioned second-turn generations; and Feedback Modeling (RLTF-FM), which predicts the feedback as an auxiliary objective.<n>Our results show that both methods consistently outperform strong baselines across benchmarks.
  arXiv  Detail & Related papers  (2026-02-02T18:56:56Z)
• Language Models Can Learn from Verbal Feedback Without Scalar Rewards [88.82702433508393]
  We propose treating verbal feedback as a conditioning signal.<n>Inspired by language priors in text-to-image generation, we introduce the feedback-conditional policy.
  arXiv  Detail & Related papers  (2025-09-26T17:58:27Z)
• Can Large Language Models Match Tutoring System Adaptivity? A Benchmarking Study [0.0]
  Large Language Models (LLMs) hold promise as dynamic instructional aids.<n>Yet, it remains unclear whether LLMs can replicate the adaptivity of intelligent tutoring systems (ITS)
  arXiv  Detail & Related papers  (2025-04-07T23:57:32Z)
• Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Model [17.4036850872656]
  Preference-based reinforcement learning (PbRL) provides a powerful paradigm to avoid meticulous reward engineering by learning rewards based on human preferences.<n>In this paper, we propose a RL Self-augmented Large Language Model Feedback (RL-SaLLM-F) technique that does not rely on privileged information for online PbRL.
  arXiv  Detail & Related papers  (2024-12-22T06:15:25Z)
• Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models [54.14602121129874]
  We introduce AutoIF, the first scalable and reliable method for automatically generating instruction-following training data. AutoIF transforms the validation of instruction-following data quality into code verification.
  arXiv  Detail & Related papers  (2024-06-19T13:29:53Z)
• Improving the Validity of Automatically Generated Feedback via Reinforcement Learning [46.667783153759636]
  We propose a framework for feedback generation that optimize both correctness and alignment using reinforcement learning (RL)<n>Specifically, we use GPT-4's annotations to create preferences over feedback pairs in an augmented dataset for training via direct preference optimization (DPO)
  arXiv  Detail & Related papers  (2024-03-02T20:25:50Z)
• Direct Language Model Alignment from Online AI Feedback [78.40436231613754]
  Direct alignment from preferences (DAP) methods have recently emerged as efficient alternatives to reinforcement learning from human feedback (RLHF) In this study, we posit that online feedback is key and improves DAP methods. Our method, online AI feedback (OAIF) uses an LLM as annotator: on each training, we sample two responses from the current model and prompt the LLM annotator to choose which one is preferred, thus providing online feedback.
  arXiv  Detail & Related papers  (2024-02-07T12:31:13Z)
• LiPO: Listwise Preference Optimization through Learning-to-Rank [62.02782819559389]
  Policy can learn more effectively from a ranked list of plausible responses given the prompt.<n>We show that LiPO-$lambda$ can outperform DPO variants and SLiC by a clear margin on several preference alignment tasks.
  arXiv  Detail & Related papers  (2024-02-02T20:08:10Z)
• Dense Reward for Free in Reinforcement Learning from Human Feedback [64.92448888346125]
  We leverage the fact that the reward model contains more information than just its scalar output. We use these attention weights to redistribute the reward along the whole completion. Empirically, we show that it stabilises training, accelerates the rate of learning, and, in practical cases, may lead to better local optima.
  arXiv  Detail & Related papers  (2024-02-01T17:10:35Z)
• Interpreting Learned Feedback Patterns in Large Language Models [31.133685891389774]
  We train probes to estimate the feedback signal implicit in the activations of a fine-tuned language model.<n>We compare these estimates to the true feedback, measuring how accurate the LFPs are to the fine-tuning feedback.<n>We validate our probes by comparing the neural features they correlate with positive feedback inputs against the features GPT-4 describes and classifies as related to LFPs.
  arXiv  Detail & Related papers  (2023-10-12T09:36:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.