The Order Matters: Sequential Fine-Tuning of LLaMA for Coherent Automated Essay Scoring

• URL: http://arxiv.org/abs/2606.10327v1
• Date: Tue, 09 Jun 2026 02:23:46 GMT
• Title: The Order Matters: Sequential Fine-Tuning of LLaMA for Coherent Automated Essay Scoring
• Abstract summary: We investigate task-aware fine-tuning of LLaMA-3.1-8B for AES using parameter-efficient LoRA with 4-bit quantization.<n>We compare three training curricula: (i) Sequential fine-tuning on lead, then position, then claim, then evidence, then conclusion, (ii) Independent (task-specific models), and (iii) Randomized.
• Score: 2.6996154332513895
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automated Essay Scoring (AES) systems must judge interdependent discourse elements (e.g., lead, claim, evidence, conclusion), yet most approaches treat these in isolation, harming coherence and generalization. We investigate task-aware fine-tuning of LLaMA-3.1-8B for AES using parameter-efficient LoRA with 4-bit quantization and compare three training curricula: (i) Sequential (progressively fine-tuning on lead, then position, then claim, then evidence, then conclusion), (ii) Independent (task-specific models), and (iii) Randomized (shuffled multi-task). Experiments on the PERSUADE~2.0 corpus show that modeling task dependencies matters: Sequential fine-tuning yields the strongest overall results, including F1 scores of 65% (evidence) and 87% (conclusion) and corresponding accuracies of 63% and 85%, surpassing Independent training and outperforming a general-purpose LLaMA-70B baseline on conclusion despite its far larger capacity. Randomized training improves position scoring (57% F1) but is less consistent elsewhere. These findings indicate that (1) curriculum design aligned with discourse structure can materially improve AES, and (2) small, task-optimized models can be competitive with substantially larger Large Language Models (LLM), offering a practical path to scalable, cost-effective assessment. We release templates and implementation details to facilitate reproduction and future work on curriculum design for educational NLP.

Related papers

• Simplicity Paradox: Debunking myths about prompting and datasets for LLM evaluation [2.221515121005437]
  Several studies claim better performance with more sophisticated prompting techniques, but do not provide a comprehensive evaluation.<n>We address this gap through a comprehensive empirical study of 8 prompting techniques across 10 multiple-choice question answering (MCQA) datasets.<n>Our findings reveal a striking paradox that baseline prompting consistently outperforms complex reasoning techniques on various benchmarks.
  arXiv  Detail & Related papers  (2026-05-07T21:11:51Z)
• Scaling Trends for Multi-Hop Contextual Reasoning in Mid-Scale Language Models [0.0]
  We present a controlled study of multi-hop contextual reasoning in large language models.<n>We show that multi-agent systems show the inverse pattern, achieving up to 80% on reasoning tasks where rule-based methods fail.
  arXiv  Detail & Related papers  (2026-01-06T20:18:55Z)
• Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling [90.87033586963828]
  Outcome-reward reinforcement learning (RL) is a common and increasingly significant way to refine the step-by-step reasoning of multimodal large language models (MLLMs)<n>We propose Self-Consistency Sampling (SCS) to correct this issue.<n>Based on Qwen2.5-VL-7B-Instruct, SCS improves accuracy by up to 7.7 percentage points on six multimodal benchmarks with negligible extra computation.
  arXiv  Detail & Related papers  (2025-11-13T18:59:57Z)
• Structured Uncertainty guided Clarification for LLM Agents [126.26213027785813]
  LLM agents extend large language models with tool-calling capabilities, but ambiguous user instructions often lead to incorrect invocations and task failures.<n>We introduce a principled formulation of structured uncertainty over tool-call parameters, modeling joint tool-argument clarification as a POMDP with Expected Value of Perfect Information (EVPI) objective for optimal question selection and aspect-based cost modeling to prevent redundancy.<n>Our SAGE-Agent leverages this structured uncertainty to achieve superior efficiency: increasing coverage on ambiguous tasks by 7-39% while reducing clarification questions by 1.5-2.7$times$ compared to strong prompting and uncertainty-based baselines.
  arXiv  Detail & Related papers  (2025-11-11T21:50:44Z)
• Once Upon an Input: Reasoning via Per-Instance Program Synthesis [19.86168542588911]
  We introduce Per-Instance Program Synthesis (PIPS), a method that generates and refines programs at the instance-level using structural feedback.<n>To further improve performance, PIPS incorporates a confidence metric that dynamically chooses between direct inference and program synthesis on a per-instance basis.
  arXiv  Detail & Related papers  (2025-10-26T21:58:33Z)
• AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models [86.83875864328984]
  We propose an automated method for synthesizing open-ended logic puzzles, and use it to develop a bilingual benchmark, AutoLogi.<n>Our approach features program-based verification and controllable difficulty levels, enabling more reliable evaluation that better distinguishes models' reasoning abilities.
  arXiv  Detail & Related papers  (2025-02-24T07:02:31Z)
• LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs [103.0226977561914]
  We propose a comprehensive framework for advancing step-by-step visual reasoning in large language models.<n>We introduce a visual reasoning benchmark specifically designed to evaluate multi-step reasoning tasks.<n>Second, we propose a novel metric that assesses visual reasoning quality at the granularity of individual steps.<n>Third, we present a new multimodal visual reasoning model, named LlamaV-o1, trained using a multi-step curriculum learning approach.
  arXiv  Detail & Related papers  (2025-01-10T18:59:51Z)
• (WhyPHI) Fine-Tuning PHI-3 for Multiple-Choice Question Answering: Methodology, Results, and Challenges [0.0]
  This work explores the potential of Microsoft's PHI-3citeAbdin2024, a compact yet efficient LLM, for answering multiple-choice questions.<n>Results show a remarkable improvement in PHI-3.5's MCQ handling post-fine-tuning, with perplexity decreasing from 4.68 to 2.27, and accuracy rising from 62% to 90.8%.
  arXiv  Detail & Related papers  (2025-01-03T00:56:46Z)
• MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale [66.73529246309033]
  multimodal large language models (MLLMs) have shown significant potential in a broad range of multimodal tasks.<n>Existing instruction-tuning datasets only provide phrase-level answers without any intermediate rationales.<n>We introduce a scalable and cost-effective method to construct a large-scale multimodal instruction-tuning dataset with rich intermediate rationales.
  arXiv  Detail & Related papers  (2024-12-06T18:14:24Z)
• The Surprising Effectiveness of Test-Time Training for Few-Shot Learning [59.309477460893916]
  Language models (LMs) have shown impressive performance on tasks within their training distribution, but often struggle with structurally novel tasks.<n>We investigate the effectiveness of test-time training (TTT) as a mechanism for improving LMs' reasoning and few-shot learning capabilities.<n>Our findings highlight the limitations of in-context learning for novel tasks and demonstrate the potential of test-time training to enhance language model adaptability.
  arXiv  Detail & Related papers  (2024-11-11T18:59:45Z)
• Advancing LLM Reasoning Generalists with Preference Trees [119.57169648859707]
  We introduce Eurus, a suite of large language models (LLMs) optimized for reasoning. Eurus models achieve state-of-the-art results among open-source models on a diverse set of benchmarks.
  arXiv  Detail & Related papers  (2024-04-02T16:25:30Z)
• L3 Ensembles: Lifelong Learning Approach for Ensemble of Foundational Language Models [15.726224465017596]
  We propose an approach that focuses on extracting meaningful representations from unseen data and constructing a structured knowledge base. We conducted experiments on various NLP tasks to validate its effectiveness, including benchmarks like GLUE and SuperGLUE. The proposed L3 ensemble method increases the model accuracy by 4% 36% compared to the fine-tuned FLM.
  arXiv  Detail & Related papers  (2023-11-11T06:59:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.