SketchVL: Policy Optimization via Fine-Grained Credit Assignment for Chart Understanding and More

• URL: http://arxiv.org/abs/2601.05688v1
• Date: Fri, 09 Jan 2026 10:13:01 GMT
• Title: SketchVL: Policy Optimization via Fine-Grained Credit Assignment for Chart Understanding and More
• Abstract summary: We introduce SketchVL, a novel MLLM that optimized with FinePO, a new RL algorithm designed for fine-grained credit assignment within each trajectory.<n>During training, the FinePO algorithm leverages a Fine-grained Process Reward Model (FinePRM) to score each drawing action within a trajectory, thereby precisely assigning credit for each step.<n>Experiments show that SketchVL learns to align its step-level behavior with the FinePRM, achieving an average performance gain of 7.23% over its base model.
• Score: 15.102512433806751
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Charts are high-density visual carriers of complex data and medium for information extraction and analysis. Due to the need for precise and complex visual reasoning, automated chart understanding poses a significant challenge to existing Multimodal Large Language Models (MLLMs). Many MLLMs trained with reinforcement learning (RL) face the challenge of credit assignment. Their advantage estimation, typically performed at the trajectory level, cannot distinguish between correct and incorrect reasoning steps within a single generated response. To address this limitation, we introduce SketchVL, a novel MLLM that optimized with FinePO, a new RL algorithm designed for fine-grained credit assignment within each trajectory. SketchVL's methodology involves drawing its intermediate reasoning steps as markers on the image and feeding the annotated image back to itself, creating a robust, multi-step reasoning process. During training, the FinePO algorithm leverages a Fine-grained Process Reward Model (FinePRM) to score each drawing action within a trajectory, thereby precisely assigning credit for each step. This mechanism allows FinePO to more strongly reward correct tokens when a trajectory is globally successful, and more heavily penalize incorrect tokens when the trajectory is globally suboptimal, thus achieving fine-grained reinforcement signals. Experiments show that SketchVL learns to align its step-level behavior with the FinePRM, achieving an average performance gain of 7.23\% over its base model across chart datasets, natural image datasets, and mathematics, providing a promising new direction for training powerful reasoning models.

Related papers

• dVLA-RL: Reinforcement Learning over Denoising Trajectories for Discrete Diffusion Vision-Language-Action Models [49.497309561043004]
  We propose textbfdVLA-RL, shifting the learning objective from the marginal action probability to the joint probability of the sampled generation path.<n>Our approach achieves a success rate of textbf99.7% on LIBERO.<n>It also establishes strong VLA-based results on RoboTwin 2.0 by delivering a textbf30.6% improvement over the SFT baseline.
  arXiv  Detail & Related papers  (2026-06-22T17:19:03Z)
• Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning [30.228760269449072]
  Group-based reinforcement learning (RL) methods have achieved remarkable success in improving the performance of large language models (LLMs)<n>We propose Graph-based Group Policy Optimization (GraphGPO), which aggregates all rollout trajectories into a unified state-transition graph.<n>It estimates the distance from each state to the task goal using the global information encoded in the graph.<n>Finally, GraphGPO assigns credit to each edge by estimating a graph-based advantage, based on how much the transition reduces the distance to the task goal.
  arXiv  Detail & Related papers  (2026-05-26T08:23:34Z)
• Sketch Then Paint: Hierarchical Reinforcement Learning for Diffusion Multi-Modal Large Language Models [52.40742159500277]
  We show how to optimize Diffusion Multi-Modal Large Language Models (dMLLMs) through reinforcement learning (RL)<n>Our approach features a Sketch-Then-Paint training scheme that organizes updates into three distinct stages: global, structure, and refinement.<n> Experiments using two popular dMLLM backbones, MMaDA and Lumina-DiMOO, demonstrate substantial gains on the GenEval and DPG benchmarks.
  arXiv  Detail & Related papers  (2026-05-16T06:59:54Z)
• Chart-RL: Policy Optimization Reinforcement Learning for Enhanced Visual Reasoning in Chart Question Answering with Vision Language Models [2.1902845922631435]
  Vision Language Models (VLMs) have demonstrated progress toward true intelligence requiring robust reasoning capabilities.<n>Current VLMs face significant limitations in Chart Question Answering (CQA) tasks involving complex data visualizations.<n>We present Chart-RL, a novel reinforcement learning framework that enhances chart understanding through feedback-driven policy optimization.
  arXiv  Detail & Related papers  (2026-04-03T16:28:03Z)
• ProRAG: Process-Supervised Reinforcement Learning for Retrieval-Augmented Generation [54.071574153853994]
  ProRAG is a process-supervised reinforcement learning framework designed to integrate learned step-level supervision into the online optimization loop.<n>Our framework consists of four stages: (1) Supervised Policy Warmup to initialize the model with a structured reasoning format; (2) construction of an MCTS-based Process Reward Model (PRM) to quantify intermediate reasoning quality; (3) PRM-Guided Reasoning Refinement to align the policy with fine-grained process preferences; and (4) Process-Supervised Reinforcement Learning with a dual-granularity advantage mechanism.
  arXiv  Detail & Related papers  (2026-01-29T16:04:59Z)
• Policies over Poses: Reinforcement Learning based Distributed Pose-Graph Optimization for Multi-Robot SLAM [1.3750624267664158]
  We consider the sequential distributed pose-graph optimization (PGO) in multi-robot localization.<n>We cast PGO as a partially observable game defined on a Markov Neural Network (GNN), where each action refines a single edge's pose estimate.<n>We show that our learned actors reduce the global trajectory by an average of 37.5%, while enhancing efficiency by at least 6X.
  arXiv  Detail & Related papers  (2025-10-26T16:21:24Z)
• Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization [56.083511902353365]
  Reinforcement learning (RL) typically applies uniform credit across an entire generation of Large language models.<n>This work positions attention as a privileged substrate that renders the internal logic of LLMs as a mechanistic blueprint of reasoning itself.<n>We introduce three novel RL strategies that dynamically perform targeted credit assignment to critical nodes.
  arXiv  Detail & Related papers  (2025-10-15T13:49:51Z)
• Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generation [12.822184232115333]
  We propose Multimodal Structured Reinforcement Learning (MSRL) for chart-to-code generation.<n>We construct the largest training corpus to date, containing 3 million chart-code pairs from real-world arXiv tables.<n>MSRL significantly breaks the SFT plateau, improving high-level metrics by 6.2% and 9.9% on ChartMimic and ReachQA benchmarks respectively.
  arXiv  Detail & Related papers  (2025-08-19T07:40:18Z)
• CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment [44.33395106709674]
  Reinforcement Learning with Verifiable Rewards (RLVR) has improved the reasoning abilities of Large Language Models (LLMs) by using rule-based binary feedback.<n>Current RLVR methods typically assign the same reward to every token.<n>This coarse-grained feedback hampers precise credit assignment, making it hard for models to identify which reasoning steps lead to success or failure.
  arXiv  Detail & Related papers  (2025-08-04T11:06:08Z)
• Compile Scene Graphs with Reinforcement Learning [69.36723767339001]
  Next-token prediction is the fundamental principle for training large language models (LLMs)<n>We introduce R1-SGG, a multimodal LLM (M-LLM) initially trained via supervised fine-tuning (SFT) on the scene graph dataset.<n>We design a set of graph-centric rewards, including three recall-based variants -- Hard Recall, Hard Recall+Relax, and Soft Recall.
  arXiv  Detail & Related papers  (2025-04-18T10:46:22Z)
• SPARC: Score Prompting and Adaptive Fusion for Zero-Shot Multi-Label Recognition in Vision-Language Models [74.40683913645731]
  Zero-shot multi-label recognition (MLR) with Vision-Language Models (VLMs) faces significant challenges without training data, model tuning, or architectural modifications.<n>Our work proposes a novel solution treating VLMs as black boxes, leveraging scores without training data or ground truth.<n>Analysis of these prompt scores reveals VLM biases and AND''/OR' signal ambiguities, notably that maximum scores are surprisingly suboptimal compared to second-highest scores.
  arXiv  Detail & Related papers  (2025-02-24T07:15:05Z)
• VinePPO: Refining Credit Assignment in RL Training of LLMs [66.80143024475635]
  We propose VinePPO, a straightforward approach that leverages the flexibility of language environments to compute unbiased Monte Carlo-based estimates.<n>Our method consistently outperforms PPO and other baselines across MATH and GSM8K datasets in less wall-clock time.
  arXiv  Detail & Related papers  (2024-10-02T15:49:30Z)
• From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems [59.40480894948944]
  Large language model (LLM) empowered agents are able to solve decision-making problems in the physical world. Under this model, the LLM Planner navigates a partially observable Markov decision process (POMDP) by iteratively generating language-based subgoals via prompting. We prove that the pretrained LLM Planner effectively performs Bayesian aggregated imitation learning (BAIL) through in-context learning.
  arXiv  Detail & Related papers  (2024-05-30T09:42:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.