Enhancing Action and Ingredient Modeling for Semantically Grounded Recipe Generation

• URL: http://arxiv.org/abs/2602.15862v1
• Date: Mon, 26 Jan 2026 10:06:18 GMT
• Title: Enhancing Action and Ingredient Modeling for Semantically Grounded Recipe Generation
• Abstract summary: We propose a semantically grounded framework that predicts and validates actions and ingredients as internal context for instruction generation.<n> Experiments on Recipe1M show state-of-the-art performance and markedly improved semantic fidelity.
• Score: 69.90401008542368
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Recent advances in Multimodal Large Language Models (MLMMs) have enabled recipe generation from food images, yet outputs often contain semantically incorrect actions or ingredients despite high lexical scores (e.g., BLEU, ROUGE). To address this gap, we propose a semantically grounded framework that predicts and validates actions and ingredients as internal context for instruction generation. Our two-stage pipeline combines supervised fine-tuning (SFT) with reinforcement fine-tuning (RFT): SFT builds foundational accuracy using an Action-Reasoning dataset and ingredient corpus, while RFT employs frequency-aware rewards to improve long-tail action prediction and ingredient generalization. A Semantic Confidence Scoring and Rectification (SCSR) module further filters and corrects predictions. Experiments on Recipe1M show state-of-the-art performance and markedly improved semantic fidelity.

Related papers

• Hierarchical Exponential-Gaussian Mixtures for Watch-Time Distribution Prediction [74.05279072166697]
  Exponential-Gaussian Mixture Network (EGMN) models the full conditional WT distribution.<n>EGMN is vulnerable to variance collapse, component redundancy, and inactive components.<n>We propose a Hierarchical Exponential-Gaussian Mixture (HEGM) model that addresses these failure modes.
  arXiv  Detail & Related papers  (2026-08-24T15:06:18Z)
• ReGraph: Learning to Generate Recipe Graphs from Food Images [70.27418607642029]
  We present ReGraph, a large-scale recipe graph dataset that represents ingredients, cooking actions, and tools as entities.<n>We propose Recipe Graph Learning (RGL), a two-stage framework that enables LMMs to generate a plausible fine-grained cooking workflow from a food image.
  arXiv  Detail & Related papers  (2026-08-07T07:51:08Z)
• Beyond Instance-Level Alignment and Uniformity: Semantic Factor Learning for Collaborative Filtering [1.5984637249476494]
  Collaborative filtering (CF) is widely used in recommender systems (RecSys) due to its simplicity and efficiency.<n>Existing CF methods follow an instance-level learning paradigm.<n>We propose Semantic Factor enhanced Alignment and Uniformity (SaFeAU), a novel framework that augments interacted instances with semantic factors.<n>SaFeAU consistently outperforms GCN-based and MF-based state-of-the-art CF methods in both recommendation accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2026-05-29T15:15:21Z)
• Towards Order Fairness: Mitigating LLMs Order Sensitivity through Dual Group Advantage Optimization [20.259122922188126]
  Large Language Models (LLMs) suffer from order bias, where their performance is affected by the arrangement order of input elements.<n>textbfDGAO aims to improve model accuracy and order stability simultaneously.
  arXiv  Detail & Related papers  (2026-05-12T11:31:18Z)
• CGU-ILALab at FoodBench-QA 2026: Comparing Traditional and LLM-based Approaches for Recipe Nutrient Estimation [10.751176062286321]
  Accurate nutrient estimation from unstructured recipe text is an important yet challenging problem in dietary monitoring.<n>We evaluate models spanning a wide range of representational capacity, from lexical matching methods to deep semantic encoders.<n>Our results reveal a clear trade-off between predictive accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2026-04-28T15:41:01Z)
• Fine-tuning MLLMs Without Forgetting Is Easier Than You Think [72.59321247529975]
  We design a 2x2 experimental framework to assess model performance across in-distribution and out-of-distribution image and text inputs.<n>Our results show that appropriate regularization, such as constraining the number of trainable parameters or adopting a low learning rate, effectively prevents forgetting when dealing with out-of-distribution images.<n>We attribute this forgetting as task-specific overfitting and address this issue by introducing a data-hybrid training strategy.
  arXiv  Detail & Related papers  (2026-03-15T17:16:19Z)
• Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models [39.290072292743226]
  We introduce Mix-GRM, a framework that reconfigures raw rationales into structured B-CoT and D-CoT through a modular synthesis pipeline.<n>Experiments demonstrate that Mix-GRM establishes a new state-of-the-art across five benchmarks, surpassing leading open-source RMs by an average of 8.2%.
  arXiv  Detail & Related papers  (2026-03-02T07:54:29Z)
• Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocols [123.73663884421272]
  Few-shot transfer has been revolutionized by stronger pre-trained models and improved adaptation algorithms.<n>We establish FEWTRANS, a comprehensive benchmark containing 10 diverse datasets.<n>By releasing FEWTRANS, we aim to provide a rigorous "ruler" to streamline reproducible advances in few-shot transfer learning research.
  arXiv  Detail & Related papers  (2026-02-28T05:41:57Z)
• Toward Faithful and Complete Answer Construction from a Single Document [1.0742675209112622]
  We present EVE, a structured framework for document-grounded reasoning.<n>Unlike free-form prompting, EVE constrains generation to a structured, verifiable pipeline that decomposes high-rigor reasoning into extraction, validation, and enumeration.
  arXiv  Detail & Related papers  (2026-02-05T18:22:08Z)
• GLOW: Graph-Language Co-Reasoning for Agentic Workflow Performance Prediction [51.83437071408662]
  We propose GLOW, a unified framework for AW performance prediction.<n>GLOW combines the graph-structure modeling capabilities of GNNs with the reasoning power of LLMs.<n>Experiments on FLORA-Bench show that GLOW outperforms state-of-the-art baselines in prediction accuracy and ranking utility.
  arXiv  Detail & Related papers  (2025-12-11T13:30:46Z)
• SynCast: Synergizing Contradictions in Precipitation Nowcasting via Diffusion Sequential Preference Optimization [62.958457694151384]
  We introduce preference optimization into precipitation nowcasting for the first time, motivated by the success of reinforcement learning from human feedback in large language models.<n>In the first stage, the framework focuses on reducing FAR, training the model to effectively suppress false alarms.
  arXiv  Detail & Related papers  (2025-10-22T16:11:22Z)
• Sculpting Features from Noise: Reward-Guided Hierarchical Diffusion for Task-Optimal Feature Transformation [18.670626228472877]
  DIFFT redefines Feature Transformation as a reward-guided generative task.<n>It produces structured, discrete features, preserving intra-feature dependencies while allowing parallel inter-feature generation.<n>It consistently outperforms state-of-the-art baselines in predictive accuracy and robustness, with significantly lower training and inference times.
  arXiv  Detail & Related papers  (2025-05-21T06:18:42Z)
• Harnessing Mixed Features for Imbalance Data Oversampling: Application to Bank Customers Scoring [5.091061468748012]
  We introduce MGS-GRF, an oversampling strategy designed for mixed features.<n>We show that MGS-GRF exhibits two important properties: (i) the coherence i.e. the ability to only generate combinations of categorical features that are already present in the original dataset and (ii) association, i.e. the ability to preserve the dependence between continuous and categorical features.
  arXiv  Detail & Related papers  (2025-03-26T08:53:40Z)
• ROSE: Revolutionizing Open-Set Dense Segmentation with Patch-Wise Perceptual Large Multimodal Model [75.750699619993]
  We propose ROSE, a Revolutionary Open-set dense SEgmentation LMM, which enables dense mask prediction and open-category generation.<n>Our method treats each image patch as an independent region of interest candidate, enabling the model to predict both dense and sparse masks simultaneously.
  arXiv  Detail & Related papers  (2024-11-29T07:00:18Z)
• Retrieval Augmented Recipe Generation [96.43285670458803]
  We propose a retrieval augmented large multimodal model for recipe generation.<n>It retrieves recipes semantically related to the image from an existing datastore as a supplement.<n>It calculates the consistency among generated recipe candidates, which use different retrieval recipes as context for generation.
  arXiv  Detail & Related papers  (2024-11-13T15:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.