FoodMLLM-JP: Leveraging Multimodal Large Language Models for Japanese Recipe Generation

• URL: http://arxiv.org/abs/2409.18459v2
• Date: Mon, 03 Mar 2025 15:04:18 GMT
• Title: FoodMLLM-JP: Leveraging Multimodal Large Language Models for Japanese Recipe Generation
• Abstract summary: We fine-tuned open MLLMs LLaVA-1.5 and Phi-3 Vision on a Japanese recipe dataset and benchmarked their performance against the closed model GPT-4o.<n>Our evaluation demonstrates that the open models trained on recipe data outperform GPT-4o, the current state-of-the-art model, in ingredient generation.
• Score: 30.349846688239293
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Research on food image understanding using recipe data has been a long-standing focus due to the diversity and complexity of the data. Moreover, food is inextricably linked to people's lives, making it a vital research area for practical applications such as dietary management. Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities, not only in their vast knowledge but also in their ability to handle languages naturally. While English is predominantly used, they can also support multiple languages including Japanese. This suggests that MLLMs are expected to significantly improve performance in food image understanding tasks. We fine-tuned open MLLMs LLaVA-1.5 and Phi-3 Vision on a Japanese recipe dataset and benchmarked their performance against the closed model GPT-4o. We then evaluated the content of generated recipes, including ingredients and cooking procedures, using 5,000 evaluation samples that comprehensively cover Japanese food culture. Our evaluation demonstrates that the open models trained on recipe data outperform GPT-4o, the current state-of-the-art model, in ingredient generation. Our model achieved F1 score of 0.531, surpassing GPT-4o's F1 score of 0.481, indicating a higher level of accuracy. Furthermore, our model exhibited comparable performance to GPT-4o in generating cooking procedure text.

Related papers

• NutriMLLM: Multimodal Large Language Models for Dietary Micronutrient Analysis [4.334404218740802]
  We show that existing multimodal large language models (MLLMs) are unreliable for comprehensive micronutrient estimation.<n>To address this gap without costly expert annotation, we repurposed a decade of 24-hour dietary recalls as structured prompts for text-to-image generation.<n>This pipeline produced a synthetic corpus of about 1.1 million image-description-nutrient triplets, each pairing a generated food image with a complete 65-nutrient label.
  arXiv  Detail & Related papers  (2026-06-08T02:50:08Z)
• Evaluating Modern Large Language Models on Low-Resource and Morphologically Rich Languages:A Cross-Lingual Benchmark Across Cantonese, Japanese, and Turkish [12.286855282078305]
  GPT-4o, GPT-4, Claude3.5Sonnet, LLaMA3.1, MistralLarge2, LLaMA-2Chat13B, and Mistral7BInstruct are evaluated.<n>Our benchmark spans four diverse tasks: open-domain question answering, document summarization, English-to-X translation, and culturally grounded dialogue.
  arXiv  Detail & Related papers  (2025-11-05T22:09:53Z)
• Beyond Specialization: Benchmarking LLMs for Transliteration of Indian Languages [19.988253285050256]
  Transliteration plays a crucial role in multilingual natural language processing.<n>Recent developments in large language models suggest a potential for general-purpose models to excel at this task without explicit task-specific training.<n>This work systematically evaluates the performance of prominent LLMs, including GPT-4o, GPT-4.5, GPT-4.1, Gemma-3-27B-it, and Mistral-Large against IndicXlit, a state-of-the-art transliteration model, across ten major Indian languages.
  arXiv  Detail & Related papers  (2025-05-26T11:35:51Z)
• Improving Multilingual Capabilities with Cultural and Local Knowledge in Large Language Models While Enhancing Native Performance [0.0]
  We present our latest Hindi-English bi-lingual LLM textbfMantra-14B with 3% average improvement in benchmark scores over both languages. We instruction tuned models such as Qwen-2.5-14B-Instruct and Phi-4 to improve performance over both English and Hindi. Our results indicate that modest fine-tuning with culturally and locally informed data can bridge performance gaps without incurring significant computational overhead.
  arXiv  Detail & Related papers  (2025-04-13T23:10:13Z)
• When Tom Eats Kimchi: Evaluating Cultural Bias of Multimodal Large Language Models in Cultural Mixture Contexts [15.78054683369659]
  We introduce MixCuBe, a cross-cultural bias benchmark, and study elements from five countries and four ethnicities. Our findings reveal that MLLMs achieve both higher accuracy and lower sensitivity to such perturbations for high-resource cultures. GPT-4o, the best-performing model overall, shows up to 58% difference in accuracy between the original and perturbed cultural settings in low-resource cultures.
  arXiv  Detail & Related papers  (2025-03-21T03:50:05Z)
• Identifying and Decomposing Compound Ingredients in Meal Plans Using Large Language Models [6.477074442920329]
  This study explores the effectiveness of Large Language Models in meal planning, focusing on their ability to identify and decompose compound ingredients. We evaluated three models-GPT-4o, Llama-3 (70b), and Mixtral (8x7b)-to assess their proficiency in recognizing and breaking down complex ingredient combinations. Preliminary results indicate that while Llama-3 (70b) and GPT-4o excels in accurate decomposition, all models encounter difficulties with identifying essential elements like seasonings and oils.
  arXiv  Detail & Related papers  (2024-11-08T12:38:10Z)
• Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages [55.36534539177367]
  This paper introduces Pangea, a multilingual multimodal large language model (MLLM) trained on a diverse 6M instruction dataset spanning 39 languages. P Pangea significantly outperforms existing open-source models in multilingual settings and diverse cultural contexts. We fully open-source our data, code, and trained checkpoints, to facilitate the development of inclusive and robust multilingual MLLMs.
  arXiv  Detail & Related papers  (2024-10-21T16:19:41Z)
• LLaVA-Chef: A Multi-modal Generative Model for Food Recipes [17.705244174235045]
  Large language models (LLMs) have paved the way for Natural Language Processing approaches to delve deeper into food-related tasks. This work proposes LLaVA-Chef, a novel model trained on a curated dataset of diverse recipe prompts. A detailed qualitative analysis reveals that LLaVA-Chef generates more detailed recipes with precise ingredient mentions.
  arXiv  Detail & Related papers  (2024-08-29T20:20:49Z)
• CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models [53.9835961434552]
  We introduce the Chinese Instruction-Following Benchmark (CIF-Bench) to evaluate the generalizability of large language models (LLMs) to the Chinese language. CIF-Bench comprises 150 tasks and 15,000 input-output pairs, developed by native speakers to test complex reasoning and Chinese cultural nuances. To mitigate data contamination, we release only half of the dataset publicly, with the remainder kept private, and introduce diversified instructions to minimize score variance.
  arXiv  Detail & Related papers  (2024-02-20T16:02:12Z)
• FoodLMM: A Versatile Food Assistant using Large Multi-modal Model [96.76271649854542]
  Large Multi-modal Models (LMMs) have made impressive progress in many vision-language tasks. This paper proposes FoodLMM, a versatile food assistant based on LMMs with various capabilities. We introduce a series of novel task-specific tokens and heads, enabling the model to predict food nutritional values and multiple segmentation masks.
  arXiv  Detail & Related papers  (2023-12-22T11:56:22Z)
• Dietary Assessment with Multimodal ChatGPT: A Systematic Analysis [17.333822848423708]
  This study explores the application of multimodal ChatGPT within the realm of dietary assessment. By guiding the model with specific language prompts, GPT-4V shifts from recognizing common staples like rice and bread to accurately identifying regional dishes like banku and ugali. GPT-4V can leverage surrounding objects as scale references to deduce the portion sizes of food items, further enhancing its accuracy in translating food weight into nutritional content.
  arXiv  Detail & Related papers  (2023-12-14T01:26:45Z)
• Efficient Finetuning Large Language Models For Vietnamese Chatbot [1.2075778142867704]
  Large language models (LLMs) have been shown to achieve remarkable performance across a variety of natural language tasks. We leverage large-scale instruction-following datasets from open-source projects, namely Alpaca, GPT4All, and Chat-Doctor. We utilize parameter-efficient tuning through Low-Rank Adaptation (LoRA) on two open LLMs, resulting four models: Bloomz-Chat, Bloomz-Doctor, GPTJ-Chat, GPTJ-Doctor.
  arXiv  Detail & Related papers  (2023-09-09T00:11:53Z)
• LLM-powered Data Augmentation for Enhanced Cross-lingual Performance [24.20730298894794]
  This paper explores the potential of leveraging Large Language Models (LLMs) for data augmentation in commonsense reasoning datasets. To achieve this, we utilise several LLMs, namely Dolly-v2, StableVicuna, ChatGPT, and GPT-4, to augment three datasets: XCOPA, XWinograd, and XStoryCloze. We evaluate the effectiveness of fine-tuning smaller multilingual models, mBERT and XLMR, using the synthesised data.
  arXiv  Detail & Related papers  (2023-05-23T17:33:27Z)
• Visual Instruction Tuning [79.70923292053097]
  We present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%.
  arXiv  Detail & Related papers  (2023-04-17T17:59:25Z)
• Building Machine Translation Systems for the Next Thousand Languages [102.24310122155073]
  We describe results in three research domains: building clean, web-mined datasets for 1500+ languages, developing practical MT models for under-served languages, and studying the limitations of evaluation metrics for these languages. We hope that our work provides useful insights to practitioners working towards building MT systems for currently understudied languages, and highlights research directions that can complement the weaknesses of massively multilingual models in data-sparse settings.
  arXiv  Detail & Related papers  (2022-05-09T00:24:13Z)
• PaLM: Scaling Language Modeling with Pathways [180.69584031908113]
  We trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks.
  arXiv  Detail & Related papers  (2022-04-05T16:11:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.