Exploring Object Status Recognition for Recipe Progress Tracking in Non-Visual Cooking

• URL: http://arxiv.org/abs/2507.03330v1
• Date: Fri, 04 Jul 2025 06:30:50 GMT
• Title: Exploring Object Status Recognition for Recipe Progress Tracking in Non-Visual Cooking
• Authors: Franklin Mingzhe Li, Kaitlyn Ng, Bin Zhu, Patrick Carrington,
• Abstract summary: We present OSCAR (Object Status Context Awareness for Recipes), a technical pipeline that explores the use of object status recognition to enable recipe progress tracking in non-visual cooking.<n> OSCAR integrates recipe parsing, object status extraction, visual alignment with cooking steps, and time-causal modeling to support real-time step tracking.<n>Our results show that object status consistently improves step prediction accuracy across vision-language models, and reveal key factors that impact performance in real-world conditions.
• Score: 24.6085205199758
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cooking plays a vital role in everyday independence and well-being, yet remains challenging for people with vision impairments due to limited support for tracking progress and receiving contextual feedback. Object status - the condition or transformation of ingredients and tools - offers a promising but underexplored foundation for context-aware cooking support. In this paper, we present OSCAR (Object Status Context Awareness for Recipes), a technical pipeline that explores the use of object status recognition to enable recipe progress tracking in non-visual cooking. OSCAR integrates recipe parsing, object status extraction, visual alignment with cooking steps, and time-causal modeling to support real-time step tracking. We evaluate OSCAR on 173 instructional videos and a real-world dataset of 12 non-visual cooking sessions recorded by BLV individuals in their homes. Our results show that object status consistently improves step prediction accuracy across vision-language models, and reveal key factors that impact performance in real-world conditions, such as implicit tasks, camera placement, and lighting. We contribute the pipeline of context-aware recipe progress tracking, an annotated real-world non-visual cooking dataset, and design insights to guide future context-aware assistive cooking systems.

Related papers

• VisualChef: Generating Visual Aids in Cooking via Mask Inpainting [50.84305074983752]
  We introduce VisualChef, a method for generating contextual visual aids tailored to cooking scenarios.<n>Given an initial frame and a specified action, VisualChef generates images depicting both the action's execution and the resulting appearance of the object.<n>We evaluate VisualChef quantitatively and qualitatively on three egocentric video datasets and show its improvements over state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-06-23T12:23:21Z)
• OSCAR: Object Status and Contextual Awareness for Recipes to Support Non-Visual Cooking [24.6085205199758]
  Following recipes while cooking is an important but difficult task for visually impaired individuals.<n>We developed OSCAR, a novel approach that provides recipe progress tracking and context-aware feedback.<n>We evaluated OSCAR's recipe following functionality using 173 YouTube cooking videos and 12 real-world non-visual cooking videos.
  arXiv  Detail & Related papers  (2025-03-07T22:03:21Z)
• Object-Centric Temporal Consistency via Conditional Autoregressive Inductive Biases [69.46487306858789]
  Conditional Autoregressive Slot Attention (CA-SA) is a framework that enhances the temporal consistency of extracted object-centric representations in video-centric vision tasks. We present qualitative and quantitative results showing that our proposed method outperforms the considered baselines on downstream tasks.
  arXiv  Detail & Related papers  (2024-10-21T07:44:44Z)
• ActionCOMET: A Zero-shot Approach to Learn Image-specific Commonsense Concepts about Actions [66.20773952864802]
  We develop a dataset consisting of 8.5k images and 59.3k inferences about actions grounded in those images. We propose ActionCOMET, a framework to discern knowledge present in language models specific to the provided visual input.
  arXiv  Detail & Related papers  (2024-10-17T15:22:57Z)
• Continuous Object State Recognition for Cooking Robots Using Pre-Trained Vision-Language Models and Black-box Optimization [18.41474014665171]
  We propose a method to recognize the continuous state changes of food for cooking robots through the spoken language. We show that by adjusting the weighting of each text prompt, more accurate and robust continuous state recognition can be achieved.
  arXiv  Detail & Related papers  (2024-03-13T04:45:40Z)
• Rethinking Cooking State Recognition with Vision Transformers [0.0]
  Self-attention mechanism of Vision Transformer (ViT) architecture is proposed for the Cooking State Recognition task. The proposed approach encapsulates the globally salient features from images, while also exploiting the weights learned from a larger dataset. Our framework has an accuracy of 94.3%, which significantly outperforms the state-of-the-art.
  arXiv  Detail & Related papers  (2022-12-16T17:06:28Z)
• Structured Vision-Language Pretraining for Computational Cooking [54.0571416522547]
  Vision-Language Pretraining and Foundation models have been the go-to recipe for achieving SoTA performance on general benchmarks. We propose to leverage these techniques for structured-text based computational cuisine tasks.
  arXiv  Detail & Related papers  (2022-12-08T13:37:17Z)
• Embodied Visual Active Learning for Semantic Segmentation [33.02424587900808]
  We study the task of embodied visual active learning, where an agent is set to explore a 3d environment with the goal to acquire visual scene understanding. We develop a battery of agents - both learnt and pre-specified - and with different levels of knowledge of the environment. We extensively evaluate the proposed models using the Matterport3D simulator and show that a fully learnt method outperforms comparable pre-specified counterparts.
  arXiv  Detail & Related papers  (2020-12-17T11:02:34Z)
• COBE: Contextualized Object Embeddings from Narrated Instructional Video [52.73710465010274]
  We propose a new framework for learning Contextualized OBject Embeddings from automatically-transcribed narrations of instructional videos. We leverage the semantic and compositional structure of language by training a visual detector to predict a contextualized word embedding of the object and its associated narration. Our experiments show that our detector learns to predict a rich variety of contextual object information, and that it is highly effective in the settings of few-shot and zero-shot learning.
  arXiv  Detail & Related papers  (2020-07-14T19:04:08Z)
• Spatio-Temporal Graph for Video Captioning with Knowledge Distillation [50.034189314258356]
  We propose a graph model for video captioning that exploits object interactions in space and time. Our model builds interpretable links and is able to provide explicit visual grounding. To avoid correlations caused by the variable number of objects, we propose an object-aware knowledge distillation mechanism.
  arXiv  Detail & Related papers  (2020-03-31T03:58:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.