Inclusive STEAM Education: A Framework for Teaching Cod-2 ing and Robotics to Students with Visually Impairment Using 3 Advanced Computer Vision

• URL: http://arxiv.org/abs/2503.16482v1
• Date: Thu, 06 Mar 2025 17:15:12 GMT
• Title: Inclusive STEAM Education: A Framework for Teaching Cod-2 ing and Robotics to Students with Visually Impairment Using 3 Advanced Computer Vision
• Authors: Mahmoud Hamash, Md Raqib Khan, Peter Tiernan,
• Abstract summary: This paper presents a framework that leverages pre-constructed robots and algorithms, such as maze-solving techniques, within an accessible learning environment.<n>The proposed system employs Contrastive Language-Image Pre-training (CLIP) to process global camera-captured maze layouts.<n>Students issue verbal commands, which are refined through CLIP, while robot-mounted stereo cameras provide real-time data processed via Simultaneous SLAM.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: STEAM education integrates Science, Technology, Engineering, Arts, and Mathematics to foster creativity and problem-solving. However, students with visual impairments (VI) encounter significant challenges in programming and robotics, particularly in tracking robot movements and developing spatial awareness. This paper presents a framework that leverages pre-constructed robots and algorithms, such as maze-solving techniques, within an accessible learning environment. The proposed system employs Contrastive Language-Image Pre-training (CLIP) to process global camera-captured maze layouts, converting visual data into textual descriptions that generate spatial audio prompts in an Audio Virtual Reality (AVR) system. Students issue verbal commands, which are refined through CLIP, while robot-mounted stereo cameras provide real-time data processed via Simultaneous Localization and Mapping (SLAM) for continuous feedback. By integrating these technologies, the framework empowers VI students to develop coding skills and engage in complex problem-solving tasks. Beyond maze-solving applications, this approach demonstrates the broader potential of computer vision in special education, contributing to improved accessibility and learning experiences in STEAM disciplines.

Related papers

• Overview of Current Challenges in Multi-Architecture Software Engineering and a Vision for the Future [0.0]
  The presented system architecture is based on the concept of dynamic, knowledge graph-based WebAssembly Twins. The resulting systems are to possess advanced autonomous capabilities, with full transparency and controllability by the end user.
  arXiv  Detail & Related papers  (2024-10-28T13:03:09Z)
• An Innovative Solution: AI-Based Digital Screen-Integrated Tables for Educational Settings [0.0]
  Digital screen-integrated tables are designed specifically for educational settings. Tables feature integrated digital screens controlled by a central processing unit (CPU) The invention facilitates the collection of student performance data during classroom activities and assessments.
  arXiv  Detail & Related papers  (2024-10-08T08:00:17Z)
• VITAL: Visual Teleoperation to Enhance Robot Learning through Human-in-the-Loop Corrections [10.49712834719005]
  We propose a low-cost visual teleoperation system for bimanual manipulation tasks, called VITAL. Our approach leverages affordable hardware and visual processing techniques to collect demonstrations. We enhance the generalizability and robustness of the learned policies by utilizing both real and simulated environments.
  arXiv  Detail & Related papers  (2024-07-30T23:29:47Z)
• Research Experience of an Undergraduate Student in Computer Vision and Robotics [0.0]
  This paper focuses on the educational journey of a computer engineering undergraduate student venturing into the domain of computer vision and robotics. It explores how optical flow and its applications can be used to detect moving objects when a camera undergoes translational motion, highlighting the challenges encountered and the strategies used to overcome them.
  arXiv  Detail & Related papers  (2024-07-14T02:01:50Z)
• MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting [97.52388851329667]
  We introduce Marking Open-world Keypoint Affordances (MOKA) to solve robotic manipulation tasks specified by free-form language instructions. Central to our approach is a compact point-based representation of affordance, which bridges the VLM's predictions on observed images and the robot's actions in the physical world. We evaluate and analyze MOKA's performance on various table-top manipulation tasks including tool use, deformable body manipulation, and object rearrangement.
  arXiv  Detail & Related papers  (2024-03-05T18:08:45Z)
• Human-oriented Representation Learning for Robotic Manipulation [64.59499047836637]
  Humans inherently possess generalizable visual representations that empower them to efficiently explore and interact with the environments in manipulation tasks. We formalize this idea through the lens of human-oriented multi-task fine-tuning on top of pre-trained visual encoders. Our Task Fusion Decoder consistently improves the representation of three state-of-the-art visual encoders for downstream manipulation policy-learning.
  arXiv  Detail & Related papers  (2023-10-04T17:59:38Z)
• Evaluating Continual Learning Algorithms by Generating 3D Virtual Environments [66.83839051693695]
  Continual learning refers to the ability of humans and animals to incrementally learn over time in a given environment. We propose to leverage recent advances in 3D virtual environments in order to approach the automatic generation of potentially life-long dynamic scenes with photo-realistic appearance. A novel element of this paper is that scenes are described in a parametric way, thus allowing the user to fully control the visual complexity of the input stream the agent perceives.
  arXiv  Detail & Related papers  (2021-09-16T10:37:21Z)
• Low Dimensional State Representation Learning with Robotics Priors in Continuous Action Spaces [8.692025477306212]
  Reinforcement learning algorithms have proven to be capable of solving complicated robotics tasks in an end-to-end fashion. We propose a framework combining the learning of a low-dimensional state representation, from high-dimensional observations coming from the robot's raw sensory readings, with the learning of the optimal policy.
  arXiv  Detail & Related papers  (2021-07-04T15:42:01Z)
• Technology Readiness Levels for Machine Learning Systems [107.56979560568232]
  Development and deployment of machine learning systems can be executed easily with modern tools, but the process is typically rushed and means-to-an-end. We have developed a proven systems engineering approach for machine learning development and deployment. Our "Machine Learning Technology Readiness Levels" framework defines a principled process to ensure robust, reliable, and responsible systems.
  arXiv  Detail & Related papers  (2021-01-11T15:54:48Z)
• The Ingredients of Real-World Robotic Reinforcement Learning [71.92831985295163]
  We discuss the elements that are needed for a robotic learning system that can continually and autonomously improve with data collected in the real world. We propose a particular instantiation of such a system, using dexterous manipulation as our case study. We demonstrate that our complete system can learn without any human intervention, acquiring a variety of vision-based skills with a real-world three-fingered hand.
  arXiv  Detail & Related papers  (2020-04-27T03:36:10Z)
• SAPIEN: A SimulAted Part-based Interactive ENvironment [77.4739790629284]
  SAPIEN is a realistic and physics-rich simulated environment that hosts a large-scale set for articulated objects. We evaluate state-of-the-art vision algorithms for part detection and motion attribute recognition as well as demonstrate robotic interaction tasks.
  arXiv  Detail & Related papers  (2020-03-19T00:11:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.