Experimenting with Large Language Models and vector embeddings in NASA SciX

• URL: http://arxiv.org/abs/2312.14211v1
• Date: Thu, 21 Dec 2023 10:19:58 GMT
• Title: Experimenting with Large Language Models and vector embeddings in NASA SciX
• Abstract summary: We have developed an experiment where we created semantic vectors for our large collection of abstracts and full-text content. We designed a prompt system to ask questions using contextual chunks from our system. Based on a non-systematic human evaluation, the experiment shows a lower degree of hallucination and better responses when using Retrieval Augmented Generation.
• Score: 26.396350836111935
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-source Large Language Models enable projects such as NASA SciX (i.e., NASA ADS) to think out of the box and try alternative approaches for information retrieval and data augmentation, while respecting data copyright and users' privacy. However, when large language models are directly prompted with questions without any context, they are prone to hallucination. At NASA SciX we have developed an experiment where we created semantic vectors for our large collection of abstracts and full-text content, and we designed a prompt system to ask questions using contextual chunks from our system. Based on a non-systematic human evaluation, the experiment shows a lower degree of hallucination and better responses when using Retrieval Augmented Generation. Further exploration is required to design new features and data augmentation processes at NASA SciX that leverages this technology while respecting the high level of trust and quality that the project holds.

Related papers

• SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models [63.357710580101894]
  Large language models (LLMs) are increasingly applied to scientific research, yet existing evaluations often fail to reflect the fine-grained capabilities required in practice.<n>We propose a new framework named SciCustom to address the problem.<n>It enables the custom construction of benchmarks from large-scale scientific data to evaluate application-specific scientific capabilities in LLMs.
  arXiv  Detail & Related papers  (2026-05-19T04:41:43Z)
• STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence [81.94084852268468]
  We formalize audio 4D intelligence that is defined as reasoning over sound dynamics in time and 3D space.<n> STAR-Bench combines a Foundational Acoustic Perception setting with a Holistic Spatio-Temporal Reasoning setting.<n>Our data curation pipeline uses two methods to ensure high-quality samples.
  arXiv  Detail & Related papers  (2025-10-28T17:50:34Z)
• SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes [105.8644620467576]
  We introduce Stextscurprise3D, a novel dataset designed to evaluate language-guided spatial reasoning segmentation in complex 3D scenes.<n>Stextscurprise3D consists of more than 200k vision language pairs across 900+ detailed indoor scenes from ScanNet++ v2.<n>The dataset contains 89k+ human-annotated spatial queries deliberately crafted without object name.
  arXiv  Detail & Related papers  (2025-07-10T14:01:24Z)
• Spatial Knowledge Graph-Guided Multimodal Synthesis [78.11669780958657]
  We introduce a novel multimodal synthesis approach guided by spatial knowledge graphs, grounded in the concept of knowledge-to-data generation.<n>In experiments, data synthesized from diverse types of spatial knowledge, including direction and distance, enhance the spatial perception and reasoning abilities of MLLMs markedly.<n>We hope that the idea of knowledge-based data synthesis can advance the development of spatial intelligence.
  arXiv  Detail & Related papers  (2025-05-28T17:50:21Z)
• OpenFly: A Versatile Toolchain and Large-scale Benchmark for Aerial Vision-Language Navigation [49.697035403548966]
  Vision-Language Navigation (VLN) aims to guide agents through an environment by leveraging both language instructions and visual cues. We propose OpenFly, a platform comprising a versatile toolchain and large-scale benchmark for aerial VLN. We construct a large-scale aerial VLN dataset with 100k trajectories, covering diverse heights and lengths across 18 scenes. The corresponding visual data are generated using various rendering engines and advanced techniques, including Unreal, GTA V, Google Earth, and 3D Splatting (3D GS)
  arXiv  Detail & Related papers  (2025-02-25T09:57:18Z)
• Towards a Reliable Offline Personal AI Assistant for Long Duration Spaceflight [4.382282101149638]
  This paper proposes enhancing systems like METIS by integrating GPTs, Retrieval-Augmented Generation (RAG), Knowledge Graphs (KGs), and Augmented Reality (AR) The idea is to allow astronauts to interact with their data more intuitively, using natural language queries and visualizing real-time information through AR.
  arXiv  Detail & Related papers  (2024-10-21T18:08:42Z)
• Does Spatial Cognition Emerge in Frontier Models? [56.47912101304053]
  We present SPACE, a benchmark that systematically evaluates spatial cognition in frontier models. Results suggest that contemporary frontier models fall short of the spatial intelligence of animals.
  arXiv  Detail & Related papers  (2024-10-09T01:41:49Z)
• Adapting a Foundation Model for Space-based Tasks [16.81793096235458]
  In the future of space robotics, we see three core challenges which motivate the use of a foundation model adapted to space-based applications. In this work, we demonstrate that 1) existing vision-language models are deficient visual reasoners in space-based applications, and 2) fine-tuning a vision-language model on extraterrestrial data significantly improves the quality of responses.
  arXiv  Detail & Related papers  (2024-08-12T05:07:24Z)
• UltraFeedback: Boosting Language Models with Scaled AI Feedback [99.4633351133207]
  We present textscUltraFeedback, a large-scale, high-quality, and diversified AI feedback dataset. Our work validates the effectiveness of scaled AI feedback data in constructing strong open-source chat language models.
  arXiv  Detail & Related papers  (2023-10-02T17:40:01Z)
• AutoHall: Automated Hallucination Dataset Generation for Large Language Models [56.92068213969036]
  This paper introduces a method for automatically constructing model-specific hallucination datasets based on existing fact-checking datasets called AutoHall. We also propose a zero-resource and black-box hallucination detection method based on self-contradiction.
  arXiv  Detail & Related papers  (2023-09-30T05:20:02Z)
• NASA Science Mission Directorate Knowledge Graph Discovery [0.0]
  This paper proposes a pipeline to generate knowledge graphs (KGs) representing different NASA domains. These KGs can be used as the basis for dataset search engines, saving researchers time and supporting them in finding new connections.
  arXiv  Detail & Related papers  (2023-03-20T05:12:43Z)
• Artificial Intelligence and Natural Language Processing and Understanding in Space: Four ESA Case Studies [48.53582660901672]
  We present a methodological framework based on artificial intelligence and natural language processing and understanding to automatically extract information from Space documents. Case studies are implemented across different functional areas of ESA, including Mission Design, Quality Assurance, Long-Term Data Preservation, and the Open Space Innovation Platform.
  arXiv  Detail & Related papers  (2022-10-07T15:50:17Z)
• SpaceQA: Answering Questions about the Design of Space Missions and Space Craft Concepts [57.012600276711005]
  We present SpaceQA, to the best of our knowledge the first open-domain QA system in Space mission design. SpaceQA is part of an initiative by the European Space Agency (ESA) to facilitate the access, sharing and reuse of information about Space mission design.
  arXiv  Detail & Related papers  (2022-10-07T09:41:39Z)
• MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text [58.655375327681774]
  We propose the first Multimodal Retrieval-Augmented Transformer (MuRAG) MuRAG accesses an external non-parametric multimodal memory to augment language generation. Our results show that MuRAG achieves state-of-the-art accuracy, outperforming existing models by 10-20% absolute on both datasets.
  arXiv  Detail & Related papers  (2022-10-06T13:58:03Z)
• GenNI: Human-AI Collaboration for Data-Backed Text Generation [102.08127062293111]
  Table2Text systems generate textual output based on structured data utilizing machine learning. GenNI (Generation Negotiation Interface) is an interactive visual system for high-level human-AI collaboration in producing descriptive text.
  arXiv  Detail & Related papers  (2021-10-19T18:07:07Z)
• A Spacecraft Dataset for Detection, Segmentation and Parts Recognition [42.27081423489484]
  In this paper, we release a dataset for spacecraft detection, instance segmentation and part recognition. The main contribution of this work is the development of the dataset using images of space stations and satellites. We also provide evaluations with state-of-the-art methods in object detection and instance segmentation as a benchmark for the dataset.
  arXiv  Detail & Related papers  (2021-06-15T14:36:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.