The importance of visual modelling languages in generative software engineering

• URL: http://arxiv.org/abs/2411.17976v3
• Date: Mon, 13 Jan 2025 17:42:09 GMT
• Title: The importance of visual modelling languages in generative software engineering
• Authors: Roberto Rossi,
• Abstract summary: GPT-4 accepts image and text inputs, rather than simply natural language. To the best of our knowledge, no other work has investigated similar use cases involving Software Engineering tasks carried out via multimodal GPTs.
• Score: 0.0
• License:
• Abstract: Multimodal GPTs represent a watershed in the interplay between Software Engineering and Generative Artificial Intelligence. GPT-4 accepts image and text inputs, rather than simply natural language. We investigate relevant use cases stemming from these enhanced capabilities of GPT-4. To the best of our knowledge, no other work has investigated similar use cases involving Software Engineering tasks carried out via multimodal GPTs prompted with a mix of diagrams and natural language.

Related papers

• Can Uniform Meaning Representation Help GPT-4 Translate from Indigenous Languages? [4.02487511510606]
  We explore the technical utility of Uniform Meaning Representation (UMR) for low-resource languages by incorporating it into GPT-4 prompts. We find that in the majority of our test cases, integrating UMR into the prompt results in a statistically significant increase in performance.
  arXiv  Detail & Related papers  (2025-02-13T02:27:30Z)
• What Is Missing in Multilingual Visual Reasoning and How to Fix It [57.37123046817781]
  We evaluate NLP models' multilingual, multimodal capabilities by testing on a visual reasoning task. proprietary systems like GPT-4V obtain the best performance on this task now, but open models lag in comparison. Our interventions achieve the best open performance on this task in a zero-shot setting, boosting open models LLaVA-v1.5-13B by 13.4%, LLaVA-v1.6-34B by 20.3%, and Qwen-VL by 16.7%.
  arXiv  Detail & Related papers  (2024-03-03T05:45:27Z)
• Comparing large language models and human programmers for generating programming code [0.0]
  GPT-4 substantially outperforms other large language models, including Gemini Ultra and Claude 2. In most LeetCode and GeeksforGeeks coding contests evaluated in this study, GPT-4 employing the optimal prompt strategy outperforms 85 percent of human participants.
  arXiv  Detail & Related papers  (2024-03-01T14:43:06Z)
• GPT-4 as an interface between researchers and computational software: improving usability and reproducibility [44.99833362998488]
  We focus on a widely used software for molecular dynamics simulations. We quantify the usefulness of input files generated by GPT-4 from task descriptions in English. We find that GPT-4 can generate correct and ready-to-use input files for relatively simple tasks. In addition, GPT-4's description of computational tasks from input files can be tuned from a detailed set of step-by-step instructions to a summary description appropriate for publications.
  arXiv  Detail & Related papers  (2023-10-04T14:25:39Z)
• The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision) [121.42924593374127]
  We analyze the latest model, GPT-4V, to deepen the understanding of LMMs. GPT-4V's unprecedented ability in processing arbitrarily interleaved multimodal inputs makes it a powerful multimodal generalist system. GPT-4V's unique capability of understanding visual markers drawn on input images can give rise to new human-computer interaction methods.
  arXiv  Detail & Related papers  (2023-09-29T17:34:51Z)
• Gpt-4: A Review on Advancements and Opportunities in Natural Language Processing [0.0]
  Generative Pre-trained Transformer 4 (GPT-4) is the fourth-generation language model in the GPT series, developed by OpenAI. GPT-4 has a larger model size (more than one trillion), better multilingual capabilities, improved contextual understanding, and reasoning capabilities than GPT-3. Some of the potential applications of GPT-4 include chatbots, personal assistants, language translation, text summarization, and question-answering.
  arXiv  Detail & Related papers  (2023-05-04T22:46:43Z)
• Visual Instruction Tuning [79.70923292053097]
  We present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%.
  arXiv  Detail & Related papers  (2023-04-17T17:59:25Z)
• ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning [70.57126720079971]
  Large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP) This paper evaluates ChatGPT on 7 different tasks, covering 37 diverse languages with high, medium, low, and extremely low resources. Compared to the performance of previous models, our extensive experimental results demonstrate a worse performance of ChatGPT for different NLP tasks and languages.
  arXiv  Detail & Related papers  (2023-04-12T05:08:52Z)
• Leveraging Language to Learn Program Abstractions and Search Heuristics [66.28391181268645]
  We introduce LAPS (Language for Abstraction and Program Search), a technique for using natural language annotations to guide joint learning of libraries and neurally-guided search models for synthesis. When integrated into a state-of-the-art library learning system (DreamCoder), LAPS produces higher-quality libraries and improves search efficiency and generalization.
  arXiv  Detail & Related papers  (2021-06-18T15:08:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.