A PRISMA-driven systematic mapping study on system assurance weakeners

• URL: http://arxiv.org/abs/2311.08328v1
• Date: Tue, 14 Nov 2023 17:17:16 GMT
• Title: A PRISMA-driven systematic mapping study on system assurance weakeners
• Abstract summary: We aim to initiate the first comprehensive systematic mapping study on assurance weakeners. We searched for primary studies in five digital libraries and focused on the 2012-2023 publication year range. Our selection criteria focused on studies addressing assurance weakeners at the modeling level.
• Score: 0.8493449152820131
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Context: An assurance case is a structured hierarchy of claims aiming at demonstrating that a given mission-critical system supports specific requirements (e.g., safety, security, privacy). The presence of assurance weakeners (i.e., assurance deficits, logical fallacies) in assurance cases reflects insufficient evidence, knowledge, or gaps in reasoning. These weakeners can undermine confidence in assurance arguments, potentially hindering the verification of mission-critical system capabilities. Objectives: As a stepping stone for future research on assurance weakeners, we aim to initiate the first comprehensive systematic mapping study on this subject. Methods: We followed the well-established PRISMA 2020 and SEGRESS guidelines to conduct our systematic mapping study. We searched for primary studies in five digital libraries and focused on the 2012-2023 publication year range. Our selection criteria focused on studies addressing assurance weakeners at the modeling level, resulting in the inclusion of 39 primary studies in our systematic review. Results: Our systematic mapping study reports a taxonomy (map) that provides a uniform categorization of assurance weakeners and approaches proposed to manage them at the modeling level. Conclusion: Our study findings suggest that the SACM (Structured Assurance Case Metamodel) -- a standard specified by the OMG (Object Management Group) -- may be the best specification to capture structured arguments and reason about their potential assurance weakeners.

Related papers

• Position: Anthropomorphic Misalignment Research Needs Stronger Evidence [52.76826423649968]
  We show how conceptual ambiguity, non-robust datasets, experimental design, and insufficient causal interventions can lead to overinterpretation of model behaviors.<n>This position paper aims to offer guidance on evidentiary considerations that can help improve methodological rigor in AMR.
  arXiv  Detail & Related papers  (2026-05-29T16:38:53Z)
• Perception Without Engagement: Dissecting the Causal Discovery Deficit in LMMs [68.58207076756237]
  We introduce ProCauEval, a perturbation-based evaluation protocol that shifts from outcome assessment to mechanism diagnosis.<n>We find that models faithfully perceive video content yet systematically underexploit it during causal reasoning.<n>We propose Anti-Distillation Policy Optimization (ADPO), a reinforcement learning framework built on negative teacher alignment.
  arXiv  Detail & Related papers  (2026-05-10T08:48:58Z)
• Security in LLM-as-a-Judge: A Comprehensive SoK [3.5168742057928246]
  We present the first Systematization of Knowledge (SoK) focusing on the security aspects of LLM-as-a-Judge systems.<n>Based on this study, we propose a taxonomy that organizes recent research according to the role played by LLM-as-a-Judge in the security landscape.<n>Our findings reveal significant vulnerabilities in LLM-based evaluation frameworks, as well as promising directions for improving their robustness and reliability.
  arXiv  Detail & Related papers  (2026-03-31T08:05:54Z)
• Resilient Microservices: A Systematic Review of Recovery Patterns, Strategies, and Evaluation Frameworks [0.0]
  This paper presents a PRISMA aligned systematic literature review of studies on microservice recovery strategies published between 2014 and 2025.<n>The review identifies nine recurring resilience themes encompassing circuit breakers, retries with jitter and budgets, sagas with compensation, idempotency, bulkheads, adaptive backpressure, observability, and chaos validation.
  arXiv  Detail & Related papers  (2025-12-18T04:13:54Z)
• Rethinking Cybersecurity Ontology Classification and Evaluation: Towards a Credibility-Centered Framework [0.19703625025720695]
  This paper analyzes the proliferation of cybersecurity, arguing that this surge cannot be explained solely by technical shortcomings but also by a credibility deficit.<n>We propose a revised framework for assessing credibility, introducing indicators such as institutional support, academic recognition, day-to-day practitioner validation, and industrial adoption.<n>We then apply this framework to a concrete use: the Franco-Luxembourgish research project ANCILE, which illustrates how a credibility-aware evaluation can reshape selection for operational contexts.
  arXiv  Detail & Related papers  (2025-12-01T13:25:05Z)
• Measuring what Matters: Construct Validity in Large Language Model Benchmarks [103.53142193393931]
  evaluating large language models (LLMs) is crucial for both assessing their capabilities and identifying safety or robustness issues prior to deployment.<n>We conduct a systematic review of 445 benchmarks from leading conferences in natural language processing and machine learning.<n>We find patterns related to the measured phenomena, tasks, and scoring metrics which undermine the validity of the resulting claims.
  arXiv  Detail & Related papers  (2025-11-03T17:39:40Z)
• LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarios [63.08653028889316]
  We propose a systematic taxonomy that decomposes agentic reasoning frameworks and analyze how these frameworks dominate framework-level reasoning.<n>Specifically, we propose an unified formal language to further classify agentic reasoning systems into single-agent methods, tool-based methods, and multi-agent methods.<n>We provide a comprehensive review of their key application scenarios in scientific discovery, healthcare, software engineering, social simulation, and economics.
  arXiv  Detail & Related papers  (2025-08-25T06:01:16Z)
• Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applications [59.721265428780946]
  Large Language Models (LLMs) in medicine have enabled impressive capabilities, yet a critical gap remains in their ability to perform systematic, transparent, and verifiable reasoning.<n>This paper provides the first systematic review of this emerging field.<n>We propose a taxonomy of reasoning enhancement techniques, categorized into training-time strategies and test-time mechanisms.
  arXiv  Detail & Related papers  (2025-08-01T14:41:31Z)
• Towards a rigorous evaluation of RAG systems: the challenge of due diligence [39.577682622066256]
  The rise of generative AI, has driven significant advancements in high-risk sectors like healthcare and finance.<n>The Retrieval-Augmented Generation (RAG) architecture, combining language models (LLMs) with search engines, is particularly notable for its ability to generate responses from document corpora.<n>This study evaluates a RAG system used in due diligence for an investment fund.
  arXiv  Detail & Related papers  (2025-07-29T12:33:16Z)
• Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers [61.57691030102618]
  We propose a novel jailbreaking method, Paper Summary Attack (llmnamePSA)<n>It synthesizes content from either attack-focused or defense-focused LLM safety paper to construct an adversarial prompt template.<n>Experiments show significant vulnerabilities not only in base LLMs, but also in state-of-the-art reasoning model like Deepseek-R1.
  arXiv  Detail & Related papers  (2025-07-17T18:33:50Z)
• SafeMLRM: Demystifying Safety in Multi-modal Large Reasoning Models [50.34706204154244]
  Acquiring reasoning capabilities catastrophically degrades inherited safety alignment. Certain scenarios suffer 25 times higher attack rates. Despite tight reasoning-answer safety coupling, MLRMs demonstrate nascent self-correction.
  arXiv  Detail & Related papers  (2025-04-09T06:53:23Z)
• Probabilistic Robustness in Deep Learning: A Concise yet Comprehensive Guide [2.152298082788376]
  Probable robustness (PR) offers a more practical perspective by quantifying the likelihood of failures under perturbations. This paper provides a concise yet comprehensive overview of PR, covering its formal definitions, evaluation and enhancement methods. We explore the integration of PR verification evidence into system-level safety assurance, addressing challenges in translating DL model-level robustness to system-level claims.
  arXiv  Detail & Related papers  (2025-02-20T18:47:17Z)
• PredictaBoard: Benchmarking LLM Score Predictability [50.47497036981544]
  Large Language Models (LLMs) often fail unpredictably. This poses a significant challenge to ensuring their safe deployment. We present PredictaBoard, a novel collaborative benchmarking framework.
  arXiv  Detail & Related papers  (2025-02-20T10:52:38Z)
• Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectives [52.863024096759816]
  Misaligned research objectives have hindered progress in adversarial robustness research over the past decade. We argue that realigned objectives are necessary for meaningful progress in adversarial alignment.
  arXiv  Detail & Related papers  (2025-02-17T15:28:40Z)
• A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations [127.52707312573791]
  This survey provides a comprehensive analysis of LVLM safety, covering key aspects such as attacks, defenses, and evaluation methods. We introduce a unified framework that integrates these interrelated components, offering a holistic perspective on the vulnerabilities of LVLMs. We conduct a set of safety evaluations on the latest LVLM, Deepseek Janus-Pro, and provide a theoretical analysis of the results.
  arXiv  Detail & Related papers  (2025-02-14T08:42:43Z)
• Causality can systematically address the monsters under the bench(marks) [64.36592889550431]
  Benchmarks are plagued by various biases, artifacts, or leakage. Models may behave unreliably due to poorly explored failure modes. causality offers an ideal framework to systematically address these challenges.
  arXiv  Detail & Related papers  (2025-02-07T17:01:37Z)
• A Taxonomy of Real-World Defeaters in Safety Assurance Cases [4.4398355848251745]
  The software engineering community could benefit from having a reusable classification of real-world defeaters in software assurance cases. We derived a taxonomy with seven broad categories, laying the groundwork for standardizing the analysis and management of defeaters in safety-critical systems.
  arXiv  Detail & Related papers  (2025-02-01T00:38:41Z)
• Controlling Risk of Retrieval-augmented Generation: A Counterfactual Prompting Framework [77.45983464131977]
  We focus on how likely it is that a RAG model's prediction is incorrect, resulting in uncontrollable risks in real-world applications. Our research identifies two critical latent factors affecting RAG's confidence in its predictions. We develop a counterfactual prompting framework that induces the models to alter these factors and analyzes the effect on their answers.
  arXiv  Detail & Related papers  (2024-09-24T14:52:14Z)
• Investigating Coverage Criteria in Large Language Models: An In-Depth Study Through Jailbreak Attacks [10.909463767558023]
  We propose an innovative approach for the real-time detection of jailbreak attacks by utilizing neural activation features. Our method holds promise for future systems integrating LLMs, offering robust real-time detection capabilities.
  arXiv  Detail & Related papers  (2024-08-27T17:14:21Z)
• Automating Semantic Analysis of System Assurance Cases using Goal-directed ASP [1.2189422792863451]
  We present our approach to enhancing Assurance 2.0 with semantic rule-based analysis capabilities. We examine the unique semantic aspects of assurance cases, such as logical consistency, adequacy, indefeasibility, etc.
  arXiv  Detail & Related papers  (2024-08-21T15:22:43Z)
• A PRISMA-Driven Bibliometric Analysis of the Scientific Literature on Assurance Case Patterns [7.930875992631788]
  Assurance cases can be used to prevent system failure. They are structured arguments that allow arguing and relaying various safety-critical systems' requirements.
  arXiv  Detail & Related papers  (2024-07-06T05:00:49Z)
• Evaluating Human Alignment and Model Faithfulness of LLM Rationale [66.75309523854476]
  We study how well large language models (LLMs) explain their generations through rationales. We show that prompting-based methods are less "faithful" than attribution-based explanations.
  arXiv  Detail & Related papers  (2024-06-28T20:06:30Z)
• Unveiling the Misuse Potential of Base Large Language Models via In-Context Learning [61.2224355547598]
  Open-sourcing of large language models (LLMs) accelerates application development, innovation, and scientific progress. Our investigation exposes a critical oversight in this belief. By deploying carefully designed demonstrations, our research demonstrates that base LLMs could effectively interpret and execute malicious instructions.
  arXiv  Detail & Related papers  (2024-04-16T13:22:54Z)
• FaultGuard: A Generative Approach to Resilient Fault Prediction in Smart Electrical Grids [53.2306792009435]
  FaultGuard is the first framework for fault type and zone classification resilient to adversarial attacks. We propose a low-complexity fault prediction model and an online adversarial training technique to enhance robustness. Our model outclasses the state-of-the-art for resilient fault prediction benchmarking, with an accuracy of up to 0.958.
  arXiv  Detail & Related papers  (2024-03-26T08:51:23Z)
• A Survey on Safe Multi-Modal Learning System [10.914595812695218]
  multimodal learning systems (MMLS) have gained traction for their ability to process and integrate information from diverse modality inputs. The absence of systematic research into their safety is a significant barrier to progress in this field. We present the first taxonomy that systematically categorizes and assesses MMLS safety.
  arXiv  Detail & Related papers  (2024-02-08T02:27:13Z)
• Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes [93.61202366677526]
  We study the offline reinforcement learning (RL) in the face of unmeasured confounders. We propose various policy learning methods with the finite-sample suboptimality guarantee of finding the optimal in-class policy.
  arXiv  Detail & Related papers  (2022-09-18T22:03:55Z)
• Estimating and Improving Fairness with Adversarial Learning [65.99330614802388]
  We propose an adversarial multi-task training strategy to simultaneously mitigate and detect bias in the deep learning-based medical image analysis system. Specifically, we propose to add a discrimination module against bias and a critical module that predicts unfairness within the base classification model. We evaluate our framework on a large-scale public-available skin lesion dataset.
  arXiv  Detail & Related papers  (2021-03-07T03:10:32Z)
• Adversarial Attacks against Face Recognition: A Comprehensive Study [3.766020696203255]
  Face recognition (FR) systems have demonstrated outstanding verification performance. Recent studies show that (deep) FR systems exhibit an intriguing vulnerability to imperceptible or perceptible but natural-looking adversarial input images.
  arXiv  Detail & Related papers  (2020-07-22T22:46:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.