Requirements for Aligned, Dynamic Resolution of Conflicts in Operational Constraints

• URL: http://arxiv.org/abs/2511.10952v2
• Date: Tue, 18 Nov 2025 03:36:47 GMT
• Title: Requirements for Aligned, Dynamic Resolution of Conflicts in Operational Constraints
• Authors: Steven J. Jones, Robert E. Wray, John E. Laird,
• Abstract summary: Deployed, autonomous AI systems must often evaluate multiple plausible courses of action in novel or under-specified contexts.<n>This paper characterizes requirements for agent decision making in these contexts.<n>It also identifies the types of knowledge agents require to make decisions robust to agent goals and aligned with human expectations.
• Score: 2.752817022620644
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Deployed, autonomous AI systems must often evaluate multiple plausible courses of action (extended sequences of behavior) in novel or under-specified contexts. Despite extensive training, these systems will inevitably encounter scenarios where no available course of action fully satisfies all operational constraints (e.g., operating procedures, rules, laws, norms, and goals). To achieve goals in accordance with human expectations and values, agents must go beyond their trained policies and instead construct, evaluate, and justify candidate courses of action. These processes require contextual "knowledge" that may lie outside prior (policy) training. This paper characterizes requirements for agent decision making in these contexts. It also identifies the types of knowledge agents require to make decisions robust to agent goals and aligned with human expectations. Drawing on both analysis and empirical case studies, we examine how agents need to integrate normative, pragmatic, and situational understanding to select and then to pursue more aligned courses of action in complex, real-world environments.

Related papers

• Operationalizing Human Values in the Requirements Engineering Process of Ethics-Aware Autonomous Systems [3.535946769391712]
  We propose a requirements engineering approach for ethics-aware autonomous systems.<n>We capture human values as normative goals and align them with functional and adaptation goals.<n>We demonstrate the feasibility of the approach through a medical Body Sensor Network case study.
  arXiv  Detail & Related papers  (2026-02-10T15:54:25Z)
• Agentic Reasoning for Large Language Models [122.81018455095999]
  Reasoning is a fundamental cognitive process underlying inference, problem-solving, and decision-making.<n>Large language models (LLMs) demonstrate strong reasoning capabilities in closed-world settings, but struggle in open-ended and dynamic environments.<n>Agentic reasoning marks a paradigm shift by reframing LLMs as autonomous agents that plan, act, and learn through continual interaction.
  arXiv  Detail & Related papers  (2026-01-18T18:58:23Z)
• Toward a Theory of Agents as Tool-Use Decision-Makers [89.26889709510242]
  We argue that true autonomy requires agents to be grounded in a coherent epistemic framework that governs what they know, what they need to know, and how to acquire that knowledge efficiently.<n>We propose a unified theory that treats internal reasoning and external actions as equivalent epistemic tools, enabling agents to systematically coordinate introspection and interaction.<n>This perspective shifts the design of agents from mere action executors to knowledge-driven intelligence systems, offering a principled path toward building foundation agents capable of adaptive, efficient, and goal-directed behavior.
  arXiv  Detail & Related papers  (2025-06-01T07:52:16Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• Towards Developing Ethical Reasoners: Integrating Probabilistic Reasoning and Decision-Making for Complex AI Systems [4.854297874710511]
  A computational ethics framework is essential for AI and autonomous systems operating in complex, real-world environments.<n>Existing approaches often lack the adaptability needed to integrate ethical principles into dynamic and ambiguous contexts.<n>We outline the necessary ingredients for building a holistic, meta-level framework that combines intermediate representations, probabilistic reasoning, and knowledge representation.
  arXiv  Detail & Related papers  (2025-02-28T17:25:11Z)
• Normative Requirements Operationalization with Large Language Models [3.456725053685842]
  Normative non-functional requirements specify constraints that a system must observe in order to avoid violations of social, legal, ethical, empathetic, and cultural norms. Recent research has tackled this challenge using a domain-specific language to specify normative requirements. We propose a complementary approach that uses Large Language Models to extract semantic relationships between abstract representations of system capabilities.
  arXiv  Detail & Related papers  (2024-04-18T17:01:34Z)
• Resolving Ethics Trade-offs in Implementing Responsible AI [18.894725256708128]
  We cover five approaches for addressing the tensions via trade-offs, ranging from rudimentary to complex.<n>None of the approaches is likely to be appropriate for all organisations, systems, or applications.<n>We propose a framework which consists of: (i) proactive identification of tensions, (ii) prioritisation and weighting of ethics aspects, (iii) justification and documentation of trade-off decisions.
  arXiv  Detail & Related papers  (2024-01-16T04:14:23Z)
• Towards Responsible AI in Banking: Addressing Bias for Fair Decision-Making [69.44075077934914]
  "Responsible AI" emphasizes the critical nature of addressing biases within the development of a corporate culture. This thesis is structured around three fundamental pillars: understanding bias, mitigating bias, and accounting for bias. In line with open-source principles, we have released Bias On Demand and FairView as accessible Python packages.
  arXiv  Detail & Related papers  (2024-01-13T14:07:09Z)
• Code Models are Zero-shot Precondition Reasoners [83.8561159080672]
  We use code representations to reason about action preconditions for sequential decision making tasks. We propose a precondition-aware action sampling strategy that ensures actions predicted by a policy are consistent with preconditions.
  arXiv  Detail & Related papers  (2023-11-16T06:19:27Z)
• Context-Aware Composition of Agent Policies by Markov Decision Process Entity Embeddings and Agent Ensembles [1.124711723767572]
  Computational agents support humans in many areas of life and are therefore found in heterogeneous contexts. In order to perform services and carry out activities in a goal-oriented manner, agents require prior knowledge. We propose a novel simulation-based approach that enables the representation of heterogeneous contexts.
  arXiv  Detail & Related papers  (2023-08-28T12:13:36Z)
• Towards a multi-stakeholder value-based assessment framework for algorithmic systems [76.79703106646967]
  We develop a value-based assessment framework that visualizes closeness and tensions between values. We give guidelines on how to operationalize them, while opening up the evaluation and deliberation process to a wide range of stakeholders.
  arXiv  Detail & Related papers  (2022-05-09T19:28:32Z)
• Learning Goal-oriented Dialogue Policy with Opposite Agent Awareness [116.804536884437]
  We propose an opposite behavior aware framework for policy learning in goal-oriented dialogues. We estimate the opposite agent's policy from its behavior and use this estimation to improve the target agent by regarding it as part of the target policy.
  arXiv  Detail & Related papers  (2020-04-21T03:13:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.