FramingQA: Does the Question Shape the Answer? Measuring the Compositional Framing Effect

• URL: http://arxiv.org/abs/2609.07448v1
• Date: Mon, 07 Sep 2026 12:56:44 GMT
• Title: FramingQA: Does the Question Shape the Answer? Measuring the Compositional Framing Effect
• Abstract summary: Large language models often change their responses to subtle rephrasings that align with an implied stance by users.<n>This can leave users with advice tainted by how they happened to phrase a question rather than by the underlying facts.<n>We introduce FramingQA, a benchmark that measures the model sensitivity to question framing across law, medicine, finance, and robotic simulations.
• Score: 69.8983512616053
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce FramingQA, a benchmark that measures the model sensitivity to question framing across law, medicine, finance, and robotic simulations. Large language models (LLMs) often change their responses to subtle rephrasings that align with an implied stance by users. This can leave users with advice tainted by how they happened to phrase a question rather than by the underlying facts, and the consequences are highly costly in high-stakes domains. Because in the realistic scenarios, both expert practitioners and non-expert users frequently ask LLMs questions containing incomplete or misleading assumptions, models are highly susceptible to those framings. To test this, we inject the framing bias across three nested levels: a framing-biased question phrasing (root), an injected framing-biased premise prepended to a neutral question (propositional), and a premise paired with a framing-biased question (global). Evaluating nine open models (3.8B-70B) across four families, we find that strong per-variant accuracy does not guarantee the robustness across differently phrased questions under the fixed factual information.

Related papers

• Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthromorphism, and Maxims [54.31707479192195]
  Large language models (LLMs) are being increasingly used to answer subjective, information-seeking questions.<n>Existing evaluations for subjective cultural queries largely focus on factual correctness, ignoring how the response is framed.<n>We introduce FRANZ, an automated FRAmework for respoNse characteriZation to conduct communicative audit of LLM responses along four dimensions.
  arXiv  Detail & Related papers  (2026-06-01T17:01:31Z)
• Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy [41.17236645999555]
  We study how model answers change under meaning-preserving paraphrases across factual question answering and mathematical reasoning tasks.<n>Across four benchmarks and 13 models, we find that model outputs frequently depend on the exact wording of the prompt.<n>For many questions, models alternate between correct and incorrect answers depending on phrasing, with mismatch rates reaching more than 23%.
  arXiv  Detail & Related papers  (2026-05-18T16:45:13Z)
• Increasing LLM response trustworthiness using voting ensembles [0.20524609401792396]
  We introduce a theoretical framework for question answering and show that, by permitting ensembles to "abstain" from providing an answer when the dominant response falls short, it is possible to dramatically increase the trustworthiness of the remaining answers.<n>We observe that large gains in answer trustworthiness can be achieved using highly restrictive voting ensembles, while incurring relatively modest reductions in response yield and accuracy.
  arXiv  Detail & Related papers  (2025-10-05T06:02:44Z)
• MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs [15.278241998033822]
  Open-ended question answering (QA) is a key task for evaluating the capabilities of large language models (LLMs)<n>We propose textbfMinosEval, a novel evaluation method that first distinguishes open-ended questions and then ranks candidate answers.
  arXiv  Detail & Related papers  (2025-06-18T07:49:13Z)
• Don't Just Say "I don't know"! Self-aligning Large Language Models for Responding to Unknown Questions with Explanations [70.6395572287422]
  Self-alignment method is capable of not only refusing to answer but also providing explanation to the unanswerability of unknown questions. We conduct disparity-driven self-curation to select qualified data for fine-tuning the LLM itself for aligning the responses to unknown questions as desired.
  arXiv  Detail & Related papers  (2024-02-23T02:24:36Z)
• Selectively Answering Ambiguous Questions [38.83930394700588]
  We find that the most reliable approach to decide when to abstain involves quantifying repetition within sampled model outputs. Our results suggest that sampling-based confidence scores help calibrate answers to relatively unambiguous questions.
  arXiv  Detail & Related papers  (2023-05-24T01:25:38Z)
• Realistic Conversational Question Answering with Answer Selection based on Calibrated Confidence and Uncertainty Measurement [54.55643652781891]
  Conversational Question Answering (ConvQA) models aim at answering a question with its relevant paragraph and previous question-answer pairs that occurred during conversation multiple times. We propose to filter out inaccurate answers in the conversation history based on their estimated confidences and uncertainties from the ConvQA model. We validate our models, Answer Selection-based realistic Conversation Question Answering, on two standard ConvQA datasets.
  arXiv  Detail & Related papers  (2023-02-10T09:42:07Z)
• CREPE: Open-Domain Question Answering with False Presuppositions [92.20501870319765]
  We introduce CREPE, a QA dataset containing a natural distribution of presupposition failures from online information-seeking forums. We find that 25% of questions contain false presuppositions, and provide annotations for these presuppositions and their corrections. We show that adaptations of existing open-domain QA models can find presuppositions moderately well, but struggle when predicting whether a presupposition is factually correct.
  arXiv  Detail & Related papers  (2022-11-30T18:54:49Z)
• TASA: Deceiving Question Answering Models by Twin Answer Sentences Attack [93.50174324435321]
  We present Twin Answer Sentences Attack (TASA), an adversarial attack method for question answering (QA) models. TASA produces fluent and grammatical adversarial contexts while maintaining gold answers.
  arXiv  Detail & Related papers  (2022-10-27T07:16:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.