Measuring Representation Robustness in Large Language Models for Geometry

• URL: http://arxiv.org/abs/2604.16421v1
• Date: Fri, 03 Apr 2026 11:36:49 GMT
• Title: Measuring Representation Robustness in Large Language Models for Geometry
• Abstract summary: In geometry, identical problems can be expressed in Euclidean, coordinate, or vector forms.<n>Existing benchmarks report accuracy on fixed formats.<n>We propose GeoRepEval, a representation-aware evaluation framework.
• Score: 7.743292557234699
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly evaluated on mathematical reasoning, yet their robustness to equivalent problem representations remains poorly understood. In geometry, identical problems can be expressed in Euclidean, coordinate, or vector forms, but existing benchmarks report accuracy on fixed formats, implicitly assuming representation invariance and masking failures caused by representational changes alone. We propose GeoRepEval, a representation-aware evaluation framework that measures correctness, invariance, and consistency at the problem level across parallel formulations, combining strict answer matching, bootstrap confidence intervals, paired McNemar tests, representation-flip analyses, and regression controls for surface complexity. We prove that our Invariance@3 metric decomposes accuracy into robust and fragile components and is bounded by the weakest representation. Evaluating eleven LLMs on 158 curated high-school geometry problems (474 instances), we find accuracy gaps of up to 14 percentage points induced solely by representation choice. Vector formulations emerge as a consistent failure point, with Invariance@3 as low as 0.044 even after controlling for length and symbolic complexity. A convert-then-solve prompting intervention improves vector accuracy by up to 52 percentage points for high-capacity models, suggesting that failures reflect representation sensitivity rather than inability; however, low-capacity models show no gains, indicating deeper limitations. These results suggest that current models rely on representation-specific heuristics rather than abstract geometric reasoning. All datasets, prompts, and scripts are released at https://github.com/vedjaw/GeoRepEval.

Related papers

• The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context [76.36635039699112]
  Large language models (LLMs) are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context.<n>We find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level.<n>This aggregate stability, however, masks significant per-example instability.
  arXiv  Detail & Related papers  (2026-07-14T17:01:12Z)
• Representation Robustness Under Executable Reasoning Constraints in Large Language Models for Mathematical Problem Solving [3.0618862102164996]
  This paper investigates representation robustness in large language models (LLMs)<n>We find substantial representational sensitivity, with nontrivial flip rates across story, symbolic, and word-equation variants.<n>We argue that representation should be treated as a first-class interface design variable in LLM evaluation and deployment.
  arXiv  Detail & Related papers  (2026-07-08T17:21:28Z)
• Articulating then Matching: Zero-Shot Shape Matching for Uncurated Data [0.5014513738509991]
  Finding dense correspondences between 3D shapes is a fundamental yet unresolved challenge, especially in real-world environments.<n>We present ATM, a framework that requires no correspondence-specific training and robustly addresses these issues at once.
  arXiv  Detail & Related papers  (2026-06-28T03:05:46Z)
• Linear Probes Detect Task Format, Not Reasoning Mode in Language Model Hidden States [16.505918019260964]
  Linear probing of large language model (LLM) hidden states is widely used to claim that models learn distinct representations for different reasoning types.<n>We test this by QwensicB on three benchmarks spanning the classical trichotomy: LogiQA 2.0 (deductive), ARC-Challenge (inductive), and $$NLI (abductive)
  arXiv  Detail & Related papers  (2026-06-01T21:22:15Z)
• Ensemble-Based Uncertainty Estimation for Code Correctness Estimation [36.53771380869671]
  We propose Ensemble Semantic Entropy (ESE), which estimates uncertainty by evaluating the consistency of samples aggregated across an ensemble of models.<n>Experiments on LiveCodeBench demonstrate that ESE correlates more strongly with program correctness than single-model semantic entropy.<n>We also propose a cascading test-time scaling framework Cas, which maintains performance while reducing FLOPs by 64.9% compared to single-model scaling.
  arXiv  Detail & Related papers  (2026-03-28T02:37:36Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• From Abstract to Contextual: What LLMs Still Cannot Do in Mathematics [79.81905350372067]
  We study gap through contextual mathematical reasoning.<n>We introduce ContextMATH, a benchmark that repurposes AIME and MATH-500 problems into two contextual settings.<n>Open-source models decline by 13 and 34 points on SG and CS, while proprietary models drop by 13 and 20.
  arXiv  Detail & Related papers  (2026-01-30T14:56:04Z)
• Catch Me If You Can: How Smaller Reasoning Models Pretend to Reason with Mathematical Fidelity [15.774418410083515]
  We introduce a diagnostic framework that distinguishes genuine mathematical reasoning from superficial pattern matching.<n>We reveal a striking disconnect between surface performance and reasoning fidelity.<n>Our diagnostics expose reasoning failures invisible to traditional accuracy metrics.
  arXiv  Detail & Related papers  (2025-11-29T16:47:01Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors [105.12462629663757]
  In this work, we aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. We compare performance of state-of-the-art factuality metrics, including recent ChatGPT-based metrics, on this stratified benchmark and show that their performance varies significantly across different types of summarization models.
  arXiv  Detail & Related papers  (2022-05-25T15:26:48Z)
• Generalization of Neural Combinatorial Solvers Through the Lens of Adversarial Robustness [68.97830259849086]
  Most datasets only capture a simpler subproblem and likely suffer from spurious features. We study adversarial robustness - a local generalization property - to reveal hard, model-specific instances and spurious features. Unlike in other applications, where perturbation models are designed around subjective notions of imperceptibility, our perturbation models are efficient and sound. Surprisingly, with such perturbations, a sufficiently expressive neural solver does not suffer from the limitations of the accuracy-robustness trade-off common in supervised learning.
  arXiv  Detail & Related papers  (2021-10-21T07:28:11Z)
• Finding Geometric Models by Clustering in the Consensus Space [61.65661010039768]
  We propose a new algorithm for finding an unknown number of geometric models, e.g., homographies. We present a number of applications where the use of multiple geometric models improves accuracy. These include pose estimation from multiple generalized homographies; trajectory estimation of fast-moving objects.
  arXiv  Detail & Related papers  (2021-03-25T14:35:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.