Judging It, Washing It: Scoring and Greenwashing Corporate Climate Disclosures using Large Language Models

• URL: http://arxiv.org/abs/2502.15094v1
• Date: Thu, 20 Feb 2025 23:23:45 GMT
• Title: Judging It, Washing It: Scoring and Greenwashing Corporate Climate Disclosures using Large Language Models
• Abstract summary: We investigate the use of the LLM-as-a-Judge (LLMJ) methodology for scoring company-submitted reports on emissions reduction targets and progress.<n>We probe the behavior of an LLM when it is prompted to greenwash a response subject to accuracy and length constraints.<n>We find that two LLMJ scoring systems, numerical rating and pairwise comparison, are effective in distinguishing high-performing companies from others.
• Score: 2.7898966850590625
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the use of large language models (LLMs) to both evaluate and greenwash corporate climate disclosures. First, we investigate the use of the LLM-as-a-Judge (LLMJ) methodology for scoring company-submitted reports on emissions reduction targets and progress. Second, we probe the behavior of an LLM when it is prompted to greenwash a response subject to accuracy and length constraints. Finally, we test the robustness of the LLMJ methodology against responses that may be greenwashed using an LLM. We find that two LLMJ scoring systems, numerical rating and pairwise comparison, are effective in distinguishing high-performing companies from others, with the pairwise comparison system showing greater robustness against LLM-greenwashed responses.

Related papers

• Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale [34.83728471879048]
  We study the LLM-as-a-judge problem by comparing two kinds of raters: humans and LLMs.<n>We find that LLM judgments are not perfectly consistent across scales on subjective benchmarks.<n>The grading scale of 0-5 yields the strongest human-LLM alignment.
  arXiv  Detail & Related papers  (2026-01-06T22:12:06Z)
• CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks [63.562924932512765]
  Large Language Models (LLMs) have advanced the state-of-the-art in various coding tasks.<n>LLMs can also serve as judges, assessing and comparing the quality of responses generated by other models.
  arXiv  Detail & Related papers  (2025-07-14T17:56:29Z)
• DRE: An Effective Dual-Refined Method for Integrating Small and Large Language Models in Open-Domain Dialogue Evaluation [21.229539297323488]
  Large Language Models (LLMs) excel at many tasks but struggle with ambiguous scenarios where multiple valid responses exist.<n>Small Language Models (SLMs) demonstrate robustness in such scenarios but are susceptible to misleading or adversarial inputs.<n>We introduce SLIDE (Small and Large Integrated for Dialogue Evaluation), a method integrating SLMs and LLMs via adaptive weighting.
  arXiv  Detail & Related papers  (2025-06-04T23:41:31Z)
• Towards Contamination Resistant Benchmarks [0.6906005491572401]
  evaluating large language models (LLMs) properly is crucial for understanding their potential and addressing concerns such as safety.<n> contamination stands out as a key issue that undermines the reliability of evaluations.<n>We propose a benchmark based on Caesar ciphers (e.g., "ab" to "bc" when the shift is 1), which, despite its simplicity, is an excellent example of a contamination resistant benchmark.
  arXiv  Detail & Related papers  (2025-05-13T09:35:40Z)
• The Other Side of the Coin: Exploring Fairness in Retrieval-Augmented Generation [73.16564415490113]
  Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by retrieving relevant document from external knowledge sources. We propose two approaches, FairFT and FairFilter, to mitigate the fairness issues introduced by RAG for small-scale LLMs.
  arXiv  Detail & Related papers  (2025-04-11T10:17:10Z)
• Towards Automated Fact-Checking of Real-World Claims: Exploring Task Formulation and Assessment with LLMs [32.45604456988931]
  This study establishes baseline comparisons for Automated Fact-Checking (AFC) using Large Language Models (LLMs)<n>We evaluate Llama-3 models of varying sizes on 17,856 claims collected from PolitiFact (2007-2024) using evidence retrieved via restricted web searches.<n>Our results show that larger LLMs consistently outperform smaller LLMs in classification accuracy and justification quality without fine-tuning.
  arXiv  Detail & Related papers  (2025-02-13T02:51:17Z)
• Preference Leakage: A Contamination Problem in LLM-as-a-judge [69.96778498636071]
  Large Language Models (LLMs) as judges and LLM-based data synthesis have emerged as two fundamental LLM-driven data annotation methods.<n>In this work, we expose preference leakage, a contamination problem in LLM-as-a-judge caused by the relatedness between the synthetic data generators and LLM-based evaluators.
  arXiv  Detail & Related papers  (2025-02-03T17:13:03Z)
• LLM2: Let Large Language Models Harness System 2 Reasoning [65.89293674479907]
  Large language models (LLMs) have exhibited impressive capabilities across a myriad of tasks, yet they occasionally yield undesirable outputs.<n>We introduce LLM2, a novel framework that combines an LLM with a process-based verifier.<n>LLMs2 is responsible for generating plausible candidates, while the verifier provides timely process-based feedback to distinguish desirable and undesirable outputs.
  arXiv  Detail & Related papers  (2024-12-29T06:32:36Z)
• Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge [84.34545223897578]
  Despite their excellence in many domains, potential issues are under-explored, undermining their reliability and the scope of their utility. We identify 12 key potential biases and propose a new automated bias quantification framework-CALM- which quantifies and analyzes each type of bias in LLM-as-a-Judge. Our work highlights the need for stakeholders to address these issues and remind users to exercise caution in LLM-as-a-Judge applications.
  arXiv  Detail & Related papers  (2024-10-03T17:53:30Z)
• An In-depth Evaluation of Large Language Models in Sentence Simplification with Error-based Human Assessment [9.156064716689833]
  This study provides in-depth insights into LLMs' performance while ensuring the reliability of the evaluation. We select both closed-source and open-source LLMs, including GPT-4, Qwen2.5-72B, and Llama-3.2-3B. Results show that GPT-4 generally generates fewer erroneous simplification outputs compared to the current state-of-the-art. However, LLMs have their limitations, as seen in GPT-4's and Qwen2.5-72B's struggle with lexical paraphrasing.
  arXiv  Detail & Related papers  (2024-03-08T00:19:24Z)
• Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomy [48.29181662640212]
  Factual inconsistencies pose a significant hurdle for the faithful summarization by generative models. We consolidate key error types of inconsistent facts in summaries, and incorporate them to facilitate both the zero-shot and supervised paradigms of LLMs.
  arXiv  Detail & Related papers  (2024-02-20T08:41:23Z)
• Measuring Moral Inconsistencies in Large Language Models [16.47371312298185]
  A Large Language Model (LLM) is considered consistent if semantically equivalent prompts produce semantically equivalent responses. We show that even state-of-the-art LLMs are highly inconsistent in their generations, questioning their reliability. We propose a novel information-theoretic measure called Semantic Graph Entropy (SGE) to measure the consistency of an LLM in moral scenarios.
  arXiv  Detail & Related papers  (2024-01-26T18:05:47Z)
• Mind's Mirror: Distilling Self-Evaluation Capability and Comprehensive Thinking from Large Language Models [20.28989820878285]
  Large language models (LLMs) have achieved remarkable advancements in natural language processing. The massive scale and computational demands of these models present formidable challenges when considering their practical deployment in resource-constrained environments.
  arXiv  Detail & Related papers  (2023-11-15T18:56:23Z)
• Assessing the Reliability of Large Language Model Knowledge [78.38870272050106]
  Large language models (LLMs) have been treated as knowledge bases due to their strong performance in knowledge probing tasks. How do we evaluate the capabilities of LLMs to consistently produce factually correct answers? We propose MOdel kNowledge relIabiliTy scORe (MONITOR), a novel metric designed to directly measure LLMs' factual reliability.
  arXiv  Detail & Related papers  (2023-10-15T12:40:30Z)
• LLMRec: Benchmarking Large Language Models on Recommendation Task [54.48899723591296]
  The application of Large Language Models (LLMs) in the recommendation domain has not been thoroughly investigated. We benchmark several popular off-the-shelf LLMs on five recommendation tasks, including rating prediction, sequential recommendation, direct recommendation, explanation generation, and review summarization. The benchmark results indicate that LLMs displayed only moderate proficiency in accuracy-based tasks such as sequential and direct recommendation.
  arXiv  Detail & Related papers  (2023-08-23T16:32:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.