What do Large Language Models Say About Animals? Investigating Risks of Animal Harm in Generated Text

• URL: http://arxiv.org/abs/2503.04804v2
• Date: Mon, 10 Mar 2025 03:02:59 GMT
• Title: What do Large Language Models Say About Animals? Investigating Risks of Animal Harm in Generated Text
• Authors: Arturs Kanepajs, Aditi Basu, Sankalpa Ghose, Constance Li, Akshat Mehta, Ronak Mehta, Samuel David Tucker-Davis, Eric Zhou, Bob Fischer,
• Abstract summary: We present the Animal Harm Assessment (AHA), a novel evaluation of risks of animal harm in large language models (LLMs)<n>Our dataset comprises 1,850 curated questions from Reddit post titles and 2,500 synthetic questions based on 50 animal categories and 50 ethical scenarios.<n>AHA produces meaningful evaluation results when applied to frontier LLMs.
• Score: 3.2905913457832057
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As machine learning systems become increasingly embedded in human society, their impact on the natural world continues to escalate. Technical evaluations have addressed a variety of potential harms from large language models (LLMs) towards humans and the environment, but there is little empirical work regarding harms towards nonhuman animals. Following the growing recognition of animal protection in regulatory and ethical AI frameworks, we present the Animal Harm Assessment (AHA), a novel evaluation of risks of animal harm in LLM-generated text. Our dataset comprises 1,850 curated questions from Reddit post titles and 2,500 synthetic questions based on 50 animal categories (e.g., cats, reptiles) and 50 ethical scenarios, with further 70-30 public-private split. Scenarios include open-ended questions about how to treat animals, practical scenarios with potential animal harm, and willingness-to-pay measures for the prevention of animal harm. Using the LLM-as-a-judge framework, answers are evaluated for their potential to increase or decrease harm, and evaluations are debiased for the tendency to judge their own outputs more favorably. We show that AHA produces meaningful evaluation results when applied to frontier LLMs, revealing significant differences between models, animal categories, scenarios, and subreddits. We conclude with future directions for technical research and the challenges of building evaluations on complex social and moral topics.

Related papers

• Measurement of LLM's Philosophies of Human Nature [113.47929131143766]
  We design the standardized psychological scale specifically targeting large language models (LLM) We show that current LLMs exhibit a systemic lack of trust in humans. We propose a mental loop learning framework, which enables LLM to continuously optimize its value system.
  arXiv  Detail & Related papers  (2025-04-03T06:22:19Z)
• Rel-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliance [73.19687314438133]
  We study how reliance is affected by contextual features of an interaction. We find that contextual characteristics significantly affect human reliance behavior. Our results show that calibration and language quality alone are insufficient in evaluating the risks of human-LM interactions.
  arXiv  Detail & Related papers  (2024-07-10T18:00:05Z)
• Animal Behavior Analysis Methods Using Deep Learning: A Survey [3.9086052572489653]
  State-of-the-art deep learning models have demonstrated remarkable accuracy in classifying various forms of animal data. The article culminates in a comprehensive discussion of key research directions within deep learning that hold potential for advancing the field of animal behavior studies.
  arXiv  Detail & Related papers  (2024-05-22T21:17:54Z)
• ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming [64.86326523181553]
  ALERT is a large-scale benchmark to assess safety based on a novel fine-grained risk taxonomy. It aims to identify vulnerabilities, inform improvements, and enhance the overall safety of the language models.
  arXiv  Detail & Related papers  (2024-04-06T15:01:47Z)
• The Ethics of ChatGPT in Medicine and Healthcare: A Systematic Review on Large Language Models (LLMs) [0.0]
  ChatGPT, Large Language Models (LLMs) have received enormous attention in healthcare. Despite their potential benefits, researchers have underscored various ethical implications. This work aims to map the ethical landscape surrounding the current stage of deployment of LLMs in medicine and healthcare.
  arXiv  Detail & Related papers  (2024-03-21T15:20:07Z)
• The Case for Animal-Friendly AI [0.0]
  We develop a proof-of-concept Evaluation System for evaluating animal consideration in large language models (LLMs) Preliminary results suggest that the outcomes of the tested models can be benchmarked regarding the consideration they give to animals. This study serves as a step towards more useful and responsible AI systems that better recognize and respect the vital interests and perspectives of all sentient beings.
  arXiv  Detail & Related papers  (2024-03-02T12:41:11Z)
• Eagle: Ethical Dataset Given from Real Interactions [74.7319697510621]
  We create datasets extracted from real interactions between ChatGPT and users that exhibit social biases, toxicity, and immoral problems. Our experiments show that Eagle captures complementary aspects, not covered by existing datasets proposed for evaluation and mitigation of such ethical challenges.
  arXiv  Detail & Related papers  (2024-02-22T03:46:02Z)
• Mapping the Ethics of Generative AI: A Comprehensive Scoping Review [0.0]
  We conduct a scoping review on the ethics of generative artificial intelligence, including especially large language models and text-to-image models. Our analysis provides a taxonomy of 378 normative issues in 19 topic areas and ranks them according to their prevalence in the literature. The study offers a comprehensive overview for scholars, practitioners, or policymakers, condensing the ethical debates surrounding fairness, safety, harmful content, hallucinations, privacy, interaction risks, security, alignment, societal impacts, and others.
  arXiv  Detail & Related papers  (2024-02-13T09:38:17Z)
• Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench [83.41621219298489]
  We evaluate Large Language Models' (LLMs) anthropomorphic capabilities using the emotion appraisal theory from psychology. We collect a dataset containing over 400 situations that have proven effective in eliciting the eight emotions central to our study. We conduct a human evaluation involving more than 1,200 subjects worldwide.
  arXiv  Detail & Related papers  (2023-08-07T15:18:30Z)
• Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity [19.94836502156002]
  Large language models (LLMs) may exhibit social prejudice and toxicity, posing ethical and societal dangers of consequences resulting from irresponsibility. We empirically benchmark ChatGPT on multiple sample datasets. We find that a significant number of ethical risks cannot be addressed by existing benchmarks.
  arXiv  Detail & Related papers  (2023-01-30T13:20:48Z)
• AP-10K: A Benchmark for Animal Pose Estimation in the Wild [83.17759850662826]
  We propose AP-10K, the first large-scale benchmark for general animal pose estimation. AP-10K consists of 10,015 images collected and filtered from 23 animal families and 60 species. Results provide sound empirical evidence on the superiority of learning from diverse animals species in terms of both accuracy and generalization ability.
  arXiv  Detail & Related papers  (2021-08-28T10:23:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.