Environmental large language model Evaluation (ELLE) dataset: A Benchmark for Evaluating Generative AI applications in Eco-environment Domain

• URL: http://arxiv.org/abs/2501.06277v1
• Date: Fri, 10 Jan 2025 12:48:29 GMT
• Title: Environmental large language model Evaluation (ELLE) dataset: A Benchmark for Evaluating Generative AI applications in Eco-environment Domain
• Authors: Jing Guo, Nan Li, Ming Xu,
• Abstract summary: Generative AI holds significant potential for ecological and environmental applications. The Environmental Large Language model Evaluation (ELLE) dataset is the first benchmark designed to assess large language models. ELLE dataset includes 1,130 question answer pairs across 16 environmental topics, categorized by domain, difficulty, and type.
• Score: 6.246205449407889
• License:
• Abstract: Generative AI holds significant potential for ecological and environmental applications such as monitoring, data analysis, education, and policy support. However, its effectiveness is limited by the lack of a unified evaluation framework. To address this, we present the Environmental Large Language model Evaluation (ELLE) question answer (QA) dataset, the first benchmark designed to assess large language models and their applications in ecological and environmental sciences. The ELLE dataset includes 1,130 question answer pairs across 16 environmental topics, categorized by domain, difficulty, and type. This comprehensive dataset standardizes performance assessments in these fields, enabling consistent and objective comparisons of generative AI performance. By providing a dedicated evaluation tool, ELLE dataset promotes the development and application of generative AI technologies for sustainable environmental outcomes. The dataset and code are available at https://elle.ceeai.net/ and https://github.com/CEEAI/elle.

Related papers

• OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain [62.89809156574998]
  We introduce an omnidirectional and automatic RAG benchmark, OmniEval, in the financial domain. Our benchmark is characterized by its multi-dimensional evaluation framework. Our experiments demonstrate the comprehensiveness of OmniEval, which includes extensive test datasets.
  arXiv  Detail & Related papers  (2024-12-17T15:38:42Z)
• Combining Observational Data and Language for Species Range Estimation [63.65684199946094]
  We propose a novel approach combining millions of citizen science species observations with textual descriptions from Wikipedia. Our framework maps locations, species, and text descriptions into a common space, enabling zero-shot range estimation from textual descriptions. Our approach also acts as a strong prior when combined with observational data, resulting in more accurate range estimation with less data.
  arXiv  Detail & Related papers  (2024-10-14T17:22:55Z)
• ExioML: Eco-economic dataset for Machine Learning in Global Sectoral Sustainability [11.925553950065895]
  This paper introduces ExioML, the first Machine Learning benchmark dataset designed for sustainability analysis. A crucial greenhouse gas emission regression task was conducted to evaluate sectoral sustainability and demonstrate the usability of the dataset.
  arXiv  Detail & Related papers  (2024-06-11T17:06:34Z)
• EcoVerse: An Annotated Twitter Dataset for Eco-Relevance Classification, Environmental Impact Analysis, and Stance Detection [0.0]
  EcoVerse is an annotated English Twitter dataset of 3,023 tweets spanning a wide spectrum of environmental topics. We propose a three-level annotation scheme designed for Eco-Relevance Classification, Stance Detection, and introducing an original approach for Environmental Impact Analysis.
  arXiv  Detail & Related papers  (2024-04-08T01:21:11Z)
• When is Off-Policy Evaluation (Reward Modeling) Useful in Contextual Bandits? A Data-Centric Perspective [64.73162159837956]
  evaluating the value of a hypothetical target policy with only a logged dataset is important but challenging. We propose DataCOPE, a data-centric framework for evaluating a target policy given a dataset. Our empirical analysis of DataCOPE in the logged contextual bandit settings using healthcare datasets confirms its ability to evaluate both machine-learning and human expert policies.
  arXiv  Detail & Related papers  (2023-11-23T17:13:37Z)
• Challenges in data-based geospatial modeling for environmental research and practice [19.316860936437823]
  Data-based geospatial modelling using machine learning (ML) has gained popularity in environmental research. This survey reviews common nuances in geospatial modelling, such as imbalanced data, spatial autocorrelation, prediction errors, model generalisation, domain specificity, and uncertainty estimation.
  arXiv  Detail & Related papers  (2023-11-18T12:30:49Z)
• FREE: The Foundational Semantic Recognition for Modeling Environmental Ecosystems [28.166089112650926]
  FREE maps available environmental data into a text space and then converts the traditional predictive modeling task in environmental science to a semantic recognition problem. When used for long-term prediction, FREE has the flexibility to incorporate newly collected observations to enhance future prediction. Free is evaluated in the context of two societally important real-world applications, predicting stream water temperature in the Delaware River Basin and predicting annual corn yield in Illinois and Iowa.
  arXiv  Detail & Related papers  (2023-11-17T00:53:09Z)
• TextEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction [131.7684896032888]
  We present TextEE, a standardized, fair, and reproducible benchmark for event extraction. TextEE comprises standardized data preprocessing scripts and splits for 16 datasets spanning eight diverse domains. We evaluate five varied large language models on our TextEE benchmark and demonstrate how they struggle to achieve satisfactory performance.
  arXiv  Detail & Related papers  (2023-11-16T04:43:03Z)
• A Comparative Study of Machine Learning Algorithms for Anomaly Detection in Industrial Environments: Performance and Environmental Impact [62.997667081978825]
  This study seeks to address the demands of high-performance machine learning models with environmental sustainability. Traditional machine learning algorithms, such as Decision Trees and Random Forests, demonstrate robust efficiency and performance. However, superior outcomes were obtained with optimised configurations, albeit with a commensurate increase in resource consumption.
  arXiv  Detail & Related papers  (2023-07-01T15:18:00Z)
• Towards Environmentally Equitable AI via Geographical Load Balancing [40.142341503145275]
  This paper takes a first step toward addressing AI's environmental inequity by balancing its regional negative environmental impact. We run trace-based simulations by considering a set of 10 geographically-distributed data centers that serve inference requests for a large language AI model. The results demonstrate that existing GLB approaches may amplify environmental inequity while our proposed equity-aware GLB can significantly reduce the regional disparity in terms of carbon and water footprints.
  arXiv  Detail & Related papers  (2023-06-20T17:13:33Z)
• From Static Benchmarks to Adaptive Testing: Psychometrics in AI Evaluation [60.14902811624433]
  We discuss a paradigm shift from static evaluation methods to adaptive testing. This involves estimating the characteristics and value of each test item in the benchmark and dynamically adjusting items in real-time. We analyze the current approaches, advantages, and underlying reasons for adopting psychometrics in AI evaluation.
  arXiv  Detail & Related papers  (2023-06-18T09:54:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.