Accuracy and Reliability of Large Language Models in Cosmetic Chemistry and Skin Health: A Benchmarking Study

• URL: http://arxiv.org/abs/2608.14631v1
• Date: Fri, 24 Jul 2026 03:40:19 GMT
• Title: Accuracy and Reliability of Large Language Models in Cosmetic Chemistry and Skin Health: A Benchmarking Study
• Abstract summary: We benchmarked 14 Large Language Models (LLMs) on a set of topics related to cosmetic chemistry.<n>Overall performance was poor, with the most pronounced deficits in quantitative reasoning and structural identification tasks.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As consumers increasingly turn to AI chatbots for skincare advice, the technical accuracy of Large Language Models (LLMs) in cosmetic chemistry remains largely under-evaluated. We benchmarked 14 LLMs on a structured set of topics related to cosmetic chemistry, including the chemical properties of specific cosmetic ingredients and common cosmetic scenarios that may be of interest to consumers. Web search was disabled throughout to assess each model's internalized knowledge rather than its internet retrieval capacity. Overall performance was poor, with the most pronounced deficits in quantitative reasoning and structural identification tasks. While models handled general skincare questions with reasonability, responses consistently lacked the technical depth required for informed consumer decision-making. Notably, conversation with AI can pose a risk: outputs that sound authoritative but contain technical errors are less likely to generate skepticism compared to responses that explicitly acknowledge uncertainty. These findings suggest that general-purpose LLMs, trained predominantly on unverified public data, are currently not reliable sources of cosmetic chemistry information. Progress on two fronts, fine-tuning verified chemical and dermatological datasets, and substantial improvements to algorithmic reasoning, will likely be needed before these tools can be considered as resources for public use.

Related papers

• Chem World: A Large-Scale Benchmark and Physics-Informed Framework for Trustworthy Chemical Property Prediction [8.367237766104159]
  We introduce Chem World, a comprehensive benchmark for chemical property prediction.<n>Chem World integrates 17 diverse chemical datasets with over 800,000 molecular samples.<n>We also propose Mixture-PINN, a physics-informed neural network based prediction framework.
  arXiv  Detail & Related papers  (2026-07-30T11:50:52Z)
• Unveiling Latent Knowledge in Chemistry Language Models through Sparse Autoencoders [42.033443425253644]
  We extend sparse autoencoder techniques to uncover and examine interpretable features within chemistry language models.<n>Our findings reveal that these models encode a rich landscape of chemical concepts.<n>Our approach provides a generalisable framework for uncovering latent knowledge in chemistry-focused AI systems.
  arXiv  Detail & Related papers  (2025-12-08T22:20:01Z)
• QCBench: Evaluating Large Language Models on Domain-Specific Quantitative Chemistry [19.804237919102903]
  QCBench is a Quantitative Chemistry oriented benchmark comprising 350 computational chemistry problems across 7 chemistry subfields.<n>Each problem is structured to prevent shortcuts and demand explicit numerical reasoning.<n>QCBench enables fine-grained diagnosis of computational weaknesses, reveals model-specific limitations, and lays the groundwork for future improvements.
  arXiv  Detail & Related papers  (2025-08-03T08:55:42Z)
• ChemAU: Harness the Reasoning of LLMs in Chemical Research with Adaptive Uncertainty Estimation [21.30938446415292]
  Chemistry problems typically involve long and complex reasoning steps, which contain specific terminology.<n>ChemAU identifies gaps in chemistry knowledge and precisely supplements chemical expertise with the specialized domain model.
  arXiv  Detail & Related papers  (2025-06-01T18:45:49Z)
• MedBrowseComp: Benchmarking Medical Deep Research and Computer Use [10.565661515629412]
  MedBrowseComp is a benchmark that systematically tests an agent's ability to retrieve and synthesize medical facts.<n>It contains more than 1,000 human-curated questions that mirror clinical scenarios.<n>Applying MedBrowseComp to frontier agentic systems reveals performance shortfalls as low as ten percent.
  arXiv  Detail & Related papers  (2025-05-20T22:42:33Z)
• Exploring the Generalizability of Factual Hallucination Mitigation via Enhancing Precise Knowledge Utilization [49.95746521480879]
  We introduce PKUE (Precise Knowledge Utilization Enhancement), which fine-tunes the model on self-generated responses to precise and simple factual questions.<n>Extensive experiments demonstrate that PKUE significantly improves LLM overall performance.
  arXiv  Detail & Related papers  (2025-02-26T13:34:52Z)
• Chemical knowledge-informed framework for privacy-aware retrosynthesis learning [72.39098405805318]
  Current machine learning-based retrosynthesis gathers reaction data from multiple sources into one single edge to train prediction models.<n>This paradigm poses considerable privacy risks as it necessitates broad data availability across organizational boundaries.<n>In the present study, we introduce the chemical knowledge-informed framework (CKIF), a privacy-preserving approach for learning retrosynthesis models.
  arXiv  Detail & Related papers  (2025-02-26T13:13:24Z)
• Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgments [108.55277188617035]
  Large language models (LLMs) have been widely adopted in various downstream task domains, but their abilities to directly recall and apply factual medical knowledge remains under-explored.<n>We introduce the Medical Knowledge Judgment dataset (MKJ), a dataset derived from the Unified Medical Language System (UMLS), a comprehensive repository of standardized vocabularies and knowledge graphs.<n>Through a binary classification framework, MKJ evaluates LLMs' grasp of fundamental medical facts by having them assess the validity of concise, one-hop statements.
  arXiv  Detail & Related papers  (2025-02-20T05:27:51Z)
• ScholarChemQA: Unveiling the Power of Language Models in Chemical Research Question Answering [54.80411755871931]
  Question Answering (QA) effectively evaluates language models' reasoning and knowledge depth. Chemical QA plays a crucial role in both education and research by effectively translating complex chemical information into readily understandable format. This dataset reflects typical real-world challenges, including an imbalanced data distribution and a substantial amount of unlabeled data that can be potentially useful. We introduce a QAMatch model, specifically designed to effectively answer chemical questions by fully leveraging our collected data.
  arXiv  Detail & Related papers  (2024-07-24T01:46:55Z)
• Are large language models superhuman chemists? [4.87961182129702]
  Large language models (LLMs) have gained widespread interest due to their ability to process human language and perform tasks on which they have not been explicitly trained. Here, we introduce "ChemBench," an automated framework for evaluating the chemical knowledge and reasoning abilities of state-of-the-art LLMs. We curated more than 2,700 question-answer pairs, evaluated leading open- and closed-source LLMs, and found that the best models outperformed the best human chemists.
  arXiv  Detail & Related papers  (2024-04-01T20:56:25Z)
• ChemMiner: A Large Language Model Agent System for Chemical Literature Data Mining [56.15126714863963]
  ChemMiner is an end-to-end framework for extracting chemical data from literature.<n>ChemMiner incorporates three specialized agents: a text analysis agent for coreference mapping, a multimodal agent for non-textual information extraction, and a synthesis analysis agent for data generation.<n> Experimental results demonstrate reaction identification rates comparable to human chemists while significantly reducing processing time, with high accuracy, recall, and F1 scores.
  arXiv  Detail & Related papers  (2024-02-20T13:21:46Z)
• Chemist-X: Large Language Model-empowered Agent for Reaction Condition Recommendation in Chemical Synthesis [55.30328162764292]
  Chemist-X is a comprehensive AI agent that automates the reaction condition optimization (RCO) task in chemical synthesis.<n>The agent uses retrieval-augmented generation (RAG) technology and AI-controlled wet-lab experiment executions.<n>Results of our automatic wet-lab experiments, achieved by fully LLM-supervised end-to-end operation with no human in the lope, prove Chemist-X's ability in self-driving laboratories.
  arXiv  Detail & Related papers  (2023-11-16T01:21:33Z)
• Unassisted Noise Reduction of Chemical Reaction Data Sets [59.127921057012564]
  We propose a machine learning-based, unassisted approach to remove chemically wrong entries from data sets. Our results show an improved prediction quality for models trained on the cleaned and balanced data sets.
  arXiv  Detail & Related papers  (2021-02-02T09:34:34Z)
• Machine Learning in Nano-Scale Biomedical Engineering [77.75587007080894]
  We review the existing research regarding the use of machine learning in nano-scale biomedical engineering. The main challenges that can be formulated as ML problems are classified into the three main categories. For each of the presented methodologies, special emphasis is given to its principles, applications, and limitations.
  arXiv  Detail & Related papers  (2020-08-05T15:45:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.