SC-Arena: A Natural Language Benchmark for Single-Cell Reasoning with Knowledge-Augmented Evaluation

• URL: http://arxiv.org/abs/2602.23199v1
• Date: Thu, 26 Feb 2026 16:50:28 GMT
• Title: SC-Arena: A Natural Language Benchmark for Single-Cell Reasoning with Knowledge-Augmented Evaluation
• Abstract summary: We present SC-ARENA, a natural language evaluation framework tailored to single-cell foundation models.<n>SC-ARENA formalizes a virtual cell abstraction that unifies evaluation targets by representing both intrinsic attributes and gene-level interactions.
• Score: 24.956743572453153
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly applied in scientific research, offering new capabilities for knowledge discovery and reasoning. In single-cell biology, however, evaluation practices for both general and specialized LLMs remain inadequate: existing benchmarks are fragmented across tasks, adopt formats such as multiple-choice classification that diverge from real-world usage, and rely on metrics lacking interpretability and biological grounding. We present SC-ARENA, a natural language evaluation framework tailored to single-cell foundation models. SC-ARENA formalizes a virtual cell abstraction that unifies evaluation targets by representing both intrinsic attributes and gene-level interactions. Within this paradigm, we define five natural language tasks (cell type annotation, captioning, generation, perturbation prediction, and scientific QA) that probe core reasoning capabilities in cellular biology. To overcome the limitations of brittle string-matching metrics, we introduce knowledge-augmented evaluation, which incorporates external ontologies, marker databases, and scientific literature to support biologically faithful and interpretable judgments. Experiments and analysis across both general-purpose and domain-specialized LLMs demonstrate that (i) under the Virtual Cell unified evaluation paradigm, current models achieve uneven performance on biologically complex tasks, particularly those demanding mechanistic or causal understanding; and (ii) our knowledge-augmented evaluation framework ensures biological correctness, provides interpretable, evidence-grounded rationales, and achieves high discriminative capacity, overcoming the brittleness and opacity of conventional metrics. SC-Arena thus provides a unified and interpretable framework for assessing LLMs in single-cell biology, pointing toward the development of biology-aligned, generalizable foundation models.

Related papers

• Position: Genomic Model Research Must Move Beyond Anecdotal Evaluation of Interpretability Methods [8.920725431658777]
  We argue for a validation framework analogous to clinical trials.<n>Just as trials require rigorous design and adverse-event reporting, genomic interpretability must move beyond cherry-picked plausibility.
  arXiv  Detail & Related papers  (2026-05-29T12:20:12Z)
• Unlocking Biological Workflows for Robust Protein-Text Question Answering: A Dual-Dimensional RAG Framework [60.82334952881798]
  Protein-Text Question Answering (QA) is crucial for interpreting biological sequences through natural language.<n>We propose 2D-ProteinRAG, a novel framework that empowers Large Language Models (LLMs) to operate within the gold-standard biological research workflow.<n>We show that 2D-Protein consistently achieves state-of-the-art performance, outperforming fine-tuned baselines and other RAG methods.
  arXiv  Detail & Related papers  (2026-05-17T05:03:24Z)
• A Systematic Survey and Benchmark of Deep Learning for Molecular Property Prediction in the Foundation Model Era [57.5440098776759]
  Molecular property prediction integrates quantum chemistry, cheminformatics, and deep learning to connect molecular structure with biological behavior.<n>This survey traces four complementary paradigms, including Quantum, Descriptor Machine Learning, Geometric Deep Learning, and Foundation Models.<n> Benchmark analyses integrate evidence from both widely used datasets and datasets reflecting industry perspectives.
  arXiv  Detail & Related papers  (2026-04-17T15:16:33Z)
• MAT-Cell: A Multi-Agent Tree-Structured Reasoning Framework for Batch-Level Single-Cell Annotation [81.82645086402366]
  We propose MAT-Cell, a neuro-symbolic reasoning framework that reframes single-cell analysis into constructive, verifiable proof generation.<n> MAT-Cell injects symbolic constraints through adaptive Retrieval-Augmented Generation (RAG) to ground neural reasoning in biological axioms and reduce transcriptomic noise.<n>It significantly outperforms state-of-the-art (SOTA) models and maintains robust per-formance in challenging scenarios.
  arXiv  Detail & Related papers  (2026-04-07T04:56:06Z)
• BABE: Biology Arena BEnchmark [51.53220868983288]
  BABE is a benchmark designed to evaluate the experimental reasoning capabilities of biological AI systems.<n>Our benchmark provides a robust framework for assessing how well AI systems can reason like practicing scientists.
  arXiv  Detail & Related papers  (2026-02-05T16:39:20Z)
• SciHorizon-GENE: Benchmarking LLM for Life Sciences Inference from Gene Knowledge to Functional Understanding [30.790301729371475]
  Large language models (LLMs) have shown growing promise in biomedical research, particularly for knowledge-driven interpretation tasks.<n>We introduce SciHorizon-GENE, a large-scale gene-centric benchmark constructed from authoritative biological databases.<n>The benchmark integrates curated knowledge for over 190K human genes and comprises more than 540K questions covering diverse gene-to-function reasoning scenarios.
  arXiv  Detail & Related papers  (2026-01-19T08:06:35Z)
• Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMs [85.69785384599827]
  Human-object interaction (HOI) detection aims to localize human-object pairs and the interactions between them.<n>Existing methods operate under a closed-world assumption, treating the task as a classification problem over a small, predefined verb set.<n>We propose GRASP-HO, a novel Generative Reasoning And Steerable Perception framework that reformulates HOI detection from the closed-set classification task to the open-vocabulary generation problem.
  arXiv  Detail & Related papers  (2025-12-19T14:41:50Z)
• Learning Cell-Aware Hierarchical Multi-Modal Representations for Robust Molecular Modeling [74.25438319700929]
  We propose CHMR (Cell-aware Hierarchical Multi-modal Representations), a robust framework that models local-global dependencies between molecules and cellular responses.<n> evaluated on nine public benchmarks spanning 728 tasks, CHMR outperforms state-of-the-art baselines.<n>Results demonstrate the advantage of hierarchy-aware, multimodal learning for reliable and biologically grounded molecular representations.
  arXiv  Detail & Related papers  (2025-11-26T07:15:00Z)
• Discovering Interpretable Biological Concepts in Single-cell RNA-seq Foundation Models [3.810388351528255]
  Single-cell RNA-seq foundation models achieve strong performance on downstream tasks but remain black boxes.<n>Recent work has shown that sparse dictionary learning can extract concepts from deep learning models.<n>We introduce a novel concept-based interpretability framework for single-cell RNA-seq models.
  arXiv  Detail & Related papers  (2025-10-29T08:52:55Z)
• Contrastive Learning Enhances Language Model Based Cell Embeddings for Low-Sample Single Cell Transcriptomics [3.7907528918903797]
  Large language models (LLMs) have shown ability in generating rich representations across domains such as natural language processing and generation, computer vision, and multimodal learning.<n>We present a computational framework that integrates single-cell RNA sequencing (scRNA-seq) with LLMs to derive knowledge-informed gene embeddings.
  arXiv  Detail & Related papers  (2025-09-28T00:45:39Z)
• CellVerse: Do Large Language Models Really Understand Cell Biology? [74.34984441715517]
  We introduce CellVerse, a unified language-centric question-answering benchmark that integrates four types of single-cell multi-omics data.<n>We systematically evaluate the performance across 14 open-source and closed-source LLMs ranging from 160M to 671B on CellVerse.
  arXiv  Detail & Related papers  (2025-05-09T06:47:23Z)
• Contextualizing biological perturbation experiments through language [3.704686482174365]
  PerturbQA is a benchmark for structured reasoning over perturbation experiments.<n>We evaluate state-of-the-art machine learning and statistical approaches for modeling perturbations.<n>As a proof of feasibility, we introduce Summer (SUMMarize, retrievE, and answeR), a simple, domain-informed LLM framework.
  arXiv  Detail & Related papers  (2025-02-28T18:15:31Z)
• GENERator: A Long-Context Generative Genomic Foundation Model [66.46537421135996]
  We present GENERator, a generative genomic foundation model featuring a context length of 98k base pairs (bp) and 1.2B parameters.<n>Trained on an expansive dataset comprising 386B bp of DNA, the GENERator demonstrates state-of-the-art performance across both established and newly proposed benchmarks.<n>It also shows significant promise in sequence optimization, particularly through the prompt-responsive generation of enhancer sequences with specific activity profiles.
  arXiv  Detail & Related papers  (2025-02-11T05:39:49Z)
• Causal Representation Learning from Multimodal Biomedical Observations [57.00712157758845]
  We develop flexible identification conditions for multimodal data and principled methods to facilitate the understanding of biomedical datasets.<n>Key theoretical contribution is the structural sparsity of causal connections between modalities.<n>Results on a real-world human phenotype dataset are consistent with established biomedical research.
  arXiv  Detail & Related papers  (2024-11-10T16:40:27Z)
• SylloBio-NLI: Evaluating Large Language Models on Biomedical Syllogistic Reasoning [3.3903891679981593]
  SylloBio-NLI is a framework to systematically instantiate diverse syllogistic arguments for Natural Language Inference (NLI)<n>We evaluate Large Language Models (LLMs) on identifying valid conclusions and extracting evidence across 28 syllogistic schemes.<n>We find that biomedical syllogistic reasoning is particularly challenging for zero-shot LLMs, which achieve an average accuracy between 70% on generalized modus ponens and 23% on disjunctive syllogism.
  arXiv  Detail & Related papers  (2024-10-18T12:02:41Z)
• GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models [56.63218531256961]
  We introduce GenBench, a benchmarking suite specifically tailored for evaluating the efficacy of Genomic Foundation Models. GenBench offers a modular and expandable framework that encapsulates a variety of state-of-the-art methodologies. We provide a nuanced analysis of the interplay between model architecture and dataset characteristics on task-specific performance.
  arXiv  Detail & Related papers  (2024-06-01T08:01:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.