SC3: The Multi-Solvent Solubility Challenge and Benchmark

• URL: http://arxiv.org/abs/2606.07656v1
• Date: Wed, 03 Jun 2026 08:57:00 GMT
• Title: SC3: The Multi-Solvent Solubility Challenge and Benchmark
• Abstract summary: We introduce SC3, a multi-solvent solubility benchmark built on BigSolDB v2.1 with three contributions.<n>We observe this is a gap unclosed by any deep alternative tested.<n>We perform three follow-on analyses: data scaling, transfer from quantum-chemistry solvation energies, and feature-level attribution.
• Score: 11.51982791748735
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Solubility prediction is a standard benchmark in computational chemistry, yet multi-solvent models which reportedly approach the experimental-noise ceiling (i.e. the aleatoric limit) are not yet reliable enough to be deployed. We argue that this gap is partly artefactual: published benchmarks differ in curation policies, evaluate on count-weighted RMSE that hides failure on tail-heavy solvent distributions, and treat the widely cited 0.6-0.8 log S inter-laboratory figure as the aleatoric ceiling even though it reflects worst-case, not expected, disagreement. We introduce SC3, a multi-solvent solubility benchmark built on BigSolDB v2.1 with three contributions: (i) a reproducible curation pipeline yielding 101,535 measurements over 1,327 solutes and 206 solvents, with a recalibrated aleatoric floor of 0.106 log S-roughly 6 times tighter than the conventional figure; (ii) nested Gold/Silver/Bronze consensus tiers with per-point standard deviation, three leakage-checked splits, and a multi-solvent metric suite (PS-RMSE, Z-RMSE); and (iii) a 31-model benchmark across six families, whose best Bronze PS-RMSE sits at 5 times the aleatoric limit, and we observe this is a gap unclosed by any deep alternative tested. We perform three follow-on analyses: data scaling, transfer from quantum-chemistry solvation energies, and feature-level attribution, which demonstrates that calibrated per-point uncertainty is a reusable infrastructure for diagnosis beyond point prediction.

Related papers

• CUSP: Decomposable Collective Uncertainty for Multi-Agent Multimodal Reasoning [6.428392127849263]
  We present CUSP, a training-free uncertainty framework that maps multiple VLM responses to a shared semantic response space.<n>We report two complementary system-level signals: collective uncertainty, the dispersion of the pooled opinion, and Jensen-Shannon divergence (JSD)<n>In static multi-VLM ensembles, collective uncertainty is the strongest signal in the small-model regime.
  arXiv  Detail & Related papers  (2026-09-04T20:27:47Z)
• Distribution-Free Conformal Prediction for Steel Fatigue Strength: Marginal Validity Is Not Enough [0.0]
  Predicting fatigue failure in steel components experimentally is costly because it requires testing across multiple compositions and processing conditions.<n>Studies using the NIMS MatNavi steel fatigue dataset often report high point-prediction accuracy but rely on aggregate error metrics.<n>This paper is the first to apply conformal prediction to steel fatigue strength.
  arXiv  Detail & Related papers  (2026-08-05T18:13:28Z)
• Conformalized Large Language Models under Configuration Shift [46.502741608809174]
  Conformal prediction is a distribution-free framework for uncertainty quantification.<n>Nonconformity scores are often induced by an inference pipeline, not just a fixed model.<n>We show that configuration shift consistently erodes CP validity.
  arXiv  Detail & Related papers  (2026-08-02T19:34:41Z)
• Modeling Normal Is All You Need: Joint Latent Clustering for Anomaly Detection in Multimodal Cyber-Physical Systems [47.027290803102666]
  Normal behaviour is the union of many imbalanced, curved, thin-fringed operating regimes.<n>We model the normal law with a jointly learned latent representation plus explicit Gaussian-mixture mode clustering.<n>We evaluate under a deliberately fair protocol: raw point-wise metrics with no point adjustment, a trivial-detector difficulty split, prevalence-matched F1, and train-normal-only calibration.
  arXiv  Detail & Related papers  (2026-07-07T10:10:00Z)
• Design and Evaluation of Multi-Agent AI Oracle Systems for Prediction Market Resolution [0.0]
  Prediction markets aggregate collective intelligence to forecast uncertain events.<n>Existing oracle systems tradeoff fast but brittle automation against accurate but costly human arbitration.<n>We evaluate whether multi-agent LLM architectures can improve oracle resolution accuracy over single-model baselines.
  arXiv  Detail & Related papers  (2026-05-29T03:44:19Z)
• CHRONOS: Temporally-Aware Multi-Agent Coordination for Evolving Data Marketplaces [3.252472808549431]
  Temporal knowledge-graph data marketplaces face three coupled failures in static designs.<n>We present CHRONOS, a three-layer architecture providing a unified treatment of these challenges with explicit public and private separation.
  arXiv  Detail & Related papers  (2026-05-22T17:47:45Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks [45.86413490112477]
  Large language models (LLMs) are used in reliability-sensitive applications.<n>Estimateing rare failure probability with tight confidence bounds requires prohibitively large LLM inference sizes.<n>We propose to learn a sampling distribution concentrated on failure-prone inputs via the cross-entropy method.
  arXiv  Detail & Related papers  (2026-05-11T20:23:44Z)
• HEOM-in-Calibration-Loop: Exposing Non-Markovian Bath Signatures That Markovian Calibration Elides in Superconducting-Qubit Tune-Up [2.3774562616857717]
  We integrate a QuTiP 5.x hierarchical-equations-of-motion solver driven by a Tier-1/f Burkard bath into a multi-protocol calibration DAG.<n>We benchmark it against sesolve and mesolve on a frozen platform in a pulse-level simulator.
  arXiv  Detail & Related papers  (2026-04-23T09:15:13Z)
• Diffusion-MPC in Discrete Domains: Feasibility Constraints, Horizon Effects, and Critic Alignment: Case study with Tetris [0.0]
  We study diffusion-based model predictive control (Diffusion-MPC) in discrete domains using Tetris as a case study.<n>Our planner samples candidate placement sequences with a MaskGIT-style discrete denoiser and selects actions via reranking.<n>We analyze three key factors: (1) feasibility-constrained sampling via logit masking over valid placements, (2) reranking strategies using a score, a pretrained DQN critic, and a hybrid combination, and (3) compute scaling in candidate count and planning horizon.
  arXiv  Detail & Related papers  (2026-03-02T19:35:38Z)
• Information Fidelity in Tool-Using LLM Agents: A Martingale Analysis of the Model Context Protocol [69.11739400975445]
  We introduce the first theoretical framework for analyzing error accumulation in Model Context Protocol (MCP) agents.<n>We show that cumulative distortion exhibits linear growth and high-probability deviations bounded by $O(sqrtT)$.<n>Key findings include: semantic weighting reduces distortion by 80%, and periodic re-grounding approximately every 9 steps suffices for error control.
  arXiv  Detail & Related papers  (2026-02-10T21:08:53Z)
• Detecting Unobserved Confounders: A Kernelized Regression Approach [46.52607207396279]
  Kernel Regression Confounder Detection (KRCD) is a novel method for detecting unobserved confounding in nonlinear observational data under single-environment conditions.<n>Test statistic whose significant deviation from zero indicates unobserved confounding.<n>Experiments on synthetic benchmarks and the Twins dataset demonstrate that KRCD not only outperforms existing baselines but also achieves superior computational efficiency.
  arXiv  Detail & Related papers  (2026-01-01T04:26:02Z)
• Causal Judge Evaluation: Calibrated Surrogate Metrics for LLM Systems [0.29465623430708904]
  Uncalibrated scores can invert preferences, naive confidence intervals on uncalibrated scores achieve near-0% coverage, and importance-weighted estimators collapse under limited overlap.<n>We introduce Causal Judge Evaluation, a framework that fixes all three failures.
  arXiv  Detail & Related papers  (2025-12-11T22:16:24Z)
• Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty [49.19257648205146]
  We propose an unsupervised conformal inference framework for generation.<n>Our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP.<n>The result is a label-free, API-compatible gate for test-time filtering.
  arXiv  Detail & Related papers  (2025-09-26T23:40:47Z)
• Distribution-free binary classification: prediction sets, confidence intervals and calibration [106.50279469344937]
  We study three notions of uncertainty quantification -- calibration, confidence intervals and prediction sets -- for binary classification in the distribution-free setting. We derive confidence intervals for binned probabilities for both fixed-width and uniform-mass binning. As a consequence of our 'tripod' theorems, these confidence intervals for binned probabilities lead to distribution-free calibration.
  arXiv  Detail & Related papers  (2020-06-18T14:17:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.