Rethinking the Evaluation and Optimization of LLM-Based Social Simulation
Abstract Overview
This paper argues that standard accuracy-based evaluation and hard-label training are poorly matched to LLM-based social simulation because human responses are inherently distributional rather than deterministic. It introduces a normalized subjectivity coefficient, based on response entropy, to analyze how the mismatch between one observed label and the underlying response distribution worsens as subjectivity increases. To address this, the authors propose Subjectivity-Adaptive soft-Label Training (SALT), which builds soft labels by aggregating responses from semantically similar contexts with a radius that adapts to estimated subjectivity. The paper also presents SubjSim, a benchmark of 19,300 persona-question contexts from 193 annotators answering 100 subjective questions, with elicited response-propensity distributions used for evaluation. Experiments evaluate models in a realistic single-observation training setting while testing against full human response distributions.
Novelty
The paper’s main novelty is to reframe social simulation as distribution matching rather than single-label prediction, supported by a formal subjectivity coefficient that links task subjectivity to failures of accuracy-based evaluation and hard-label optimization. It also introduces SALT, a context-specific soft-label aggregation method with an adaptive neighborhood radius, and pairs it with a new benchmark that provides full human response distributions for this setting.
Results
On SubjSim with Qwen3-8B, SALT outperforms the tested baselines across KL, JSD, TVD, and MMD, including against standard supervised fine-tuning. Over the full test set, it reduces KL by 77.6%, JSD by 45.9%, TVD by 31.2%, and MMD by 52.7% relative to SFT, with the largest gains in mid- and high-subjectivity settings; in the high-subjectivity stratum, KL is reduced by 96.9%. Ablations further indicate that both semantic similarity and subjectivity-adaptive neighborhood selection contribute to the improvement.
Key Points
- The paper shows theoretically that accuracy-based evaluation can mis-rank models and that single-observation hard-label training increasingly fails as response subjectivity grows.
- SALT constructs per-context soft labels by aggregating responses from semantically similar contexts, with larger neighborhoods used when predicted behavior is more diffuse.
- The new SubjSim benchmark enables distributional evaluation of social simulation by providing 19,300 contexts with elicited human response distributions, while training remains restricted to one observed action per context.