IIRSim Studio: A Dashboard for User Simulation

• URL: http://arxiv.org/abs/2604.23406v1
• Date: Sat, 25 Apr 2026 18:53:17 GMT
• Title: IIRSim Studio: A Dashboard for User Simulation
• Abstract summary: IIRSim Studio is a visual environment for composing simulation pipelines on top of simulation frameworks.<n>IIRSim Studio is available as a hosted service and as a portable containerized deployment.
• Score: 2.3423826856721557
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: User simulation is a valuable methodology for evaluation in Information Retrieval (IR), enabling low-cost experimentation and counterfactual analysis. However, existing simulation frameworks are primarily code-centric libraries that require substantial setup effort, which limits adoption and hinders reproducibility. The bottleneck is not the simulation engines themselves, but the lack of infrastructure connecting experiment design, execution, and sharing into a single verifiable workflow. This paper introduces IIRSim Studio, a web-based workbench that addresses this gap through four contributions: (1) a visual environment for composing simulation pipelines on top of simulation frameworks, serving both novices learning simulation concepts and experts piloting large-scale experiments; (2) a component lifecycle that supports authoring, versioning, and sharing custom simulation components through Git-backed storage and runtime injection; (3) a provenance model based on experiment bundles and environment templates that makes the scope of replication explicit; and (4) a shared-task workflow, demonstrated through the re-deployment of a Sim4IA micro-task. IIRSim Studio is available as a hosted service and as a portable containerized deployment.

Related papers

• Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment [51.83961778298564]
  We present Flower Hub, a platform for publishing, discovering, and executing decentralized and federated applications.<n>We show how it enables reproducible benchmarking by packaging benchmarks as executable, versioned applications with standardized metadata, pinned dependencies, and explicit evaluation.
  arXiv  Detail & Related papers  (2026-08-25T20:09:44Z)
• Simthesizer: An Agent-Driven Simulation Framework for LLM Serving Systems [10.265484416904352]
  We present Simthesizer, a framework that realizes agent-driven simulator development.<n>Simthesizer introduces a composable simulator infrastructure that uniformly expresses the complete serving workflow.<n>On identical workloads, Simthesizer simulates up to 284.96x and 23.19x faster than two state-of-the-art simulators.
  arXiv  Detail & Related papers  (2026-08-25T14:58:26Z)
• Mind the Sim2Real Gap in User Simulation for Agentic Tasks [101.69142591891234]
  We formalize the Sim2Real gap in user simulation and present the first study running the full $$-bench protocol with real humans.<n> Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity.<n>These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle.
  arXiv  Detail & Related papers  (2026-03-11T19:12:31Z)
• Large Language Model Agent for User-friendly Chemical Process Simulations [0.0]
  A large language model (LLM) agent is integrated with AVEVA Process Model Protocol (MCP), allowing natural language simulations.<n>Two case studies assess the framework across different task complexities and interaction modes.<n>The framework benefits both educational purposes, by translating technical concepts and demonstrating, and experienced practitioners by automating data extraction, speeding routine tasks, and supporting.<n>While current limitations such as oversimplification, calculation errors, and technical hiccups mean expert oversight is still needed, the framework suggests LLM-based agents can become valuable collaborators.
  arXiv  Detail & Related papers  (2026-01-15T12:18:45Z)
• Simulating Environments with Reasoning Models for Agent Training [55.98861707136674]
  Building bespoke environments for training is heavy, brittle, and limits progress.<n>We propose two frameworks: Simia-SFT and Simia-RL.<n>Simia-SFT and Simia-RL enable scalable agent training without environment engineering.
  arXiv  Detail & Related papers  (2025-11-03T18:29:57Z)
• YuLan-OneSim: Towards the Next Generation of Social Simulator with Large Language Models [50.35333054932747]
  We introduce a novel social simulator called YuLan-OneSim.<n>Users can simply describe and refine their simulation scenarios through natural language interactions with our simulator.<n>We implement 50 default simulation scenarios spanning 8 domains, including economics, sociology, politics, psychology, organization, demographics, law, and communication.
  arXiv  Detail & Related papers  (2025-05-12T14:05:17Z)
• Phantora: Maximizing Code Reuse in Simulation-based Machine Learning System Performance Estimation [13.326000659635378]
  Phantora is a hybrid GPU cluster simulator for performance estimation of machine learning training workloads.<n>It allows direct reuse of ML framework source code in simulation, avoiding the need for reimplementation.<n>Phantora supports three state-of-the-art training frameworks out-of-the-box.
  arXiv  Detail & Related papers  (2025-05-02T22:36:24Z)
• Metadata practices for simulation workflows [0.0]
  We present general practices for acquiring and handling metadata that are agnostic to software and hardware.<n>These consist of two steps: 1) recording and storing raw metadata, and 2) selecting and structuring metadata.<n>As a proof of concept, we develop the Archivist, a Python tool to help with the second step, and use it to apply our practices to high-performance computing use cases from neuroscience and hydrology.
  arXiv  Detail & Related papers  (2024-08-30T14:12:31Z)
• In Situ Framework for Coupling Simulation and Machine Learning with Application to CFD [51.04126395480625]
  Recent years have seen many successful applications of machine learning (ML) to facilitate fluid dynamic computations. As simulations grow, generating new training datasets for traditional offline learning creates I/O and storage bottlenecks. This work offers a solution by simplifying this coupling and enabling in situ training and inference on heterogeneous clusters.
  arXiv  Detail & Related papers  (2023-06-22T14:07:54Z)
• Synthetic Data-Based Simulators for Recommender Systems: A Survey [55.60116686945561]
  This survey aims at providing a comprehensive overview of the recent trends in the field of modeling and simulation. We start with the motivation behind the development of frameworks implementing the simulations -- simulators. We provide a new consistent classification of existing simulators based on their functionality, approbation, and industrial effectiveness.
  arXiv  Detail & Related papers  (2022-06-22T19:33:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.