Scorio.jl: A Julia package for ranking stochastic responses

• URL: http://arxiv.org/abs/2603.14103v1
• Date: Sat, 14 Mar 2026 20:12:56 GMT
• Title: Scorio.jl: A Julia package for ranking stochastic responses
• Abstract summary: Scorio.jl is a Julia package for evaluating and ranking systems from repeated responses to shared tasks.<n>It provides a common tensor-based interface for direct score-based, pairwise, psychometric, voting, graph, and listwise methods, so the same benchmark can be analyzed under multiple ranking assumptions.
• Score: 4.784378058952302
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scorio.jl is a Julia package for evaluating and ranking systems from repeated responses to shared tasks. It provides a common tensor-based interface for direct score-based, pairwise, psychometric, voting, graph, and listwise methods, so the same benchmark can be analyzed under multiple ranking assumptions. We describe the package design, position it relative to existing Julia tools, and report pilot experiments on synthetic rank recovery, stability under limited trials, and runtime scaling.

Related papers

• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• TensorKit.jl: A Julia package for large-scale tensor computations, with a hint of category theory [0.0]
  This paper introduces the design philosophy, core functionalities, and distinctive features, including how to handle abelian, non-abelian, and anyonic symmetries.<n>We highlight the software's flexibility, performance, and its capability to extend to new tensor types and symmetries.
  arXiv  Detail & Related papers  (2025-08-13T16:35:11Z)
• Bipartite Ranking Fairness through a Model Agnostic Ordering Adjustment [54.179859639868646]
  We propose a model agnostic post-processing framework xOrder for achieving fairness in bipartite ranking. xOrder is compatible with various classification models and ranking fairness metrics, including supervised and unsupervised fairness metrics. We evaluate our proposed algorithm on four benchmark data sets and two real-world patient electronic health record repositories.
  arXiv  Detail & Related papers  (2023-07-27T07:42:44Z)
• A novel evaluation methodology for supervised Feature Ranking algorithms [0.0]
  This paper proposes a new evaluation methodology for Feature Rankers. By making use of synthetic datasets, feature importance scores can be known beforehand, allowing more systematic evaluation. To facilitate large-scale experimentation using the new methodology, a benchmarking framework was built in Python, called fseval.
  arXiv  Detail & Related papers  (2022-07-09T12:00:36Z)
• Just Rank: Rethinking Evaluation with Word and Sentence Similarities [105.5541653811528]
  intrinsic evaluation for embeddings lags far behind, and there has been no significant update since the past decade. This paper first points out the problems using semantic similarity as the gold standard for word and sentence embedding evaluations. We propose a new intrinsic evaluation method called EvalRank, which shows a much stronger correlation with downstream tasks.
  arXiv  Detail & Related papers  (2022-03-05T08:40:05Z)
• pygrank: A Python Package for Graph Node Ranking [13.492381728793612]
  We introduce pygrank, an open source Python package to define, run and evaluate node ranking algorithms. We provide object-oriented and extensively unit-tested algorithm components, such as graph filters, post-processors, measures, benchmarks and online tuning.
  arXiv  Detail & Related papers  (2021-10-18T13:13:21Z)
• Estimating leverage scores via rank revealing methods and randomization [50.591267188664666]
  We study algorithms for estimating the statistical leverage scores of rectangular dense or sparse matrices of arbitrary rank. Our approach is based on combining rank revealing methods with compositions of dense and sparse randomized dimensionality reduction transforms.
  arXiv  Detail & Related papers  (2021-05-23T19:21:55Z)
• PiRank: Learning To Rank via Differentiable Sorting [85.28916333414145]
  We propose PiRank, a new class of differentiable surrogates for ranking. We show that PiRank exactly recovers the desired metrics in the limit of zero temperature.
  arXiv  Detail & Related papers  (2020-12-12T05:07:36Z)
• BAT.jl -- A Julia-based tool for Bayesian inference [0.0]
  We describe the development of a multi-purpose software for Bayesian statistical inference, BAT.jl, written in the Julia language. The major design considerations and implemented algorithms are summarized here, together with a test suite that ensures the proper functioning of the algorithms.
  arXiv  Detail & Related papers  (2020-08-07T12:55:52Z)
• The JuliaConnectoR: a functionally oriented interface for integrating Julia in R [0.0]
  We develop the R package JuliaConnectoR, available from the CRAN repository and GitHub. For maintainability and stability, we base communication between R and Julia on TCP. This makes it easy to develop R extensions with Julia or to simply call functionality from Julia packages in R.
  arXiv  Detail & Related papers  (2020-05-13T14:18:34Z)
• SetRank: A Setwise Bayesian Approach for Collaborative Ranking from Implicit Feedback [50.13745601531148]
  We propose a novel setwise Bayesian approach for collaborative ranking, namely SetRank, to accommodate the characteristics of implicit feedback in recommender system. Specifically, SetRank aims at maximizing the posterior probability of novel setwise preference comparisons. We also present the theoretical analysis of SetRank to show that the bound of excess risk can be proportional to $sqrtM/N$.
  arXiv  Detail & Related papers  (2020-02-23T06:40:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.