Incentive-Theoretic Bayesian Inference for Collaborative Science
- URL: http://arxiv.org/abs/2307.03748v2
- Date: Thu, 8 Feb 2024 17:47:12 GMT
- Title: Incentive-Theoretic Bayesian Inference for Collaborative Science
- Authors: Stephen Bates, Michael I. Jordan, Michael Sklar, Jake A. Soloff
- Abstract summary: We study hypothesis testing when there is an agent with a private prior about an unknown parameter.
We show how the principal can conduct statistical inference that leverages the information that is revealed by an agent's strategic behavior.
- Score: 59.15962177829337
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Contemporary scientific research is a distributed, collaborative endeavor,
carried out by teams of researchers, regulatory institutions, funding agencies,
commercial partners, and scientific bodies, all interacting with each other and
facing different incentives. To maintain scientific rigor, statistical methods
should acknowledge this state of affairs. To this end, we study hypothesis
testing when there is an agent (e.g., a researcher or a pharmaceutical company)
with a private prior about an unknown parameter and a principal (e.g., a
policymaker or regulator) who wishes to make decisions based on the parameter
value. The agent chooses whether to run a statistical trial based on their
private prior and then the result of the trial is used by the principal to
reach a decision. We show how the principal can conduct statistical inference
that leverages the information that is revealed by an agent's strategic
behavior -- their choice to run a trial or not. In particular, we show how the
principal can design a policy to elucidate partial information about the
agent's private prior beliefs and use this to control the posterior probability
of the null. One implication is a simple guideline for the choice of
significance threshold in clinical trials: the type-I error level should be set
to be strictly less than the cost of the trial divided by the firm's profit if
the trial is successful.
Related papers
- Online Decision Mediation [72.80902932543474]
Consider learning a decision support assistant to serve as an intermediary between (oracle) expert behavior and (imperfect) human behavior.
In clinical diagnosis, fully-autonomous machine behavior is often beyond ethical affordances.
arXiv Detail & Related papers (2023-10-28T05:59:43Z) - Causal Strategic Learning with Competitive Selection [10.237954203296187]
We study the problem of agent selection in causal strategic learning under multiple decision makers.
We show that the optimal selection rule is a trade-off between selecting the best agents and providing incentives to maximise the agents' improvement.
We provide a cooperative protocol which all decision makers must collectively adopt to recover the true causal parameters.
arXiv Detail & Related papers (2023-08-30T18:43:11Z) - Ground(less) Truth: A Causal Framework for Proxy Labels in
Human-Algorithm Decision-Making [29.071173441651734]
We identify five sources of target variable bias that can impact the validity of proxy labels in human-AI decision-making tasks.
We develop a causal framework to disentangle the relationship between each bias.
We conclude by discussing opportunities to better address target variable bias in future research.
arXiv Detail & Related papers (2023-02-13T16:29:11Z) - Policy Learning with Asymmetric Counterfactual Utilities [0.6138671548064356]
We consider optimal policy learning with asymmetric counterfactual utility functions.
We derive minimax decision rules by minimizing the maximum expected utility loss.
We show that one can learn minimax loss decision rules from observed data by solving intermediate classification problems.
arXiv Detail & Related papers (2022-06-21T15:44:49Z) - Principal-Agent Hypothesis Testing [54.154244569974864]
We consider the relationship between a regulator (the principal) and an experimenter (the agent) such as a pharmaceutical company.
The efficacy of the drug is not known to the regulator, so the pharmaceutical company must run a costly trial to prove efficacy to the regulator.
We show how to design protocols that are robust to an agent's strategic actions, and derive the optimal protocol in the presence of strategic entrants.
arXiv Detail & Related papers (2022-05-13T17:59:23Z) - Private Sequential Hypothesis Testing for Statisticians: Privacy, Error
Rates, and Sample Size [24.149533870085175]
We study the sequential hypothesis testing problem under a slight variant of differential privacy, known as Renyi differential privacy.
We present a new private algorithm based on Wald's Sequential Probability Ratio Test (SPRT) that also gives strong theoretical privacy guarantees.
arXiv Detail & Related papers (2022-04-10T04:15:50Z) - Empirical Estimates on Hand Manipulation are Recoverable: A Step Towards
Individualized and Explainable Robotic Support in Everyday Activities [80.37857025201036]
Key challenge for robotic systems is to figure out the behavior of another agent.
Processing correct inferences is especially challenging when (confounding) factors are not controlled experimentally.
We propose equipping robots with the necessary tools to conduct observational studies on people.
arXiv Detail & Related papers (2022-01-27T22:15:56Z) - Statistical discrimination in learning agents [64.78141757063142]
Statistical discrimination emerges in agent policies as a function of both the bias in the training population and of agent architecture.
We show that less discrimination emerges with agents that use recurrent neural networks, and when their training environment has less bias.
arXiv Detail & Related papers (2021-10-21T18:28:57Z) - Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment [61.24399136715106]
We consider the issue of strategic behaviour in various peer-assessment tasks, including peer grading of exams or homeworks and peer review in hiring or promotions.
Our focus is on designing methods for detection of such manipulations.
Specifically, we consider a setting in which agents evaluate a subset of their peers and output rankings that are later aggregated to form a final ordering.
arXiv Detail & Related papers (2020-10-08T15:08:40Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.