The Counterfactual-Shapley Value: Attributing Change in System Metrics

• URL: http://arxiv.org/abs/2208.08399v1
• Date: Wed, 17 Aug 2022 16:48:20 GMT
• Title: The Counterfactual-Shapley Value: Attributing Change in System Metrics
• Authors: Amit Sharma, Hua Li, Jian Jiao
• Abstract summary: A key component of an attribution question is estimating counterfactual: the (hypothetical) change in the system metric due to a specified change in a single input. We propose a method to estimate counterfactuals using time-series predictive models and construct an attribution score, CF-Shapley. As a real-world application, we analyze a query-ad matching system with the goal of attributing observed change in a metric for ad matching density.
• Score: 10.804568364995982
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Given an unexpected change in the output metric of a large-scale system, it is important to answer why the change occurred: which inputs caused the change in metric? A key component of such an attribution question is estimating the counterfactual: the (hypothetical) change in the system metric due to a specified change in a single input. However, due to inherent stochasticity and complex interactions between parts of the system, it is difficult to model an output metric directly. We utilize the computational structure of a system to break up the modelling task into sub-parts, such that each sub-part corresponds to a more stable mechanism that can be modelled accurately over time. Using the system's structure also helps to view the metric as a computation over a structural causal model (SCM), thus providing a principled way to estimate counterfactuals. Specifically, we propose a method to estimate counterfactuals using time-series predictive models and construct an attribution score, CF-Shapley, that is consistent with desirable axioms for attributing an observed change in the output metric. Unlike past work on causal shapley values, our proposed method can attribute a single observed change in output (rather than a population-level effect) and thus provides more accurate attribution scores when evaluated on simulated datasets. As a real-world application, we analyze a query-ad matching system with the goal of attributing observed change in a metric for ad matching density. Attribution scores explain how query volume and ad demand from different query categories affect the ad matching density, leading to actionable insights and uncovering the role of external events (e.g., "Cheetah Day") in driving the matching density.

Related papers

• Uncertainty measurement for complex event prediction in safety-critical systems [0.36832029288386137]
  Complex events processing (CEP) uncertainty is critical for embedded and safety-critical systems. This paper exemplifies how we can measure uncertainty for the perception and prediction of events. We present and discuss our results, which are very promising within our field of research and work.
  arXiv  Detail & Related papers  (2024-11-02T15:51:37Z)
• Multiply-Robust Causal Change Attribution [15.501106533308798]
  We develop a new estimation strategy that combines regression and re-weighting methods to quantify the contribution of each causal mechanism. Our method demonstrates excellent performance in Monte Carlo simulations, and we show its usefulness in an empirical application.
  arXiv  Detail & Related papers  (2024-04-12T22:57:01Z)
• Enriching Disentanglement: From Logical Definitions to Quantitative Metrics [59.12308034729482]
  Disentangling the explanatory factors in complex data is a promising approach for data-efficient representation learning. We establish relationships between logical definitions and quantitative metrics to derive theoretically grounded disentanglement metrics. We empirically demonstrate the effectiveness of the proposed metrics by isolating different aspects of disentangled representations.
  arXiv  Detail & Related papers  (2023-05-19T08:22:23Z)
• Variable Importance Matching for Causal Inference [73.25504313552516]
  We describe a general framework called Model-to-Match that achieves these goals. Model-to-Match uses variable importance measurements to construct a distance metric. We operationalize the Model-to-Match framework with LASSO.
  arXiv  Detail & Related papers  (2023-02-23T00:43:03Z)
• Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics [64.81682222169113]
  How reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system-level correlations. We identify two ways in which the definition of the system-level correlation is inconsistent with how metrics are used to evaluate systems in practice.
  arXiv  Detail & Related papers  (2022-04-21T15:52:14Z)
• Time Series of Non-Additive Metrics: Identification and Interpretation of Contributing Factors of Variance by Linear Decomposition [0.0]
  The research paper addresses linear decomposition of time series of non-additive metrics. Non-additive metrics, such as ratios, are widely used in a variety of domains.
  arXiv  Detail & Related papers  (2022-04-14T01:15:28Z)
• TACTiS: Transformer-Attentional Copulas for Time Series [76.71406465526454]
  estimation of time-varying quantities is a fundamental component of decision making in fields such as healthcare and finance. We propose a versatile method that estimates joint distributions using an attention-based decoder. We show that our model produces state-of-the-art predictions on several real-world datasets.
  arXiv  Detail & Related papers  (2022-02-07T21:37:29Z)
• Disentanglement Analysis with Partial Information Decomposition [31.56299813238937]
  disentangled representations aim at reversing the process by mapping data to multiple random variables that individually capture distinct generative factors. Current disentanglement metrics are designed to measure the concentration, e.g., absolute deviation, variance, or entropy, of each variable conditioned by each generative factor. In this work, we use the Partial Information Decomposition framework to evaluate information sharing between more than two variables, and build a framework, including a new disentanglement metric.
  arXiv  Detail & Related papers  (2021-08-31T11:09:40Z)
• Grounding Representation Similarity with Statistical Testing [8.296135566684065]
  We argue that measures should have sensitivity to changes that affect functional behavior, and specificity against changes that do not. We quantify this through a variety of functional behaviors including probing accuracy and robustness to distribution shift. We find that current metrics exhibit different weaknesses, note that a classical baseline performs surprisingly well, and highlight settings where all metrics appear to fail.
  arXiv  Detail & Related papers  (2021-08-03T17:58:16Z)
• Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests [87.60900567941428]
  A spurious correlation' is the dependence of a model on some aspect of the input data that an analyst thinks shouldn't matter. In machine learning, these have a know-it-when-you-see-it character. We study stress testing using the tools of causal inference.
  arXiv  Detail & Related papers  (2021-05-31T14:39:38Z)
• AttriMeter: An Attribute-guided Metric Interpreter for Person Re-Identification [100.3112429685558]
  Person ReID systems only provide a distance or similarity when matching two persons. We propose an Attribute-guided Metric Interpreter, named AttriMeter, to semantically and quantitatively explain the results of CNN-based ReID models.
  arXiv  Detail & Related papers  (2021-03-02T03:37:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.