Mediators in Determining what Processing BERT Performs First

• URL: http://arxiv.org/abs/2104.06400v1
• Date: Tue, 13 Apr 2021 17:58:52 GMT
• Title: Mediators in Determining what Processing BERT Performs First
• Abstract summary: We consider the prediction's context length, namely the length of the span whose processing is minimally required to perform the prediction. We show that not controlling for context length may lead to contradictory conclusions as to the localization patterns of the network.
• Score: 35.32217580058933
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Probing neural models for the ability to perform downstream tasks using their activation patterns is often used to localize what parts of the network specialize in performing what tasks. However, little work addressed potential mediating factors in such comparisons. As a test-case mediating factor, we consider the prediction's context length, namely the length of the span whose processing is minimally required to perform the prediction. We show that not controlling for context length may lead to contradictory conclusions as to the localization patterns of the network, depending on the distribution of the probing dataset. Indeed, when probing BERT with seven tasks, we find that it is possible to get 196 different rankings between them when manipulating the distribution of context lengths in the probing dataset. We conclude by presenting best practices for conducting such comparisons in the future.

Related papers

• Unified Video Dense Prediction from Disjoint Data [71.9957801898161]
  We introduce UniD, a unified video model that jointly predicts eight dense scene properties-depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials.<n>We propose a simple yet effective distillation step in which per-task experts supervise a unified backbone through lightweight task projectors.<n>UniD achieves competitive performance against per-task specialists and multi-task baselines.
  arXiv  Detail & Related papers  (2026-07-23T17:59:50Z)
• Multi-Task Bayesian In-Context Learning [41.16196712546687]
  We introduce a multi-task in-context learning framework for amortized hierarchical Bayesian predictive inference.<n>A transformer trained on sequences of prior and target tasks learns to adapt its predictions across families of priors.<n>Our method matches oracle predictors while being orders of magnitude faster than existing approaches.
  arXiv  Detail & Related papers  (2026-06-18T17:50:10Z)
• Prediction-Powered Inference Across Many Tasks for AI Evaluation & Social Science Research [6.716363754264257]
  Many applications require statistically valid inference across many related tasks, while using only a handful of high-quality labels per hypothesis.<n>We introduce a prediction-powered inference framework that uses labeled data from related tasks to improve power while preserving task-specific inference.<n>We show that cross-task recalibration can substantially reduce confidence interval widths when labels are scarce.
  arXiv  Detail & Related papers  (2026-05-28T02:09:38Z)
• A Framework for Quantifying How Pre-Training and Context Benefit In-Context Learning [52.07397258423034]
  We propose a new framework to analyze the ICL performance in a class of realistic settings.<n>We derive the precise relationship between ICL performance, context length and the KL divergence between pre-train and query task distribution.
  arXiv  Detail & Related papers  (2025-10-26T09:21:29Z)
• Provable Benefits of Task-Specific Prompts for In-context Learning [44.768199865867494]
  In this work, we consider a novel setting where the global task distribution can be partitioned into a union of conditional task distributions. We then examine the use of task-specific prompts and prediction heads for learning the prior information associated with the conditional task distribution using a one-layer attention model.
  arXiv  Detail & Related papers  (2025-03-03T22:37:03Z)
• Multi-Task Self-Supervised Time-Series Representation Learning [3.31490164885582]
  Time-series representation learning can extract representations from data with temporal dynamics and sparse labels. We propose a new time-series representation learning method by combining the advantages of self-supervised tasks. We evaluate the proposed framework on three downstream tasks: time-series classification, forecasting, and anomaly detection.
  arXiv  Detail & Related papers  (2023-03-02T07:44:06Z)
• Effective Cross-Task Transfer Learning for Explainable Natural Language Inference with T5 [50.574918785575655]
  We compare sequential fine-tuning with a model for multi-task learning in the context of boosting performance on two tasks. Our results show that while sequential multi-task learning can be tuned to be good at the first of two target tasks, it performs less well on the second and additionally struggles with overfitting.
  arXiv  Detail & Related papers  (2022-10-31T13:26:08Z)
• Wasserstein Task Embedding for Measuring Task Similarities [14.095478018850374]
  Measuring similarities between different tasks is critical in a broad spectrum of machine learning problems. We leverage the optimal transport theory and define a novel task embedding for supervised classification. We show that the proposed embedding leads to a significantly faster comparison of tasks compared to related approaches.
  arXiv  Detail & Related papers  (2022-08-24T18:11:04Z)
• Conditional Meta-Learning of Linear Representations [57.90025697492041]
  Standard meta-learning for representation learning aims to find a common representation to be shared across multiple tasks. In this work we overcome this issue by inferring a conditioning function, mapping the tasks' side information into a representation tailored to the task at hand. We propose a meta-algorithm capable of leveraging this advantage in practice.
  arXiv  Detail & Related papers  (2021-03-30T12:02:14Z)
• Time-series Imputation and Prediction with Bi-Directional Generative Adversarial Networks [0.3162999570707049]
  We present a model for the combined task of imputing and predicting values for irregularly observed and varying length time-series data with missing entries. Our model learns how to impute missing elements in-between (imputation) or outside of the input time steps (prediction), hence working as an effective any-time prediction tool for time-series data.
  arXiv  Detail & Related papers  (2020-09-18T15:47:51Z)
• Temporally Correlated Task Scheduling for Sequence Learning [143.70523777803723]
  In many applications, a sequence learning task is usually associated with multiple temporally correlated auxiliary tasks. We introduce a learnable scheduler to sequence learning, which can adaptively select auxiliary tasks for training. Our method significantly improves the performance of simultaneous machine translation and stock trend forecasting.
  arXiv  Detail & Related papers  (2020-07-10T10:28:54Z)
• Calibrated Adversarial Refinement for Stochastic Semantic Segmentation [5.849736173068868]
  We present a strategy for learning a calibrated predictive distribution over semantic maps, where the probability associated with each prediction reflects its ground truth correctness likelihood. We demonstrate the versatility and robustness of the approach by achieving state-of-the-art results on the multigrader LIDC dataset and on a modified Cityscapes dataset with injected ambiguities. We show that the core design can be adapted to other tasks requiring learning a calibrated predictive distribution by experimenting on a toy regression dataset.
  arXiv  Detail & Related papers  (2020-06-23T16:39:59Z)
• Low Resource Multi-Task Sequence Tagging -- Revisiting Dynamic Conditional Random Fields [67.51177964010967]
  We compare different models for low resource multi-task sequence tagging that leverage dependencies between label sequences for different tasks. We find that explicit modeling of inter-dependencies between task predictions outperforms single-task as well as standard multi-task models.
  arXiv  Detail & Related papers  (2020-05-01T07:11:34Z)
• Improving Multi-Turn Response Selection Models with Complementary Last-Utterance Selection by Instance Weighting [84.9716460244444]
  We consider utilizing the underlying correlation in the data resource itself to derive different kinds of supervision signals. We conduct extensive experiments in two public datasets and obtain significant improvement in both datasets.
  arXiv  Detail & Related papers  (2020-02-18T06:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.