The vast majority of text transformation techniques in NLP are inherently
limited in their ability to expand input space coverage due to an implicit
constraint to preserve the original class label. In this work, we propose the
notion of sibylvariance (SIB) to describe the broader set of transforms that
relax the label-preserving constraint, knowably vary the expected class, and
lead to significantly more diverse input distributions. We offer a unified
framework to organize all data transformations, including two types of SIB: (1)
Transmutations convert one discrete kind into another, (2) Mixture Mutations
blend two or more classes together. To explore the role of sibylvariance within
NLP, we implemented 41 text transformations, including several novel techniques
like Concept2Sentence and SentMix. Sibylvariance also enables a unique form of
adaptive training that generates new input mixtures for the most confused class
pairs, challenging the learner to differentiate with greater nuance. Our
experiments on six benchmark datasets strongly support the efficacy of
sibylvariance for generalization performance, defect detection, and adversarial
robustness.
Abstract The vast majority of text transformation techniques in NLP are inherently limited in their ability to expand input space coverage due to an implicit constraint to preserve the original class label.
In this work, we propose the notion of sibylvariance (SIB) to describe the broader set of transforms that relax the labelpreserving constraint, knowably vary the expected class, and lead to significantly more diverse input distributions.
We offer a unified framework to organize all data transformations, including two types of SIB: (1) Transmutations convert one discrete kind into another, (2) Mixture Mutations blend two or more classes together.
To explore the role of sibylvariance within NLP, we implemented 41 text transformations, including several novel techniques like Concept2Sentence and SentMix.
Sibylvariance also enables a unique form of adaptive training that generates new input mixtures for the most confused class pairs, challenging the learner to differentiate with greater nuance.
Our experiments on six benchmark datasets strongly support the efficacy of sibylvariance for generalization performance, defect detection, and adversarial robustness.
During training, data augmentation can expose models to a larger portion of potential input space, consistently leading to better generalization and performance (Simard et al , 1998; Krizhevsky et al , 2012; Perez and Wang, 2017).
トレーニング中、データ拡張はモデルに潜在的な入力空間の大部分を露出させ、一貫してより良い一般化とパフォーマンスをもたらす(simard et al , 1998; krizhevsky et al , 2012; perez and wang, 2017)。
0.85
After training, creating effective test instances from existing data can expose specific model failure modes and provide actionable corrective feedback (Zhang et al , 2019; Ribeiro et al , 2020).
トレーニングの後、既存のデータから効果的なテストインスタンスを作成することで、特定のモデル障害モードを公開し、実行可能な修正フィードバックを提供することができる(Zhang et al , 2019; Ribeiro et al , 2020)。 訳抜け防止モード: トレーニングの後、既存のデータから効果的なテストインスタンスを作成する 特定のモデル障害モードを公開し、実行可能な修正フィードバックを提供する(Zhang et al, 2019; Ribeiro et al.)。 2020 ) .
0.74
While many techniques can artificially expand labeled training sets or test suites, nearly all of them
多くの技術はラベル付きトレーニングセットやテストスイートを人工的に拡張できるが、そのほとんどは
0.72
are class-preserving. That is to say, the model outputs are invariant (INV) with respect to the transformations.
クラス保存です。 つまり、モデル出力は変換に関して不変(INV)である。
0.59
This cautious constraint ensures the new data does not lie in an out-of-distribution null class which might impede the learning objective.
From the Greek sibyls, or oracles, the term parallels the oracle construction problem in software testing (Barr et al , 2015).
ギリシャ語の sibyls または oracles から、この用語はソフトウェアテストにおける oracle construction problem (barr et al , 2015) に類似している。
0.77
In a nutshell, sibylvariants either fully transmute a datum from one class ci to another cj, or mix data from multiple classes together to derive a new input with a soft label that reflects the mixed membership.
一言で言えば、sibylvariants は、あるクラス ci から別の cj へ完全に datum を変換するか、複数のクラスからデータを混合して新しい入力と混合メンバシップを反映したソフトラベルを導出する。
0.72
In this way, SIB can more strongly perturb and diversify the underlying distribution.
このように、SIBはより強く摂動し、基礎となる分布を多様化することができる。
0.58
Moreover, SIB makes possible a new type of adaptive training by synthesizing data from frequently confused class pairs, challenging the model to differentiate with greater refinement.
In the following sections, we position SIB within a broader conceptual framework for all data transforms (Section 2) and highlight several newly proposed techniques (Section 3).
To support a comprehensive evaluation of how SIB may complement or even surpass its INV counterparts, we implemented 41 new and existing techniques into an open source tool called Sibyl.
In the generalization study, SIB attained the highest accuracies in 89% (16 out of 18) of experimental configurations, with the adaptive mixture mutations being the most consistently effective.
SIB also revealed the greatest number of model defects in 83% (5 out of 6) of the testing configurations.
SIBはまた、テスト構成の83%(6つ中5つ)で最大のモデル欠陥を明らかにした。
0.80
Lastly, of all the experimental configurations where adversarial robustness was improved over the no-transform baseline, 92% (11 out of 12) of them involved SIB.
Lastly, we describe how SIB may operate theoretically and discuss potential threats to validity (Section 5) before contrasting it with related work (Section 6).
TSIB({Xi, yi}) → {Xj, yj} where Xi (cid:54)= Xj and yi (cid:54)= yj.
TSIB({Xi, yi}) → {Xj, yj} ここで Xi (cid:54) = Xj と yi (cid:54) = yj である。
0.93
(2) SIB transforms both the input Xi to Xj and the output label from yi to yj label, corresponding to the new Xj; such transformation is analogous to mutating an input and setting a corresponding oracle in metamorphic testing (Chen et al , 2020b).
For example, performing a verb-targeted antonym substitution on “I love pizza.” to generate “I hate pizza.” has the effect of negating the original semantics and will knowably affect the outcome of binary sentiment analysis.
例えば、"I hate pizza." を生成するために "I love pizza." で動詞をターゲットとした反語置換を行う場合、元の意味論を否定する効果があり、二分感情分析の結果に影響を及ぼす。
0.77
It is important to note that transformation functions are not inherently INV nor SIB.
変換関数は本質的には INV や SIB ではないことに注意する必要がある。
0.65
The same exact transformation may have a different effect on expected model behavior depending on the particular classification task.
For example, random word insertions generally have an INV effect on topic classification tasks, but would be SIB with respect to grammaticality tasks (Warstadt et al , 2018).
例えば、ランダムな単語挿入は通常、トピック分類タスクにINV効果を持つが、文法的なタスクに関してはSIBである(Warstadt et al , 2018)。
0.72
2.1 Sibylvariant Subtypes SIB can be further refined based on the types and degree of semantic shift in newly generated data:
where the final term indicates a λ-degree of membership in each label l belonging to the expected input space and is normalized as l λl = 1).
最終項は、期待入力空間に属する各ラベル l における λ-次数を表し、 l λl = 1) として正規化される。
0.75
For example, a document with topic ‘surfing’ can be combined with another document with topic ‘machine learning’ to yield a new label with probability mass placed on both topics.
2.2 Adaptive Sibylvariant Training One unique and promising aspect of SIB is to target specific class pairings dynamically during training.
2.2 Adaptive Sibylvariant Training SIBのユニークな特徴は、トレーニング中に特定のクラスペアを動的にターゲットすることである。
0.75
In much the same way that a human teacher might periodically assess a students’ understanding and alter their lesson plan accordingly, Sybil computes a confusion matrix and constructs more examples containing classes for which the model has the most difficulty differentiating.
if a topic model most frequently misclassifies ‘science’ articles as ‘business,’ adaptive SIB (denoted as αSIB) will generate new blended examples of those classes in every mini-batch until the next evaluation cycle.
Sybil supports built-in runtime monitoring for αSIB training.
sybilは組み込みのランタイム監視とαsibトレーニングをサポートする。
0.52
3 Transformations In Sybil, we defined 18 new transforms and adapt 23 existing techniques from prior work (Ribeiro et al , 2020; Morris et al , 2020; Wei and Zou, 2019) to expand the coverage of SIB and INV text transformations.
3変態 Sybilでは18の新しいトランスフォーメーションを定義し、以前の作業(Ribeiro et al , 2020; Morris et al , 2020; Wei and Zou, 2019)から既存の23のテクニックを適用し、SIBおよびINVテキストトランスフォーメーションの範囲を広げました。
0.69
At a high level, Table 1 shows these 41 transforms organized into 8 categories: Mixture (i.e., blending text), Generative (i.e. concept-based text generation), Swap (e g , substituting antonyms, synonyms, hypernyms, etc.), Negation (e g , adding or removing negation), Punctuation (e g , adding or removing punctuation), Text Insert (e g , adding negative, neutral, or positive phrases), Typos (e g adding various typos), and Emojis (e g adding or removing positive or negative emoji).
C2S is a two step process: (1) extract a short list of key concepts from a document and (2) generate a new sentence that retains critical semantic content of the original while varying its surface form, style, and even subject matter.
To accomplish this, we leveraged integrated gradients (Sundararajan et al , 2017; Pierse, 2021) to produce saliency attributions that identify the most relevant tokens for a given class label.
これを実現するために、統合的な勾配(Sundararajan et al , 2017; Pierse, 2021)を活用して、与えられたクラスラベルの最も関連性の高いトークンを識別する。
0.70
We then generate a well-composed sentence from the extracted concepts using a pre-trained BART (Lewis et al , 2019) model fine-tuned on the CommonGen dataset (Lin et al , 2019).
次に、抽出された概念から、CommonGenデータセット(Lin et al , 2019)に基づいて、事前訓練されたBART(Lewis et al , 2019)モデルを用いて、よく構成された文を生成する。
0.63
Prior to generation, it is possible to apply other transformations to the extracted concepts to encourage diversity or knowably alter the label.
For example, on the left hand side of Figure 1 an antonym substitution produces a SIB effect by changing the extracted concepts from [’stupid’, ’worse’] to [’intelligent’, ’better’].
Mixture mutations, like mixup (Zhang et al , 2017) and cutmix (Yun et al , 2019) from the image domain, take a batch of inputs and blend them together to form new inputs with an interpolated loss and they have shown robustness to adversarial attacks.
画像領域からのミックスアップ(Zhang et al , 2017)やカットミックス(Yun et al , 2019)のような混合突然変異は、一連の入力を取り込み、それらをブレンドして補間された損失で新しい入力を形成し、敵の攻撃に対して堅牢性を示している。
0.73
TextMix translates this idea to the text domain by merging two inputs and interpolating a soft label according to the proportion of tokens belonging to the constituent classes.
4 Experiments 4.1 Transformation Pipelines & Datasets To compare the potential of INV, SIB, and both (INVSIB) in aggregate, we construct a transformation pipeline (T P ) (Cubuk et al , 2019; Xie et al , 2019), where we uniformly sample n transformations of the selected kind to generate new {Xi, yi} pairs.
4 実験 4.1 変換パイプラインとデータセットによる inv, sib, and both (invsib) のポテンシャルを比較するため,変換パイプライン (cubuk et al , 2019; xie et al , 2019) を構築し,選択した種類の n 変換を一様にサンプリングして新しい {xi, yi} ペアを生成する。
0.77
We also create T P s that apply a single transform, TSINGLE, to highlight the efficacy of C2S, TextMix, SentMix, WordMix and their adaptive versions, prefixed with α.
For RQ1, we also compare against TMix (Chen et al , 2020a), EDA (Wei and Zou, 2019), and AEDA (Karimi et al , 2021).
RQ1では、TMix(Chen et al , 2020a)、EDA(Wei and Zou, 2019)、AEDA(Karimi et al , 2021)と比較する。 訳抜け防止モード: RQ1については、TMix ( Chen et al, 2020a ) と比較する。 EDA (We and Zou, 2019 ) と AEDA (Karimi et al, 2021 )。
0.77
TMix is a recent hidden-space mixture mutation for text, as opposed to Sybil’s direct mixture mutation on the input space with greater transparency and examinability.
These subsets were expanded 30× via augmentation for each T P .
これらの部分集合は各tpの増補によって30×拡大された。
0.51
In total, we generated 144 new datasets
合計144のデータセットを生成しました
0.71
(144 = 6 benchmarks * 3 levels of data availability * 8 T P s which persist data. αSIB is runtime only.)
(144 = 6 ベンチマーク * 3 レベルのデータ可用性 * 8 t p s データを保持する。 αsib は実行時のみ)
0.86
4.2 Model Setting We used a bert-base-uncased model (Devlin et al , 2018) with average pooling of encoder output, followed by a dropout layer (Srivastava et al , 2014) with probability 0.1, and a single linear layer with hidden size 768 and GELU (Hendrycks and Gimpel, 2016) activation.
4.2モデル設定 平均エンコーダ出力プーリングを伴うbert-base-uncasedモデル (Devlin et al , 2018) と, 確率0.1のドロップアウト層 (Srivastava et al , 2014) と, 隠れサイズ768およびGELU (Hendrycks and Gimpel, 2016) の1つの線形層 (Hendrycks and Gimpel, 2016) を用いた。
0.83
Maximum sentence length was set to 250. We use a batch size 16, an Adam optimizer (Kingma and Ba, 2014) with a linear warmup, a 0.1 weight decay, and compute accuracy every 2, 000 steps.
最大文長は250。 バッチサイズ16とadamオプティマイザ(kingma and ba, 2014)を使用し,線形ウォームアップ,0.1重量崩壊,20000ステップ毎に精度を計算した。
0.58
All models were trained for 30 epochs on eight Nvidia RTX A6000 GPUs, with early stopping.
(5) where λj is the degree of class membership, 1(·) is the indicator function, and yj and ˆyj are the indices of the j-th largest predicted score for the ground truth label and predicted label, respectively.
Generalization Performance For RQ1, we explore how model accuracy on the original test set is influenced by training data augmented with INV and SIB transformations.
Table 4 shows the results on six benchmarks with three levels of data availability.
表4は、データ可用性のレベルが3つある6つのベンチマークの結果を示しています。
0.56
We observe the most significant performance gains when training 10 examples per class —accuracy is improved by 4.7% on average across all datasets and by a maximum of up to 15% for IMDB.
For every dataset, either αSentMix or αTextMix is the best performing T P , while INV* actually leads to performance decreases for DBPedia, Yahoo! Answers, and IMDB.
すべてのデータセットにおいて、αSentMixまたはαTextMixが最高のパフォーマンスのTPであるのに対して、INV*はDBPedia、Yahoo! Answers、IMDBのパフォーマンス低下につながる。 訳抜け防止モード: すべてのデータセットに対して、αSentMix または αTextMix が最高のパフォーマンスの T P である。 INV * は DBPedia, Yahoo ! Answers, IMDB のパフォーマンス低下につながる。
0.87
One key reason that aided SIB in attaining strong performance is the use of adaptive training.
Test represents the number of examples per class in the test set.
テストはテストセット内のクラス毎の例の数を表します。
0.80
Source Task (Zhang et al , 2015) Topic (Zhang et al , 2015) Topic (Zhang et al , 2015) Topic (Zhang et al , 2015) Sentiment Product Reviews (Zhang et al , 2015) Sentiment Business Reviews (Maas et al , 2011) Sentiment Movies Reviews
source task (zhang et al , 2015) topic (zhang et al , 2015) topic (zhang et al , 2015) topic (zhang et al , 2015) sentiment product reviews (zhang et al , 2015) sentiment business reviews (maas et al , 2011) sentiment movies reviews
0.39
Subject News Articles Wikipedia Articles QA Posts
ニュース記事 ウィキペディア 記事 qa ポスト
0.59
on 10 examples per class with a 3× augmentation multiplier.
クラス毎に3×増倍乗算器を持つ10の例について。
0.61
Based on this experiment, we then computed each transform’s performance by averaging the accuracy change relative to a TORIG baseline across all datasets.
Table 5 shows the top ten best performing transforms, six of which employ SIB techniques.
表5は、最高の10の変換であり、そのうち6つはSIB技術を使用している。
0.54
These results expand support for the overall conclusion that sibylvariance represents an especially effective class of transformations for improving generalization performance.
Generalization Performance. Models trained upon SIB-augmented data attained the highest test set accuracy in 89% (16 out of 18) of experimental configurations, with the adaptive mixture mutations being the most consistently effective.
We then report an average accuracy for each D and T P pair.
次に、各DとTPのペアの平均精度を報告する。
0.72
Figure 4 shows how defect detection is enabled by INV and SIB.
図4は、invとsibによる欠陥検出の方法を示しています。
0.60
With the exception of Yahoo! Answers, the models scored nearly perfect accuracy on TORIG; however, when the same models are tested using data generated with INV and SIB, they struggle to generalize.
Test data synthesized with SIB can reveal most defects in these models, indicating the value of sibylvariance in constructing test oracles for ML models in the absence of
Figure 3: The best performing TP for each dataset trained on 200 examples per class.
図3: クラス毎に200の例でトレーニングされた各データセットで最高のパフォーマンスtp。
0.72
αSentMix or αTextMix leads to the highest performance gains.
αSentMixまたはαTextMixは最高のパフォーマンス向上をもたらす。
0.74
SIB* consistently outperforms INV*.
SIB* は INV* を一貫して上回る。
0.60
model’s primary confusions during training added approximately 1% to accuracy relative to mixing classes uniformly at random.
トレーニング中のモデルの主な混乱は、クラスをランダムに混合する際の精度を約1%向上させた。
0.70
This shows another unique benefit of sibylvariance that is not transferable to its INV counterparts.
このことは、そのINVに転移できないシリル分散の別のユニークな利点を示している。
0.54
While our full scale experiments show a clear trend that SIB generally outperforms INV, we primarily evaluated T P s combining multiple transforms instead of assessing the efficacy of each in isolation.
SIB は一般に INV より優れる傾向を示したが,本実験では個別に各変換の有効性を評価するのではなく,複数の変換を組み合わせた TP s を主に評価した。
0.70
Initially, this was a logistical decision due to computational limitations.
当初、これは計算上の制限による論理的な決定であった。
0.64
To investigate each transformation’s effect individually, we conducted a small scale experiment training 756 models ((39 transformations + 3 αSIB) × 6 datasets × 3 runs)
The INV / SIB types were SIB for the sentiment analysis datasets and INV for the topic classification datasets.
INV/SIB型は感情分析データセットのSIB型とトピック分類データセットのINV型であった。
0.80
See Table 11 in the Appendix for more details.
詳細はAppendixのテーブル11を参照してください。
0.88
expensive human labeling and judgements.
高価な人間のラベルと判断
0.74
Tests which lie outside the expected input distribution are not likely to be fair nor actionable.
期待される入力分布の外にあるテストは、公正で、実行可能なものではない。
0.67
Since SIB transforms generally perturb data more aggressively than INV ones, they likewise possess more potential for creating unreasonable, out-of-domain tests of model quality.
However, the positive results in RQ1 may justify the use of SIB transformations as reasonable for testing.
しかし、rq1の正の結果は、sib変換をテストに適していると正当化することができる。
0.58
Had the newly transformed data truly belonged to a different distribution, model performance on the in-domain test set should have decreased as a result of dataset shift (Quiñonero-Candela et al , 2009; Hu et al , 2022).
新たに変換されたデータが本当に異なる分布に属していた場合、データセットシフトの結果、ドメイン内テストセットのモデル性能が低下したはずだ(Quiñonero-Candela et al , 2009; Hu et al , 2022)。
0.75
In fact, we observed the opposite as model performance was consistently improved.
実際、モデルの性能が一貫して改善されるにつれて、その逆が観察された。
0.53
This suggests that SIB transforms yield data that is tenably indomain and therefore may complement INV transforms in exposing defective model behavior.
In contrast, most INV transformations involve minor changes — e g expand contractions — and test the aspect of language already well modeled from extensive pre-training.
Defect Detection. Models tested with SIBtransformed data exhibited the greatest number of defects in 83% (5 out of 6) of experimental configurations.
欠陥検出 SIB変換データを用いたモデルでは,83%(6つ中5つ)に最大の欠陥が認められた。
0.62
4.5 RQ3. Adversarial Robustness For RQ3, we assess whether models trained on INV or SIB are more resilient to adversarial attacks than models trained on an original data.
An adversarial text input is typically obtained via semantic preserving (i.e. invariant) perturbations to legitimate examples in order to deteriorate the model performance.
The changes are typically generated by ascending the gradient of the loss surface with respect to the original example and improving robustness to adversarial attacks is a necessary precondition for real-world NLP deployment.
We select three attack algorithms based on their popularity and effectiveness: (1) TextFooler (Jin et al , 2019), (2) DeepWordBug (Gao et al , 2018), and (3) TextBugger (Li et al , 2018), all as implemented in TextAttack (Morris et al , 2020).
その人気と有効性に基づいて,(1)textfooler (jin et al , 2019), (2) deepwordbug (gao et al , 2018), (3) textbugger (li et al , 2018) の3つの攻撃アルゴリズムをtextattack (morris et al , 2020) で実装した。
0.68
We focus on models trained with 10 examples per class because the largest changes in generalization performance are more likely to exhibit the clearest trend for adversarial robustness.
For each of 11 models and 3 attacks, we randomly sample 100 inputs from the original data and perturb them to create a total of 3,300 adversarial examples.
On average, SIB*-trained models improve robustness by 4%, while INV*-trained models sustain a 1% decrease.
平均すると、SIB*訓練モデルではロバスト性は4%向上し、INV*訓練モデルは1%低下する。
0.73
Topic classification is made more robust via training with augmented data.
トピック分類は、拡張データによるトレーニングによってより堅牢になる。
0.66
Consistently, Tα-SentMix produces the most resilient models.
対照的に、Tα-SentMixは最も回復力のあるモデルを生成する。
0.41
For sentiment analysis, improved generalization performance enabled by SIB does not necessarily lead to improved robustness to existing adversarial attacks.
The underlying sentiment models trained with augmented data improves generalization over TORIG by an average of 5%.
強化データでトレーニングされた根底にある感情モデルは、TORIGに対する一般化を平均5%改善する。
0.67
However, counter-intuitively, the models are not more robust to the three attacks than TORIG and that Pearson correlation is -0.28 between accuracy and adversarial robustness.
This finding motivates future work to investigate why there is a negative correlation and how to design SIB such that accuracy improvement also translates to corresponding adversarial robustness.
Adversarial Robustness. Of all the experimental configurations where adversarial robustness was improved over the notransform baseline, 92% (11 out of 12) of them involved models trained on SIBaugmented data.
The primary purpose of data transformations in ML is to diversify datasets in the neighborhood of existing points, a principle formalized as Vicinal Risk Minimization (VRM) (Chapelle et al , 2001).
MLにおけるデータ変換の主な目的は、既存の点付近のデータセットを多様化することであり、この原則は Vicinal Risk Minimization (VRM) として定式化されている(Chapelle et al , 2001)。
0.72
Synthetic examples can be drawn from a vicinal distribution to find similar but different points that enlarge the original data distribution.
For instance, within image classification, it is common to define the vicinity of an image as the set of its random crops, axal reflections, and other label-preserving INV transforms.
While VRM can expose ML models to more diverse input space and consequently reduce generalization errors, the neighborhoods created by INV are relatively restricted.
A lower attack success rate indicates a higher adversarial robustness.
攻撃成功率の低下は、高い敵の堅牢性を示す。
0.65
sualize the effects of various transformations on SST-2 (Socher et al , 2013).
SST-2(Socher et al , 2013)に対する様々な変換の効果をsalizeする。
0.79
Figure 5 presents the UMAP-reduced (McInnes et al , 2020) [CLS] tokens produced by a BERT transformer for sentiment classification.
図5は、感情分類のためのBERT変換器によって生成されたUMAP(McInnes et al , 2020)[CLS]トークンを提示する。 訳抜け防止モード: 図5は UMAP - reduce ( McInnes et al, 感性分類のためのBERT変換器によって生成される[CLS ]トークン。
0.71
Figure 5a shows that the classes are initially well separated and high performance can be obtained by selecting any separating surface between the two clusters.
However, a more reasonable choice for the best boundary is one that exhibits the largest margin between classes — the very intuition behind Support Vector Machines (Cortes and Vapnik, 1995).
しかし、最良の境界に対するより合理的な選択は、クラス間の最大のマージンを示すものである ― Support Vector Machines (Cortes and Vapnik, 1995) の背後にある直感である。
0.81
Figure 5d suggests that a model trained on mixture mutations is likely to arrive at a boundary with the lowest loss.
For example, in 5d, the augmented examples in green provide additional loss feedback from uncovered portions of the input space to encourage a decision boundary that maximizes the margin between class clusters.
Threats to Validity. External threats to validity include the generalization of our results to model architectures dissimilar to BERT (i.e. bert-base-uncased).
It is possible that larger autoencoder models like RoBERTa (Liu et al , 2019) and auto-regressive models like XLNet (Yang et al , 2019) may respond differently to SIB transformations.
RoBERTa (Liu et al , 2019) のような大規模なオートエンコーダモデルや XLNet (Yang et al , 2019) のような自動回帰モデルは、SIB変換に対して異なる反応を示す可能性がある。
0.71
Secondly, while the framework of sibylvariance is applicable to all data types, we have only provided empirical results supporting their efficacy for text classification models.
This indirectly supports the validity of SIB-transformed data for testing in RQ2, although we acknowledge that additional caution is required for using any aggressively modified, synthetic data as a substitute for real data for the purpose of exposing defective model behavior.
6 Related Work In this section, we broadly cover data transformations within and outside of the text domain because our proposed framework for sibylvariance is applicable to all classification contexts.
Data Augmentation. Effective data augmentation is a key factor enabling superior model performance on a wide range of tasks (Krizhevsky et al , 2012; Jiang et al , 2018; Xie et al , 2019).
データ拡張。 効果的なデータ拡張は、幅広いタスクにおいて優れたモデルパフォーマンスを実現する重要な要因である(Krizhevsky et al , 2012; Jiang et al , 2018; Xie et al , 2019)。
0.79
In many cases, practitioners leverage domain knowledge to reinforce critical invariances in the underlying data.
多くの場合、実践者はドメイン知識を利用して基礎となるデータにおける重要な不変性を補強する。
0.53
In computer vision, for example, translation invariance is the idea that no matter where the objects of interest reside within an image, the model will still classify them correctly.
Image translations and random crops encourage this more generalized conceptualization within the model (Simard et al , 1998) and all other transforms have a similar goal: reinforce a particular invariance that helps the learner perform well on future unseen data.
画像翻訳とランダムな作物は、モデル内のこのより一般化された概念化を促進する(Simard et al , 1998)。 訳抜け防止モード: 画像翻訳とランダムな作物はモデル内のより一般化された概念化を促進する(Simard et al, 1998)。 他の変換にも同じような目標があります 学習者が将来の見当たらないデータに対してうまく振る舞うのに役立つ特定の不変性を補強する。
0.63
Numerous techniques have been proposed to assist with this learning objective and thereby improve generalization.
Random erasing (Zhong et al , 2017; Devries and Taylor, 2017) and noise injection (Wen et al , 2020; Xie et al , 2019) support invariance to occlusions and promote robust features.
ランダム消去(Zhong et al , 2017; Devries and Taylor, 2017)とノイズ注入(Wen et al , 2020; Xie et al , 2019)は、閉塞に対する不変性をサポートし、堅牢な機能を促進する。
0.79
Interpolating (Bowyer et al , 2011) and extrapolating (DeVries and Taylor, 2017) nearest neighbors in the input / feature space reinforces a linear relationship between the newly created data and the supervision signal while reducing class imbalance.
入力/特徴空間における補間(Bowyer et al , 2011)と外挿(DeVries and Taylor, 2017)は、クラス不均衡を低減しつつ、新たに作成されたデータと監視信号の間の線形関係を強化する。
0.86
However, nearly all of these approaches, and many others (Shorten and Khoshgoftaar, 2019;
Feng et al , 2021), are label-preserving and therefore limited in their capacity to induce deeper learning of invariant concepts.
Feng et al , 2021) はラベル保存であり、不変概念の深い学習を促す能力に制限がある。
0.71
Sibylvariant transforms enjoy several desirable aspects of INV transformations while mitigating their drawbacks.
シビン変態変換は、その欠点を緩和しながら、INV変換のいくつかの望ましい側面を享受する。
0.45
Similar to feature space functions (DeVries and Taylor, 2017), mixture mutations do not require significant domain knowledge.
特徴空間関数 (DeVries and Taylor, 2017) と同様に、混合突然変異は重要なドメイン知識を必要としない。
0.73
Like approaches that reduce dataset imbalance (Bowyer et al , 2011), SIB transforms can increase class representation through mixed membership or targeted transmutations that inherit diverse characteristics of the source inputs.
データセットの不均衡を低減するアプローチ(Bowyer et al , 2011)のように、SIB変換は、ソース入力の多様な特性を継承する混合メンバシップやターゲット変換を通じてクラス表現を向上させることができる。
0.68
In all cases, relaxing the labelpreserving constraint enables SIB functions to both complement and enhance the learning of critical invariances by further expanding the support of the dataset in new directions.
Adversarial attacks are a special class of INV transformations that simultaneously minimize perturbations to the input while maximizing the perception of change to a learner.
This task is more difficult within the NLP domain due to the discrete nature of text, but several works (Alzantot et al , 2018; Zhang et al , 2020) have proven successful at inducing model errors.
このタスクはテキストの離散的な性質のためにNLPドメイン内では難しいが、いくつかの研究(Alzantot et al , 2018; Zhang et al , 2020)はモデルエラーの誘発に成功した。
0.82
Real-world use of NLP requires resilience to such attacks and our work complements robust training (Parvez et al , 2018) and robust certification (Ye et al , 2020; Pruksachatkun et al , 2021) to produce more reliable models.
NLPの実際の使用にはこのような攻撃に対するレジリエンスが必要であり、我々の作業はより信頼性の高いモデルを作成するための堅牢なトレーニング(Parvez et al , 2018)と堅牢な認定(Ye et al , 2020; Pruksachatkun et al , 2021)を補完する。
0.70
Emerging Sibylvariant Transforms.
Emerging Sibylvariant Transforms
0.31
Specific transformations designed to alter the expected class of an input have existed prior to this work (Zhang et al , 2017; Yun et al , 2019; Guo, 2020; Zhu et al , 2017), albeit primarily in the image domain and also in a more isolated, ad hoc fashion.
入力の期待クラスを変更するように設計された特定の変換(Zhang et al , 2017; Yun et al , 2019; Guo, 2020; Zhu et al , 2017)は、主に画像領域と、より孤立したアドホックな方法で存在する。 訳抜け防止モード: この研究以前には、入力の期待クラスを変更するように設計された特定の変換(Zhang et al)があった。 2017年、Yun et al, 2019年、Guo, 2020年、Zhu et al, 2017年) 主にイメージドメインと、より孤立したアドホックな方法でもあります。
0.78
Among our primary contributions is to propose a unifying name, framework, and taxonomy for this family of sibylvariant functions.
我々の主な貢献は、このシビル変種関数の族に対する統一的な名前、枠組み、分類の提案である。
0.66
Furthermore, most prior works introduce a single transformation and evaluate its efficacy on training alone.
In contrast, we proposed several novel transformations, a new adaptive training routine, and evaluated the broader impacts of 41 INV and SIB transforms on training, defect detection, and robustness simultaneously.
All of our transformations operate in the input space, which is both more general and more challenging because we have to contend with rules of grammar and style.
7 Conclusion Inspired by metamorphic testing, we proposed the notion of sibylvariance to jointly transform both input and output class (Xi, yi) pairs in a knowable way.
To explore the potential of sibylvariance, we define 18 new text transformations and adapt 23 existing transformations into an open source tool called Sybil.
In particular, we define several types of mixture mutations and design a novel concept-based text transformation technique utilizing salience attribution and neural sentence generation.
Across six benchmarks from two different NLP classification tasks, we systematically assess the effectiveness of INV and SIB for generalization performance, defect detection, and adversarial robustness.
Acknowledgements This work is supported in part by National Science Foundations via grants CCF-2106420, CCF2106404, CNS-2106838, CCF-1764077, CHS1956322, CCF-1723773, ONR grant N00014-18-12037, Intel CAPA grant, Samsung, and a CISCO research contract.
認定 CCF-2106420, CCF2106404, CNS-2106838, CCF-1764077, CHS 1956322, CCF-1723773, ONR grant N00014-18-12037, Intel CAPA grant, Samsung, CISCO Research Contractを通じて,National Science Foundationsが部分的にサポートしている。
0.78
We would also like to thank Atharv Sakhala for early contributions to the Sybil project as well as Jason Teoh, Sidi Lu, Aaron Hatrick, Sean Gildersleeve, Hannah Pierce, and all the anonymous reviewers for their many helpful suggestions.
References Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani B. Srivastava, and Kai-Wei Chang.
参照:Moustafa Alzantot、Yash Sharma、Ahmed Elgohary、Bo-Jhang Ho、Mani B. Srivastava、Kai-Wei Chang。 訳抜け防止モード: Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani B. Srivastava, and Kai - Wei Chang
0.43
2018. Generating natural language adversarial examples.
2018. 自然言語の逆例を生成する。
0.54
CoRR, abs/1804.07998.
CoRR, abs/1804.07998。
0.30
E. Barr, M. Harman, P. McMinn, M. Shahbaz, and Shin Yoo.
e・バー、m・ハーマン、p・マクミン、m・シャーバズ、シン・ヨー。
0.49
2015. The oracle problem in software testing: A survey.
2015. ソフトウェアテストにおけるオラクルの問題: 調査。
0.61
IEEE Transactions on Software Engineering, 41:507–525.
IEEE Transactions on Software Engineering, 41:507–525。
0.38
Kevin W. Bowyer, Nitesh V. Chawla, Lawrence O. Hall, and W. Philip Kegelmeyer.
Kevin W. Bowyer、Nitesh V. Chawla、Lawrence O. Hall、W. Philip Kegelmeyer。
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik.
オリヴィエ・チャペル、ジェイソン・ウェストン、レオン・ボトウ、ウラジーミル・ヴァプニク。
0.60
2001. Vicinal risk minimization.
2001. リスク最小化。
0.51
In Advances in Neural Information Processing Systems, volume 13.
ニューラル・インフォメーション・プロセッシング・システム (neural information processing system) における第13巻。
0.57
MIT Press. Jiaao Chen, Zichao Yang, and Diyi Yang.
mitプレス。 Jiaao Chen、Zichao Yang、Diyi Yang。
0.51
2020a. Mixtext: Linguistically-infor med interpolation of hidden space for semi-supervised text classification.
2020年。 mixtext: 半教師付きテキスト分類のための隠れ空間の言語的不定補間。
0.70
CoRR, abs/2004.12239.
CoRR, abs/2004.12239。
0.59
T. Chen, S. Cheung, and S. Yiu.
T. Chen、S. Cheung、S. Yiu。
0.88
2020b. Metamorphic testing: A new approach for generating next test cases.
2020年。 メタモルフィックテスト: 次のテストケースを生成するための新しいアプローチ。
0.76
ArXiv, abs/2002.12543.
ArXiv, abs/2002.12543
0.34
Corinna Cortes and Vladimir Vapnik.
コリンナ・コルテスとウラジーミル・ヴァプニク
0.64
1995. Supportvector networks.
1995. サポートベクターネットワーク。
0.60
Machine learning, 20(3):273–297.
機械学習、20(3):273-297。
0.79
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le.
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, Quoc V. Le
0.40
2019. Randaugment: Practical data CoRR, augmentation with no separate search.
2019. Randaugment: 実用的なデータCoRR、個別の検索のない拡張。
0.57
abs/1909.13719.
abs/1909.13719。
0.46
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova
0.38
2018. BERT: pre-training of deep bidirectional transformers for language understanding.
2018. BERT: 言語理解のための双方向トランスフォーマーの事前トレーニング。
0.59
CoRR, abs/1810.04805.
CoRR, abs/1810.04805。
0.29
Terrance DeVries and Graham W. Taylor.
テラス・デヴリーズとグラハム・w・テイラー。
0.48
2017. Dataset augmentation in feature space.
2017. 特徴空間におけるデータセット拡張。
0.56
Terrance Devries and Graham W. Taylor.
テラス・デヴリーズとグラハム・w・テイラー。
0.48
2017. Improved regularization of convolutional neural networks with cutout.
2017. 切断による畳み込みニューラルネットワークの正則化の改善
0.61
CoRR, abs/1708.04552.
corr、abs/1708.04552。
0.37
Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy.
Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, Eduard Hovy 訳抜け防止モード: スティーブン・y・フェン、ヴァーン・ガンガル、ジェイソン・ウェイ、サラス・チャンダール soroush vosoughi氏、teruko mitamura氏、eduard hovy氏。
0.52
2021. A survey of data augmentation approaches for nlp.
2021. nlpのデータ拡張アプローチに関する調査
0.49
Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi.
Ji Gao、Jack Lanchantin、Mary Lou Soffa、Yanjun Qi。
0.70
2018. Black-box generation of adversarial text sequences to evade deep learning classifiers.
2018. 深層学習分類器回避のための逆文列のブラックボックス生成
0.53
CoRR, abs/1801.04354.
CoRR, abs/1801.04354。
0.30
Hongyu Guo. 2020.
ホンユ・グオ。 2020.
0.39
Nonlinear mixup: Out-ofmanifold data augmentation for text classification.
非線形混合:テキスト分類のための外部データ拡張。
0.71
In AAAI. Hongyu Guo, Yongyi Mao, and Richong Zhang.
AAAI所属。 ホンユ・グオ、ヨンギ・マオ、リチョン・チャン。
0.47
2019. Augmenting data with mixup for sentence classification: An empirical study.
2019. 文分類のためのミックスアップによるデータ拡張:実証的研究
0.57
CoRR, abs/1905.08941.
CoRR, abs/1905.08941。
0.58
Dan Hendrycks and Kevin Gimpel.
ダン・ヘンドリックスとケビン・ギンペル
0.42
2016. Bridging nonlinearities and stochastic regularizers with gaussian error linear units.
2016. ガウス誤差線形単位を持つ非線形性と確率正規化子
0.58
CoRR, abs/1606.08415.
CoRR, abs/1606.08415。
0.29
Qiang Hu, Yuejun Guo, Maxime Cordy, Xiaofei Xie, Lei Ma, Mike Papadakis, and Yves Le Traon.
Qiang Hu, Yuejun Guo, Maxime Cordy, Xiaofei Xie, Lei Ma, Mike Papadakis, Yves Le Traon 訳抜け防止モード: Qiang Hu, Yuejun Guo, Maxime Cordy, Xiaofei Xie レイ・マ(Lei Ma)、マイク・パパダキス(Mike Papadakis)、イヴ・ル・トラオン(Yves Le Traon)。
0.65
2022. An empirical study on data distribution-aware test selection for deep learning enhancement.
2022. 深層学習向上のためのデータ分布認識テスト選択に関する実証的研究
0.60
Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.
Alex Krizhevsky、Ilya Sutskever、Geoffrey E Hinton。
0.33
2012. Imagenet classification with deep convolutional neural networks.
2012. 深部畳み込みニューラルネットワークを用いた画像ネット分類
0.63
In Advances in Neural Information Processing Systems, volume 25.
ニューラル情報処理システムの進歩, 第25巻
0.53
Curran Associates, Inc.
Curran Associates, Inc.
0.42
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer。
0.39
2019. BART: denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension.
2019. BART: 自然言語の生成、翻訳、理解のためのシーケンス・ツー・シーケンスの事前学習。
0.50
CoRR, abs/1910.13461.
corr、abs/1910.13461。
0.41
Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Textbugger: Generating adversarCoRR,
Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, Ting Textbugger: Generating adversarCoRR,
0.39
text against real-world applications.
現実世界のアプリケーションに対するテキスト。
0.61
Wang. 2018.
Wang 2018.
0.34
ial abs/1812.05271.
ial abs/1812.05271。
0.26
Bill Yuchen Lin, Ming Shen, Yu Xing, Pei Zhou, and Xiang Ren.
ビル・ユーチェン・リン、ミン・シェン、ユー・シン、ピー・周、チャン・レン。
0.40
2019. Commongen: A constrained text generation dataset towards generative commonsense reasoning.
2019. Commongen: 生成コモンセンス推論のための制約付きテキスト生成データセット。
0.57
CoRR, abs/1911.03705.
CoRR, abs/1911.03705。
0.30
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov。 訳抜け防止モード: yinhan liu, myle ott, naman goyal, jingfei du, マンダー・ジョシ、ダンチー・チェン、オマー・レヴィ、マイク・ルイス ルーク・ゼトルモイヤー(luke zettlemoyer)とヴェセリン・ストヤノフ(veslin stoyanov)。
0.61
2019. Roberta: A robustly optimized BERT pretraining approach.
2019. Roberta: 堅牢に最適化されたBERT事前トレーニングアプローチです。
0.51
CoRR, abs/1907.11692.
CoRR、abs/1907.11692。
0.54
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts.
Andrew L. Maas、Raymond E. Daly、Peter T. Pham、Dan Huang、Andrew Y. Ng、Christopher Potts。
0.86
2011. Learning word vectors for sentiment analysis.
2011. 感情分析のための単語ベクトルの学習
0.60
In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA.
第49回計算言語学会年次総会の議事録:人間言語技術142-150ページ、オレゴン州ポートランド。
0.56
Association for Computational Linguistics.
計算言語学会会員。
0.52
英語（論文から抽出）
日本語訳
スコア
Leland McInnes, John Healy, and James Melville.
リーランド・マキンズ、ジョン・ヒーリー、ジェームズ・メルヴィル。
0.66
2020. Umap: Uniform manifold approximation and projection for dimension reduction.
2020. Umap: 次元減少のための一様多様体近似と射影。
0.56
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman.
アレックス・ウォーシュタット、アマンプリート・シン、サミュエル・r・ボウマン。
0.41
2018. Neural network acceptability judgments.
2018. ニューラルネットワークの受容性判断。
0.54
arXiv preprint arXiv:1805.12471.
arXiv preprint arXiv:1805.12471
0.36
John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi.
John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, Yanjun Qi 訳抜け防止モード: John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby ディ・ジンとヤンジュン・カイ。
0.81
2020. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp.
the Trade, This Book is an Outgrowth of a 1996 NIPS Workshop, page 239–27, Berlin, Heidelberg.
The Trade, This Book is a Outgrowth of a 1996 NIPS Workshop, page 239–27, Berlin, Heidelberg. 訳抜け防止モード: The Trade, This Book is a Outgrowth of a 1996 NIPS Workshop, 239-27頁、ベルリン、ハイデルベルク。
0.83
Springer-Verlag.
Springer-Verlag
0.41
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts.
Richard Socher、Alex Perelygin、Jean Wu、Jason Chuang、Christopher D. Manning、Andrew Ng、Christopher Potts。
0.76
2013. Recursive deep models for semantic compositionality over a sentiment treeIn Proceedings of the 2013 Conference on bank.
2013. 感情木上の意味的構成性の再帰的深層モデル 銀行に関する2013年会議の成果
0.58
Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washington, USA.
自然言語処理における経験的手法 1631-1642頁、ワシントン州シアトル。
0.68
Association for Computational Linguistics. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.
2020. Time series data augmentation for deep learning: A survey.
2020. ディープラーニングのための時系列データ拡張: 調査
0.54
CoRR, abs/2002.12478.
corr、abs/2002.12478。
0.40
Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le.
Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh-Thang Luong, Quoc V. Le
0.44
2019. Unsupervised data augmentation.
2019. 教師なしのデータ拡張。
0.45
CoRR, abs/1904.12848.
corr、abs/1904.12848。
0.43
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, Quoc V. Le 訳抜け防止モード: Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell Ruslan Salakhutdinov と Quoc V. Le 。
0.82
2019. Xlnet: Generalized autoregressive pretraining for language understanding.
2019. Xlnet: 言語理解のための汎用的な自己回帰事前トレーニング。
0.50
CoRR, abs/1906.08237.
corr、abs/1906.08237。
0.41
Mao Ye, Chengyue Gong, and Qiang Liu.
毛沢東、チェンジュ・ゴン、チアン・リウ。
0.38
2020. SAFER: A structure-free approach for certified robustness to adversarial word substitutions.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros.
Jun-Yan Zhu、Taesung Park、Phillip Isola、Alexei A. Efros。
0.43
2017. Unpaired image-to-image translation using cycle-consistent adversarial networks.
2017. 周期整合対向ネットワークを用いた不対向画像変換
0.55
CoRR, abs/1703.10593.
CoRR, abs/1703.10593。
0.29
英語（論文から抽出）
日本語訳
スコア
A Implemented Sybil Transformations
実装されたsybil変換
0.60
Category Mixture Mixture Mixture Generative Generative Word Swap Word Swap Word Swap Word Swap Word Swap Word Swap Word Swap Word Swap Negation Negation Punctuation Punctuation Text Insertion Text Insertion Text Insertion Text Insertion Typos Typos Typos Typos Typos Typos Typos Typos Typos Typos Typos Typos Emojis Emojis Emojis Emojis Emojis Emojis Emojis Emojis
Table 7: Transform descriptions currently implemented in Sybil, sampled from according to task (sentiment analysis or topic) and T P (INV, SIB, or INVSIB).
B Other Possible Text Transformations Transformation replace synonym (embedding) word swap (masked) change gendered pronoun change protected class change "for" to 4 change "to" to 2 swap phrase with acronym negation of negative clause negation of neutral clause negation of positive clause backtranslation add exclamation add period add question mark remove exclamation remove period remove question mark add random URL (404) add neutral phrase
Category Word Swap Word Swap Word Swap Word Swap Word Swap Word Swap Word Swap Negation Negation Negation Paraphrase Punctuation Punctuation Punctuation Punctuation Punctuation Punctuation Text Insertion Text Insertion Tense / Voice make continuous future tense Tense / Voice make continuous past tense Tense / Voice make continuous present tense Tense / Voice make perfect continuous future tense Tense / Voice make perfect continuous past tense Tense / Voice make perfect continuous present tense Tense / Voice make perfect future tense Tense / Voice make perfect past tense Tense / Voice make perfect present tense Tense / Voice make simple future tense Tense / Voice make simple past tense Tense / Voice make simple present tense Tense / Voice Tense / Voice Emojis Emojis Emojis Emojis
Category Word Swap Word Swap Word Swap Word Swap Word Swap Word Swap Word Swap Negation Negation Negation Paraphrase Punctuation Punctuation Punctuation Punctuation Punctuation Punctuation Text Insertion Text Insertion Tense / Voice make continuous future tense Tense / Voice make continuous past tense Tense / Voice make continuous present tense Tense / Voice make perfect continuous future tense Tense / Voice make perfect continuous past tense Tense / Voice make perfect continuous present tense Tense / Voice make perfect future tense Tense / Voice make perfect past tense Tense / Voice make perfect present tense Tense / Voice make simple future tense Tense / Voice make simple past tense Tense / Voice make simple present tense Tense / Voice Tense / Voice Emojis Emojis Emojis Emojis
0.42
change voice active change voice passive replace emoji with word antonym replace emoji with word synonym replace word with emoji antonym replace word with emoji synonym
= love visually is is essentially rumination on it stunning this a [1, 0] + [0, 1] → [0.33, 0.67]
= love visual is is is is essentially rumination on it amazing this a [1, 0] + [0, 1] → [0.33, 0.67] 訳抜け防止モード: = love visual isは、基本的にこのa[1]をルミネーションするものだ。 0 ] + [ 0, 1 ] → [ 0.33,0.67 ]
0.60
empty Table 9: Examples of SIB transformations for the image and text domains.
空 表9: 画像とテキストドメインに対するSIB変換の例。
0.52
For mixture mutations, we show a soft label proportional to the pixel and word counts of their constituent parts.