{"title": "Understanding the Downstream Instability of Word Embeddings", "book": "Proceedings of Machine Learning and Systems", "page_first": 262, "page_last": 290, "abstract": "Many industrial machine learning (ML) systems require frequent retraining to keep up-to-date with constantly changing data. This retraining exacerbates a large challenge facing ML systems today: model training is unstable, i.e., small changes in training data can cause significant changes in the model's predictions. In this paper, we work on developing a deeper understanding of this instability, with a focus on how a core building block of modern natural language processing (NLP) pipelines---pre-trained word embeddings---affects the instability of downstream NLP models. We first empirically reveal a tradeoff between stability and memory: increasing the embedding memory 2x can reduce the disagreement in predictions due to small changes in training data by 5% to 39% (relative). To theoretically explain this tradeoff, we introduce a new measure of embedding instability---the eigenspace instability measure. We relate the eigenspace instability measure to downstream instability by proving a bound on the disagreement in downstream predictions introduced by the change in word embeddings. Practically, we show that the eigenspace instability measure can be a cost-effective way to choose embedding parameters to minimize instability without training downstream models, achieving up to 3.71x lower error rates than existing embedding distance measures. Finally, we demonstrate that the observed stability-memory tradeoffs extend to other types of embeddings as well, including knowledge graph and contextual word embeddings.", "full_text": " UNDERSTANDING THE DOWNSTREAM INSTABILITY OF WORD EMBEDDINGS\r\n 1 1 2 1 2 \u00b4 1\r\n MeganLeszczynski AvnerMay JianZhang SenWu ChristopherR.Aberger ChristopherRe\r\n ABSTRACT\r\n Manyindustrial machine learning (ML) systems require frequent retraining to keep up-to-date with constantly\r\n changing data. This retraining exacerbates a large challenge facing ML systems today: model training is unstable,\r\n i.e., small changes in training data can cause signi\ufb01cant changes in the model\u2019s predictions. In this paper, we\r\n work on developing a deeper understanding of this instability, with a focus on how a core building block of\r\n modernnatural language processing (NLP) pipelines\u2014pre-trained word embeddings\u2014affects the instability of\r\n downstream NLP models. We \ufb01rst empirically reveal a tradeoff between stability and memory: increasing the\r\n embedding memory2\u00d7canreducethedisagreementinpredictions due to small changes in training data by 5% to\r\n 37%(relative). To theoretically explain this tradeoff, we introduce a new measure of embedding instability\u2014the\r\n eigenspace instability measure\u2014which we prove bounds the disagreement in downstream predictions introduced\r\n by the change in word embeddings. Practically, we show that the eigenspace instability measure can be a\r\n cost-effective way to choose embedding parameters to minimize instability without training downstream models,\r\n outperforming other embedding distance measures and performing competitively with a nearest neighbor-based\r\n measure. Finally, we demonstrate that the observed stability-memory tradeoffs extend to other types of embeddings\r\n as well, including knowledge graph and contextual word embeddings.\r\n 1 INTRODUCTION In this work, we take a \ufb01rst step toward addressing the\r\n Data is more dynamic than ever before: every input, inter- problem of ML model instability by examining in detail a\r\n action, and response is captured and archived in hopes of core building block of most modern natural language pro-\r\n extracting insights with machine learning (ML) models. To cessing (NLP) applications: word embeddings (Mikolov\r\n stay up-to-date, models must be frequently retrained, with et al., 2013a;b; Pennington et al., 2014; Bojanowski et al.,\r\n the freshness of models becoming a requirement for user sat- 2017). Several recent works have shown that word em-\r\n isfaction in numerous products, from ads (He et al., 2014) to beddings are unstable, with the nearest neighbors to words\r\n recommendationsystems(Covingtonetal.,2016). However, varying signi\ufb01cantly across embeddings trained under dif-\r\n frequent retraining can lead to large and unwanted \ufb02uctu- ferent settings (Hellrich & Hahn, 2016; Antoniak & Mimno,\r\n ations in model predictions due to the instability of many 2018; Wendlandt et al., 2018; Pierrejean & Tanguy, 2018;\r\n machine learning training algorithms: minimal changes Chugh et al., 2018; Hellrich et al., 2019). These results\r\n in training data can produce signi\ufb01cantly different predic- may cause researchers using embeddings for analysis to\r\n tions (Fard et al., 2016). From discussions with engineers reassess the reliability of their conclusions. Moreover, these\r\n in an e-commerce \ufb01rm, an online social media company, results raise questions about how the embedding instability\r\n and a Fortune 500 software company, we found that in- impacts downstream NLP tasks\u2014an area which remains\r\n stability from retraining is one of their largest, and also largely unexplored and which we focus on in this work. We\r\n most under-addressed, pain points. As a result of instability, de\ufb01ne the downstream instability between a pair of word\r\n MLengineers struggle to identify genuine concept shifts, embeddings as the percentage of predictions which change\r\n spend more time tracking down regressions, and require between the models trained on the two embeddings for a\r\n moreresources retraining downstream model dependencies. given task. By this notion of instability, we \ufb01nd that 15% of\r\n Diagnosing and reducing instability in a cost-effective way predictions on a sentiment analysis task can disagree due to\r\n is a major challenge for today\u2019s machine learning pipelines. training the embeddings on an accumulated dataset with just\r\n 1%moredata. In embedding servers, where an embedding\r\n 1Department of Computer Science, Stanford University is reused among multiple downstream tasks (Hermann &\r\n 2SambaNova Systems. Correspondence to: Megan Leszczynski Balso, 2017; Gordon, 2018; Shiebler et al., 2018; Sell &\r\n . Pienaar, 2019), the impact of this instability can be quickly\r\n Proceedings of the 3rd MLSys Conference, Austin, TX, USA, ampli\ufb01ed. Understanding this downstream instability is\r\n 2020. Copyright 2020 by the author(s). challenging, however, because it requires both theoretical\r\n Understanding the Downstream Instability of Word Embeddings\r\n and empirical insights on how the embedding instability strongly correlates with downstream instability than the ma-\r\n propagates to the downstream tasks. jority of the other embedding distance measures (i.e. seman-\r\n Thegoalofthis paper is to develop a deeper understanding tic displacement (Hamilton et al., 2016), the PIP loss (Yin &\r\n of the downstream instability of word embeddings. This Shen, 2018), and the eigenspace overlap score (May et al.,\r\n understanding could both drive the design choices for em- 2019)) and attains Spearman correlations from 0.04 better\r\n bedding systems (i.e. choosing hyperparameters) and lead to 0.09 worse than the other top-performing measure, the\r\n to ef\ufb01cient techniques to distinguish among unstable and k-NN measure (e.g., Hellrich & Hahn (2016); Antoniak\r\n stable embeddings without training downstream models. To &Mimno(2018); Wendlandt et al. (2018)), which lacks\r\n achieve this, we perform a study on the downstream in- theoretical guarantees. Next, we show that when using\r\n stability of word embeddings across multiple embedding an embedding distance measure to choose the more stable\r\n algorithms and downstream tasks. Our study exposes a dimension-precision parameters out of a pair of choices, the\r\n novel trade-off between stability and another critical prop- eigenspace instability measure achieves up to 3.33\u00d7 lower\r\n erty of embeddings\u2014memory. We \ufb01nd that increasing the error rates than the weaker measures and from 0.95\u00d7 to\r\n memory can lead to more stable embeddings, with a 2\u00d7 1.55\u00d7 the error rate of the k-NN measure. On the more\r\n increase in memory reducing the percentage prediction dis- challenging task of selecting the combination of dimen-\r\n agreement on downstream tasks by 5% to 37% (relative). sion and precision under a memory budget, we show that\r\n Determining how the memory affects the instability is not eigenspace instability measure attains a difference in predic-\r\n straightforward: factors like the dimension, a hyperparam- tion disagreement to the oracle up to 2.98% (absolute) better\r\n eter controlling the expressiveness of the embedding, and than the weaker baselines and within 0.35% (absolute) of\r\n the precision, the number of bits used per entry in the em- the k-NN measure.\r\n bedding after compression, can independently affect the Tosummarize, we make the following contributions:\r\n instability and interact in unexpected ways. To better under- \u2022 Westudythedownstreaminstability of word embed-\r\n stand the stability-memory tradeoff empirically, we study dings, revealing a novel stability-memory tradeoff. In\r\n the effects of dimension and precision both in isolation and particular, we study the impact of two key parameters,\r\n together. This important stability-memory tradeoff leads dimension and precision, and propose a simple rule\r\n us to ask two key questions: (1) theoretically, how can we of thumb relating the embedding memory and down-\r\n explain this tradeoff, and (2) practically, how can we se- stream instability (Section 3).\r\n 1\r\n lect the dimension-precision parameters to minimize the \u2022 To theoretically explain this tradeoff, we introduce a\r\n downstream instability? newmeasureforembeddinginstability, the eigenspace\r\n Totheoretically explain the stability-memory trade-off, we instability measure, that we prove theoretically deter-\r\n introduce a new measure for embedding instability\u2014the mines the expected downstream disagreement on a\r\n eigenspace instability measure\u2014which we theoretically re- linear regression task (Section 4).\r\n late to downstream instability in the case of linear regression \u2022 Toempirically validate our theory, we perform an eval-\r\n models. The eigenspace instability measure builds on the uation of methods for selecting embedding hyperpa-\r\n eigenspace overlap score (May et al., 2019), and measures rameters to minimize downstream instability. Practi-\r\n the degree of similarity between the eigenvectors of the cally, we show that the eigenspace instability measure\r\n Grammatrices of a pair of embeddings, weighted by their can outperform the majority of other embedding dis-\r\n eigenvalues. We show that the expected downstream dis- tance measures and perform similarly to the k-NN\r\n agreement between the linear regression models trained on measure, for which we have no theoretical guarantees.\r\n two embedding matrices can be expressed in terms of the \u2022 Finally, we show that the stability-memory tradeoffs\r\n eigenspace instability measure. Furthermore, these theo- extend to knowledge graph embeddings (Bordes et al.,\r\n retical insights have a practical application: we propose 2013) and contextual word embeddings, such as BERT\r\n using the eigenspace instability measure to ef\ufb01ciently se- embeddings(Devlinetal., 2019). For instance, we \ufb01nd\r\n lect dimension-precision parameters with low downstream that increasing the memory of knowledgegraphembed-\r\n instability, without having to train downstream models. dings 2\u00d7 decreases the instability on a link prediction\r\n Weempirically validate that the eigenspace instability mea- task by 7% to 19% (relative) (Section 6).\r\n sure correlates strongly with the downstream instability and\r\n that the measure is effective as a selection criterion for the 2 PRELIMINARIES\r\n dimension-precision parameters. First, we show that the\r\n theoretically grounded eigenspace instability measure more Webeginbyformallyde\ufb01ningthenotionofinstability we\r\n 1 use in this work. We then review the word embedding\r\n For brevity, we refer to a pair of dimension and precision algorithms and compression technique used in our study,\r\n parameters as the \u201cdimension-precision\u201d parameters. and discuss existing measures to compare two embeddings.\r\n Understanding the Downstream Instability of Word Embeddings\r\n 2.1 Instability De\ufb01nition embeddingmatrixX istrainedviaSGD,wherethelossmax-\r\n Wede\ufb01nethedownstreaminstability as follows: imizes the probability that an observed word and context\r\n pair co-occurs in the corpus and minimizes the probability\r\n n\u00d7d \u02dc n\u00d7k\r\n De\ufb01nition1. LetX \u2208 R andX \u2208R betwoembed- that a negative sample co-occurs. We use the word2vec\r\n ding matrices, and let fX and f \u02dc represent models trained implementation of CBOW.2\r\n X\r\n \u02dc\r\n using X and X, respectively, for a downstream task T. Then 2.3 CompressionTechnique\r\n \u02dc\r\n the instability between X and X with respect to task T is\r\n de\ufb01ned as We use a standard technique\u2014uniform quantization\u2014to\r\n N\r\n \u02dc 1 X compress word embeddings. Recent work (May et al.,\r\n DI (X,X)= L(f (z ),f\u02dc(z )),\r\n T N X i X i 2019) demonstrates that uniform quantization performs on\r\n i=1 par in terms of downstream quality with more complex\r\n N\r\n where {z } is a heldout set for task T, and L is a \ufb01xed compression techniques, such as k-means compression (An-\r\n i i=1\r\n loss function. drews, 2016) and deep compositional code learning (Shu\r\n 3\r\n Whenthezero-one loss is used for L, this measure captures &Nakayama,2018). Weleveragetheir implementation to\r\n the percentage of predictions which disagree on downstream applyuniformquantizationtowordembeddingstostudythe\r\n models trained on each embedding. impact of the precision on instability. Under uniform quanti-\r\n zation, each entry in the word embedding matrix is rounded\r\n b\r\n to a discrete value in a set of 2 equally spaced values within\r\n 2.2 WordEmbeddingAlgorithms an interval, such that each entry can be represented with\r\n Word embedding algorithms learn distributed representa- just b bits. For more details on the way we use uniform\r\n tions of words by taking as input a textual corpus C and quantization for our experiments, see Appendix C.2.\r\n returning the word embedding X \u2208 Rn\u00d7d, where d is the 2.4 EmbeddingDistanceMeasures\r\n dimension of the embeddings and n is the vocabulary size.\r\n We evaluate matrix completion (MC) (Jin et al., 2016), Weconsider four embedding distance measures from the\r\n GloVe (Pennington et al., 2014), and continuous bag-of- literature to quantify the differences between embeddings.\r\n words (CBOW) (Mikolov et al., 2013a;b) embedding al- For each measure, we assume we have a pair of embed-\r\n gorithms. MC and GloVe factor the co-occurrence matrix n\u00d7d \u02dc n\u00d7d\r\n A\u2208Rn\u00d7ngeneratedfromC,whereasCBOWoperateson dings X \u2208 R and X \u2208 R trained on corpora C\r\n \u02dc\r\n the sequential corpus C directly. We elaborate below. and C, respectively, where n is the size of the vocabulary\r\n and d is the dimension of the embedding. Due to computa-\r\n Matrix completion (MC) Matrix completion uses the tional ef\ufb01ciency and our observation that downstream tasks\r\n word embeddings to approximate the observed word co- use a majority of high frequency words, we only consider\r\n occurrence A and can be formally written as: the top 10k most frequent words to compute each measure\r\n (including the eigenspace instability measure).\r\n V =argmin X (XXT \u2212A )2\r\n X i j ij k-NearestNeighbors(k-NN)Measure Variantsofthek-\r\n (i,j)\u2208\u0398 NNmeasurewereusedinrecentworksonwordembedding\r\n where\u0398aretheobserved(non-zero)entriesinA. Following stability to characterize the intrinsic stability of embeddings\r\n standard technique, A is the positive pointwise mutual infor- (e.g., Hellrich & Hahn (2016); Antoniak & Mimno (2018);\r\n mation (PPMI) matrix, rather than the true co-occurrence Wendlandt et al. (2018)). The k-NN measure is de\ufb01ned\r\n P \u02dc\r\n 1 Q |N (X;q)\u2229N (X;q)|\r\n matrix (Bullinaria & Levy, 2007). as k k , where Q is the number of\r\n Q q=0 k\r\n Wesolve the matrix completion problem using an online randomly sampled query words (we use Q=1000), and the\r\n N function takes an embedding and the index of a query\r\n algorithm similar to that proposed in Jin et al. (2016). We k\r\n iteratively train X via stochastic gradient descent (SGD) word, and returns the indices of the k most similar words to\r\n after computing the loss on sampled entries of the observed the query word by the cosine distance.\r\n co-occurrence matrix A. Semantic Displacement Researchers have used seman-\r\n GloVe SimilartoMC,GloVesolvesamatrixfactorization tic displacement to compute the distance that words have\r\n shifted over time (Hamilton et al., 2016). Semantic displace-\r\n P\r\n problem, but approximates the co-occurrence information in 1 n \u02dc\r\n ment can be de\ufb01ned as cos-dist(X ,RX ), where\r\n a weighted form to reduce noise from rare co-occurrences. n i=0 i i\r\n \u02dc T\r\n R=argmin ||X\u2212X\u2126|| ,subjectto\u2126 \u2126=I (i.e.,the\r\n GloVemodelsthewordandcontextembeddingsseparately. \u2126 F\r\n \u00a8\r\n orthogonal Procrustes solution (Schonemann, 1966)).\r\n Continuous bag-of-words (CBOW) The CBOW algo- 2https://github.com/tmikolov/word2vec\r\n rithm predicts a word given its local context words. The 3https://github.com/HazyResearch/smallfry\r\n Understanding the Downstream Instability of Word Embeddings\r\n Pairwise Inner Product Loss The Pairwise Inner Prod- SST-2 CoNLL-2003\r\n uct (PIP) loss was proposed for dimensionality selection to 20 CBOW 8 CBOW\r\n GloVe GloVe\r\n optimize for the intrinsic quality of an embedding (Yin & 15 MC 6 MC\r\n T \u02dc \u02dcT\r\n Shen, 2018). The PIP loss is de\ufb01ned as kXX \u2212XX k .\r\n F 10\r\n EigenspaceOverlapScore Theeigenspaceoverlapscore % Disagreement5 % Disagreement4\r\n was recently proposed as a measure of compression qual- 25 50 Dimension100200400800 25 50 Dimension100200400800\r\n ity (May et al., 2019). The eigenspace overlap is de\ufb01ned as\r\n 1 T \u02dc 2 T \u02dc \u02dc \u02dc\u02dcT SST-2 CoNLL-2003\r\n kU Uk ,whereX = USV andX = USV arethe\r\n d F\r\n \u02dc CBOW 8 CBOW\r\n singular value decompositions (SVDs) of X and X. 15 GloVe GloVe\r\n MC 7 MC\r\n 3 ASTABILITY-MEMORYTRADEOFF 10 6\r\n % Disagreement % Disagreement5\r\n Wenowpresenttheempiricalstudythatexposesthetradeoff 4\r\n weobservebetweendownstreamstability and embedding 5 1 2 4 8 16 32 1 2 4 8 16 32\r\n memory, and demonstrate that as the memory increases, the Precision Precision\r\n instability decreases. We consider the dimension and pre-\r\n cision of the embedding as two important axes controlling Figure 1. Downstream instability of sentiment analysis (SST-2)\r\n the memory of the embedding. We \ufb01rst study the impact of and NER (CoNLL-2003) tasks under different dimensions (top)\r\n the embedding\u2019s dimension and precision on downstream and precisions (bottom) for CBOW, GloVe, and MC embeddings.\r\n instability in isolation in Sections 3.1 and 3.2, respectively, Appendix C.3 for more experimental setup details.\r\n followed by a discussion of their joint effect in Section 3.3.\r\n Sentiment Analysis. We evaluate on a binary sentiment\r\n Corpora WeusetwofullWikipediadumps4: Wiki\u201917and analysis task where given a sentence, the model determines\r\n Wiki\u201918, which we collected approximately a year apart, to if the sentence is positive or negative (Kim, 2014). We\r\n train embeddings. The corpora are pre-processed by a Face- train a linear bag-of-words model for this task and evalu-\r\n book script5, which we modify to keep the letter cases. We ate on four benchmark datasets: MR (Pang & Lee, 2005),\r\n use these two corpora as examples of the temporal changes MPQA(Wiebeetal.,2005),Subj(Pang&Lee,2004),and\r\n which can occur to the text corpora used to train word em- SST-2 (Socher et al., 2013b). We will be showing results on\r\n beddings. Each corpora has about 4.5 billion tokens, and SST-2; for more results, see Appendix D.1.\r\n when training the embeddings, we only learn the embed- NamedEntityRecognition (NER). The named entity recog-\r\n dings for the top 400k most frequent words. nition task is a multi-class classi\ufb01cation task to predict\r\n DownstreamNLPTasks Aftertrainingthewordembed- whether each token in the dataset is an entity, and if so,\r\n dings, we compress the embeddings with uniform quantiza- what type. We use a BiLSTM model (Akbik et al., 2018)\r\n tion and train models for downstream NLP tasks on top of for this task and evaluate on the benchmark CoNLL-2003\r\n the embeddings, \ufb01xing the embeddings during training. We dataset (Tjong Kim Sang & De Meulder, 2003). Each token\r\n train word embeddings with three seeds, and use the same is assigned an entity label of PER, ORG, LOC, and MISC,\r\n corresponding seeds for the downstream models. Results or an O label, indicating outside of any entities (i.e., no en-\r\n are reported as averages over the three seeds, with error bars tity). We measure instability only over the tokens for which\r\n indicating the standard deviation. We also align all pairs the true value is an entity. We use the BiLSTM without the\r\n of Wiki\u201917 and Wiki\u201918 embeddings (same dimension and conditional random \ufb01eld (CRF) decoding layer for compu-\r\n \u00a8 tational ef\ufb01ciency; in Appendix E.2 we show that the trends\r\n seed) with orthogonal Procrustes (Schonemann, 1966) prior also hold on a subset of the results with a BiLSTM-CRF.\r\n to compressing and training downstream models, as prelim-\r\n inary experiments found this helped to decrease instability. 3.1 Effect of Dimension\r\n For each downstream task, we perform a hyperparameter\r\n search for the learning rate using 400-dimensional Wiki\u201917 Weevaluate the impact of the dimension of the embedding\r\n embeddings, and use the same learning rate across all di- onits downstream stability, and show that generally as the\r\n mensions to minimize the impact of learning rate on our dimension increases, the instability decreases.\r\n analysis. Here, we discuss the two standard downstream\r\n NLPtasks we consider throughout our paper. Please see Tradeoffs Toperformourtradeoffstudy,wetrainWiki\u201917\r\n 4https://dumps.wikimedia.org and Wiki\u201918 embeddings with dimensions in {25, 50, 100,\r\n 5https://github.com/facebookresearch/ 200, 400, 800}, and train downstream models on top of the\r\n fastText/blob/master/get-wikimedia.sh embeddings. We compute the prediction disagreement be-\r\n Understanding the Downstream Instability of Word Embeddings\r\n CBOW GloVe MC\r\n 10 12 12 b=1\r\n b=2\r\n 8 10 10 b=4\r\n 8 b=8\r\n 8 b=16\r\n 6 6 6 b=32\r\n % Disagreement4 % Disagreement4 % Disagreement4\r\n 102 103 104 10 2 103 104 1 02 1 03 10 4\r\n Memory (Bits / Word) Memory (Bits / Word) Memory (Bits / Word)\r\n Figure 2. Downstream instability of NER (CoNLL-2003) tasks for various memory budgets with different dimension-precision combina-\r\n tions. The red line indicates the average linear-log model relating instability and memory.\r\n tween models trained on Wiki\u201917 and Wiki\u201918 embeddings memory budgets. Across the memory budgets, embed-\r\n of the same dimension. In Figure 1 (top), we see that as the ding algorithms, and tasks, we see that as we increase\r\n dimension increases, the downstream instability often de- the memory, the downstream instability decreases (Fig-\r\n creases across embedding algorithms and downstream tasks, ure 2). To propose a simple rule of thumb for the stability-\r\n plateauing at larger dimensions. In Section 3.3, we see that memory tradeoff, we \ufb01t a single linear-log model to the\r\n these trends are even more pronounced in lower memory dimension-precision pairs for all memory budgets less than\r\n regimes when we also consider different precisions. 103 bits/word (after which the instability plateaus) across\r\n \ufb01vedownstreamtasks(i.e., the four sentiment analysis tasks\r\n 3.2 Effect of Precision and one NER task) and two embedding algorithms. We \ufb01nd\r\n Weevaluate the effect of the precision, the number of bits the following average stability-memory relationship for the\r\n downstream instability DI for a task T with respect to the\r\n used to store each entry of the embedding matrix, on the T\r\n memory, or bits/word, M : DI \u2248C \u22121.3\u2217log (M),\r\n downstream stability, and show that as the precision in- T T 2\r\n where C is a task-speci\ufb01c constant. For instance, if we\r\n creases, the instability decreases. T\r\n increase the memory 2\u00d7, then the instability decreases on\r\n Tradeoffs We compress 100-dimensional Wiki\u201917 and average by 1.3% (absolute). Across the tasks, embedding\r\n Wiki\u201918 embeddings with uniform quantization to preci- algorithms, and memory budgets we consider, this 1.3% (ab-\r\n sions b \u2208 {1,2,4,8,16,32},6 and train downstream models solute) difference corresponds to an approximately 5% to\r\n on top of the compressed embeddings. We compute the pre- 37%relativereductionindownstreaminstability, depending\r\n diction disagreement between models trained on Wiki\u201917 onthe original instability value (3.6% to 25.9%).\r\n and Wiki\u201918 embeddings of the same precision. In Figure 1 Tounderstandtherelativeimpactoninstabilityofincreasing\r\n (bottom), we show that as the precision increases, the insta- the dimension vs. the precision, we \ufb01t independent linear-\r\n bility generally decreases on sentiment analysis and NER log models to each parameter. We \ufb01nd that precision has\r\n tasks for CBOW, GloVe, and MC embedding algorithms. a larger impact on instability than dimension, with a 2\u00d7\r\n Moreover, we see that for precisions greater than 4 bits, the increase in precision decreasing instability by 1.4% (abso-\r\n impact of compression on instability is minimal. lute) vs. a 2\u00d7 increase in dimension decreasing instability\r\n 3.3 Joint Effect of Dimension and Precision by1.2%(absolute). Please see Appendix C.4 for more de-\r\n tails on how we \ufb01t these trends. In Appendix E, we further\r\n Westudytheeffectofdimensionandprecisiontogether,and demonstrate the robustness of the stability-memory tradeoff\r\n showthat overall, as the memory increases, the downstream (e.g., to more complex downstream models, other sources of\r\n instability decreases. We also propose a simple rule of downstream randomness).\r\n thumbrelating the memory and instability, and evaluate the This stability-memory tradeoff raises two key questions:\r\n relative impact of dimension and precision on the instability. (1) how can we theoretically explain this tradeoff between\r\n Finally, we discuss two key questions based on our empirical the embedding memory and the downstream stability, and\r\n observations, which motivate the rest of the work. (2) how can we jointly select the embedding\u2019s dimension-\r\n Tradeoffs We uniformly quantize the Wiki\u201917 and precision parameters to minimize the downstream instabil-\r\n Wiki\u201918 embeddings of dimensions {25, 50, 100, 200, 400, ity? Practically, choosing these parameters is important,\r\n 800} to precisions {1, 2, 4, 8, 16, 32} to generate many because downstream instability can vary over 3% across\r\n dimension-precision pairs spanning over a wide range of the different combinations of dimension and precision for a\r\n given memory budget (Figure 2). The goal of the remainder\r\n 6b = 32 signi\ufb01es full-precision embeddings. of the paper will be to shed light on these questions.\r\n Understanding the Downstream Instability of Word Embeddings\r\n \u02dc 8\r\n 4 ANALYZING EMBEDDING INSTABILITY difference between the linear models fy and fy trained on\r\n \u02dc\r\n To address both questions raised above, we present a new label vector y using embeddings X and X satis\ufb01es\r\n hP i\r\n measure of embedding instability, the eigenspace instability n \u02dc 2\r\n E (f (x ) \u2212f (x\u02dc ))\r\n measure, which we show is both theoretically and empir- y i=1 y i y i \u02dc\r\n 2 =EI\u03a3(X,X). (1)\r\n E [kyk ]\r\n ically related to the downstream instability of the embed- y\r\n dings. The goal of this measure is to ef\ufb01ciently estimate, Theaboveresult exactly characterizes the expected down-\r\n giventwoembeddings,howdifferentthepredictionsofmod- stream instability of linear regression models trained on X\r\n els trained with these embeddings will be. We \ufb01rst de\ufb01ne \u02dc\r\n the eigenspace instability measure and present its theoretical andX,intermsoftheeigenspaceinstability measure, given\r\n connection with downstream instability in Section 4.1; we the covariance matrix \u03a3 of the label vector; but how should\r\n then propose using this measure to ef\ufb01ciently select param- weselect \u03a3? One desirable property for \u03a3 could be that it\r\n eters to minimize downstream instability in Section 4.2. produce label vectors with higher variance in directions be-\r\n lieved to be important, for example because they correspond\r\n 4.1 Eigenspace Instability Measure to eigenvectors with large eigenvalues of an embedding\u2019s\r\n Grammatrix. InSection5, where we evaluate the instability\r\n Wenowde\ufb01netheeigenspaceinstability measure between of pairs of embeddings of various dimensions and preci-\r\n twoembeddings, and show that this measure is directly re- T \u03b1 \u02dc \u02dcT \u03b1\r\n sions, we consider \u03a3 = (EE ) +(EE ) ; in those ex-\r\n lated to the expected disagreement between linear regression \u02dc\r\n periments, E and E are the highest-dimensional (d = 800),\r\n models trained using these embeddings. full-precision embeddings for Wiki\u201917 and Wiki\u201918, respec-\r\n T n\u00d7d \u02dc tively, and \u03b1 is a scalar controlling the relative importance\r\n De\ufb01nition 2. Let X = USV \u2208 R and X =\r\n \u02dc \u02dc\u02dcT n\u00d7k of the directions of high eigenvalue. This choice of \u03a3 re-\r\n USV \u2208 R be the singular value decompositions\r\n \u02dc sults in label vectors with large variance in the directions\r\n (SVDs) of two embedding matrices X and X, and let\r\n \u03a3 \u2208 Rn\u00d7n be a positive semide\ufb01nite matrix. Then the of high eigenvalues of these embedding matrices. In Sec-\r\n eigenspace instability measure between X and \u02dc, with re- tion 5.1 we show that when \u03b1 is chosen appropriately, there\r\n X\r\n spect to \u03a3, is de\ufb01ned as is strong empirical correlation between the eigenspace in-\r\n \u0012 \u0013 stability measure (with this \u03a3) and downstream instability.\r\n 1 \u0010 \u0011\r\n \u02dc T \u02dc \u02dcT \u02dc \u02dcT T\r\n EI\u03a3(X,X):=tr(\u03a3) tr UU +UU \u22122UU UU \u03a3 . 4.2 Jointly Selecting Dimension and Precision\r\n Wenowdemonstrate a practical utility of the eigenspace\r\n Intuitively, this measure captures how different the sub- instability measure: we propose using the measure to ef\ufb01-\r\n \u02dc\r\n spaces spanned by the left singular vectors of X and X are ciently select embedding dimension-precision parameters\r\n to one another; the measure will be equal to zero when the to minimize downstream instability without training the\r\n \u02dc\r\n left singular vectors of X and X span identical subspaces downstream models. In particular, we propose an algorithm\r\n of Rn, and will be equal to one when these singular vectors that takes two or more pairs of embeddings with different\r\n span orthogonal subspaces of Rn whose union covers the dimension-precision parameters as input, and outputs the\r\n whole space. We note that the left singular vectors are par- pair with the lowest eigenspace instability measure between\r\n ticularly important in the case of linear regression models, embeddings. In Section 5.2, we evaluate the performance\r\n because the predictions of the learned model on the train- of this proposed selection algorithm in two settings: \ufb01rst, a\r\n ing examples depend only on the label vector and the left simple setting where the goal is to select the pair with the\r\n 7\r\n singular vectors of the data matrix. lowest downstream instability out of two randomly selected\r\n Wenowpresentourresult showing that the expected mean pairs, and second, a more challenging setting where the goal\r\n squared difference between the linear regression models is to select the pair with the lowest downstream instability\r\n \u02dc out of two or more pairs with the same memory budget. In\r\n trained on X vs. X is equal to the the eigenspace instability both settings, we demonstrate that the eigenspace instability\r\n measure, where \u03a3 corresponds to the covariance matrix of measure outperforms the majority of embedding distance\r\n the regression label vector. For the proof, see Appendix B. measures and is competitive with the other top-performing\r\n n\u00d7d \u02dc n\u00d7k\r\n Proposition 1. Let X \u2208 R , X \u2208 R be two full- embedding distance measure, the k-NN measure.\r\n rank embedding matrices, where xi and x\u02dci correspond to\r\n th \u02dc n\r\n the i rows of X and X respectively. Let y \u2208 R be a ran- 5 EXPERIMENTS\r\n domregression label vector with zero mean and covariance Wenowempirically validate the eigenspace instability mea-\r\n \u03a3\u2208Rn\u00d7n.Thenthe(normalized)expectedmeansquared sure\u2019s relation with downstream instability and demonstrate\r\n 7The linear model trained on data matrix X = USVT \u2208\r\n n\u00d7d n 8 T T \u22121 T \u02dc T\r\n R with label vector y \u2208 R makes predictions Xw = fy(x) = w x,forw = (X X) X y,andfy(x\u02dc) = w\u02dc x\u02dc,\r\n T \u22121 T T n \u02dcT \u02dc \u22121 \u02dcT\r\n X(X X) X y=UU y\u2208R onthentrainingpoints. for w\u02dc = (X X) X y.\r\n Understanding the Downstream Instability of Word Embeddings\r\n that the eigenspace instability measure is an effective selec- distance measure is to the ranking of the pairs of embed-\r\n tion criterion for dimension-precision parameters. In Sec- dings based on their downstream prediction disagreement,\r\n tion 5.1, we show that the theoretically grounded eigenspace with a maximum value of 1.0. In Table 1, we see that the\r\n instability measure strongly correlates with downstream in- eigenspace instability measure and the k-NN measure are\r\n stability, attaining Spearman correlations greater than the the top-performing embedding distance measures by Spear-\r\n weaker baselines (semantic displacement, PIP loss, and man correlation, with the eigenspace instability measure\r\n eigenspace overlap score) and between 0.04 better and 0.09 attaining Spearman correlations between 0.04 better and\r\n worse than the strongest baseline (the k-NN measure). In 0.09 worse than the k-NN measure on all tasks. Moreover,\r\n Section 5.2, when selecting dimension-precision parameters the strong correlation of at least 0.68 for the eigenspace\r\n without training the downstream models, we show that the instability measure across embedding algorithms and down-\r\n eigenspace instability measure attains up to 3.33\u00d7 lower streamtasksvalidatesourtheoreticalclaimthatthismeasure\r\n error rates than weaker baselines and from 0.95\u00d7 to 1.55\u00d7 relates to downstream disagreement. In Appendix D.4, we\r\n 9\r\n the error rate of the k-NN measure. include additional plots showing the downstream prediction\r\n Experimental Setup Toevaluate how predictive the var- disagreement versus the embedding distance measures.\r\n ious embedding distance measures are of downstream in- 5.2 EmbeddingDistanceMeasuresfor\r\n stability, we take the embedding pairs and corresponding Dimension-Precision Selection\r\n downstream models we trained in Section 3 and measure\r\n the embedding distance measures between these pairs of Wedemonstrate that the eigenspace instability measure is\r\n embeddings. Speci\ufb01cally, we compute the k-NN measure, an effective selection criterion for dimension-precision pa-\r\n semantic displacement, PIP loss, eigenspace overlap score, rameters, outperforming the majority of existing embedding\r\n and eigenspace instability measure between the embedding distance measures and competitive with the k-NN measure,\r\n pairs (Section 2.4). Recall that the k-NN measure and the for which there are no theoretical guarantees. Speci\ufb01cally,\r\n eigenspace instability measure each have an important hy- weevaluate the embedding distance measures as selection\r\n perparameter: the k in the k-NN measure, which deter- criteria in two settings of increasing dif\ufb01culty: in the \ufb01rst\r\n mines how many neighbors we compare, and the \u03b1 in the setting the goal is, given two pairs of embeddings (each\r\n eigenspace instability measure, which controls how impor- corresponding to an arbitrary dimension-precision combina-\r\n tant the eigenvectors of high eigenvalue are. For both hyper- tion), to select the pair with the lowest downstream instabil-\r\n parameters, we choose the values with the highest average ity. In the second, more challenging setting, the goal is to\r\n correlation across four sentiment analysis tasks (SST-2, MR, select, among all dimension-precision combinations corre-\r\n Subj, and MPQA) and one NER task (CoNLL-2003) and sponding to the same total memory, the one with the lowest\r\n 10 downstream instability. This setting is challenging, as for\r\n two embedding algorithms (CBOW and MC) whenusing\r\n validation datasets for the downstream tasks (k = 5 and \u03b1 manymemorybudgets,therearemorethantwochoicesof\r\n =3). See Appendix D.3 for more details. The eigenspace embedding pairs, and some choices may have very similar\r\n instability measure also requires additional embeddings E expected downstream instability. We now discuss each of\r\n \u02dc these settings, and the corresponding results, in more detail.\r\n and E: we use 800-dimensional, full-precision Wiki\u201917 and\r\n Wiki\u201918 embeddings as these are the highest dimensional, For the \ufb01rst, simpler setting, we \ufb01rst form all groupings\r\n full-precision embeddings in our study. of two embedding pairs with different dimension-precision\r\n 5.1 Predictive Performance of the Eigenspace combinations. For instance, a grouping may have one em-\r\n Instability Measure bedding pair with dimension 800, precision 32, and another\r\n embedding pair with dimension 200, precision 2, where a\r\n Weevaluate how predictive the eigenspace instability mea- pair consists of a Wiki\u201917 and a Wiki\u201918 embedding from\r\n sure is of downstream instability, showing that the theoret- the same algorithm. For each embedding distance measure,\r\n ically grounded eigenspace instability measure correlates wereport the fraction of groupings where the embedding\r\n strongly with downstreaminstability and is competitive with distance measure correctly chooses the embedding pair with\r\n other embedding distance measures. To do this, we measure lower downstream instability on a given task. We repeat\r\n the Spearman correlations between the downstream pre- over three seeds, comparing embedding pairs of the same\r\n diction disagreement and the embedding distance measure seed, and report the average. In Table 2, we show that\r\n for each of the \ufb01ve tasks and three embedding algorithms. the eigenspace instability measure and k-NN measure are\r\n TheSpearmancorrelation quanti\ufb01es how similar the rank- the most accurate embedding distance measures, with up\r\n ing of the pairs of embeddings based on the embedding to 3.33\u00d7 and 3.73\u00d7 lower selection error rates than the\r\n 9 other embedding distance measures, respectively. Moreover,\r\n Our code is available at https://github.com/ across downstream tasks, the eigenspace instability measure\r\n HazyResearch/anchor-stability. attains 0.95\u00d7 to 1.55\u00d7 the error rate of the k-NN measure.\r\n 10These values also worked well for GloVe (added later).\r\n Understanding the Downstream Instability of Word Embeddings\r\n Table 1. Spearman correlation scores between embedding distance measures and downstream prediction disagreement across varying\r\n dimension-precision pairs for the embedding. Downstream models are trained for sentiment analysis (SST-2, Subj) and NER (CoNLL-\r\n 2003) tasks. Strongest correlation values are bolded.\r\n DownstreamTask SST-2 Subj CoNLL-2003\r\n EmbeddingAlgorithm CBOW GloVe MC CBOW GloVe MC CBOW GloVe MC\r\n Eigenspace Instability 0.68 0.84 0.84 0.72 0.77 0.78 0.80 0.78 0.83\r\n 1\u2212k-NN 0.74 0.86 0.89 0.74 0.76 0.76 0.76 0.86 0.92\r\n Semantic Displacement 0.70 0.34 0.28 0.45 0.43 0.46 0.53 0.16 0.32\r\n PIP Loss -0.40 -0.06 0.39 -0.14 -0.14 0.56 0.01 0.11 0.44\r\n 1\u2212EigenspaceOverlap 0.63 0.18 0.26 0.50 0.29 0.45 0.58 0.01 0.31\r\n Table 2. Selection error when using embedding distance measures to predict the most stable embedding dimension-precision parameters\r\n onsentiment analysis (SST-2, Subj) and NER (CoNLL-2003) downstream tasks. Lowest errors are bolded.\r\n DownstreamTask SST-2 Subj CoNLL-2003\r\n EmbeddingAlgorithm CBOW GloVe MC CBOW GloVe MC CBOW GloVe MC\r\n Eigenspace Instability 0.23 0.15 0.17 0.24 0.21 0.20 0.20 0.20 0.17\r\n 1\u2212k-NN 0.21 0.14 0.13 0.23 0.21 0.21 0.21 0.16 0.11\r\n Semantic Displacement 0.24 0.40 0.42 0.34 0.36 0.34 0.29 0.47 0.41\r\n PIP Loss 0.64 0.50 0.35 0.57 0.54 0.28 0.50 0.44 0.32\r\n 1\u2212EigenspaceOverlap 0.28 0.46 0.43 0.32 0.41 0.34 0.29 0.52 0.41\r\n For the second, more challenging setting, we enumerate all knowledge graph embeddings and contextual word embed-\r\n embedding pairs with different dimension-precision com- dings: as the memory of the embedding increases, the in-\r\n binations which correspond to the same total memory. For stability decreases. We \ufb01rst show how these trends hold on\r\n each embedding measure, we report the average absolute knowledge graph embeddings in Section 6.1 and then on\r\n percentage difference between the downstream instability of contextual word embeddings in Section 6.2.\r\n the pair selected by the measure to the most stable \u201coracle\u201d 6.1 KnowledgeGraphEmbeddings\r\n pair, across different memory budgets. We also introduce\r\n two naive baselines that do not require an embedding dis- Knowledgegraphembeddings(KGEs)areapopulartype\r\n tance measure: high precision, which selects the pair with of embedding that is used for multi-relational data, such\r\n the highest precision possible at each memory budget, and as social networks, knowledge bases, and recommender\r\n low precision, which selects the pair with the lowest preci- systems. Here, we show that as the dimension and preci-\r\n sion possible at each memory budget. As before, we repeat sion of the KGE increases, the stability on two standard\r\n over three seeds, comparing embedding pairs of the same KGEtasksimproves, aligning with the trends we observed\r\n seed, and report the average. We see that the eigenspace on pre-trained word embedding algorithms. Unlike word\r\n instability measure and k-NN measure again outperform the embedding algorithms, the input to KGE algorithms is a di-\r\n other baselines on the majority of downstream tasks, with rected graph, where the relations are the edges in the graph\r\n the eigenspace instability measure attaining a distance up and the entities are the nodes in the graph. The graph can\r\n to 2.98% (absolute) closer to the oracle than the other base- be represented as a set of triplets (h,r,t), where the entity\r\n lines, and average distance to the oracle 0.03% (absolute) head h is related by the relation r to the entity tail t. The\r\n better to 0.35% (absolute) worse than the k-NN measure output is two sets of embeddings: (1) entity embeddings\r\n across downstream tasks (Table 3). For both settings, we (e ) and (2) relation embeddings (r ). We study the stabil-\r\n include additional results measuring the worst-case perfor- h r\r\n ity of these embeddings on two standard benchmark tasks:\r\n manceoftheembeddingdistancemeasureinAppendixD.5, link prediction and triplet classi\ufb01cation. We summarize the\r\n where we \ufb01nd that the eigenspace instability measure and datasets and protocols, and then discuss the results.\r\n k-NNmeasurecontinuetobethetop-performingmeasures.\r\n 6 EXTENSIONS Datasets Weusetwodatasetstotrain KGEembeddings:\r\n FB15K-95andFB15K.FB15KwasintroducedinBordes\r\n We demonstrate that the stability-memory tradeoffs we et al. (2013) and is composed of a subset of triplets from\r\n observe with pre-trained word embeddings can extend to the Freebase knowledge base. We construct FB15K-95 by\r\n randomly sampling 95% of the the triplets from the training\r\n Understanding the Downstream Instability of Word Embeddings\r\n Table 3. Average difference (absolute percentage) to the oracle downstream instability when using embedding distance measures as the\r\n selection criteria for dimension and precision parameters under \ufb01xed memory budgets. Top-performing values are bolded.\r\n DownstreamTask SST-2 Subj CoNLL-2003\r\n EmbeddingAlgorithm CBOW GloVe MC CBOW GloVe MC CBOW GloVe MC\r\n Eigenspace Instability 0.65 0.55 1.42 0.39 0.41 0.63 0.28 0.45 0.43\r\n 1\u2212k-NN 0.57 0.43 1.07 0.38 0.44 0.57 0.32 0.48 0.23\r\n Semantic Displacement 0.37 1.58 3.73 0.48 0.64 0.94 0.27 0.89 1.17\r\n PIP Loss 3.63 2.54 3.32 1.16 1.71 0.74 0.83 0.83 0.99\r\n 1\u2212EigenspaceOverlap 0.88 1.58 3.60 0.34 0.64 0.93 0.20 0.89 1.15\r\n High Precision 0.85 1.58 3.94 0.61 0.64 1.01 0.60 0.89 1.28\r\n LowPrecision 3.63 2.54 1.23 1.16 1.71 1.39 0.83 0.83 0.74\r\n dataset of FB15K. The validation and test datasets remain 100 Link Prediction Triplet Classification\r\n the same for both datasets. We use these datasets to study b=1 35 b=1\r\n the stability of KGEs under small changes in training data. b=2 b=2\r\n 80 b=4 b=4\r\n b=8 30 b=8\r\n Training Protocol We consider a standard KGE b=16 b=16\r\n algorithm\u2014TransE (Bordes et al., 2013). The TransE ob- 60 b=32 25 b=32\r\n jective function minimizes the distances d(e + r ,e ) for % Disagreement20\r\n h r t Unstable-Rank@10 (%)40\r\n observed triplets and maximizes the distances for negatively\r\n sampled triplets, where either h or t has been corrupted. We 10 1 1 02 103 104 10 1 1 02 103 104\r\n use the L distance for the distance function d, and learn Memory (Bits / Vector) Memory (Bits / Vector)\r\n 1\r\n the embeddings iteratively via stochastic gradient descent. Figure 3. Stability of link prediction (left) and triplet classi\ufb01cation\r\n To measure the impact of the dimension and precision on (right) when evaluating embeddings trained on 95% of FB15K\r\n the stability of TransE embeddings, we train TransE embed- training triplets and all of FB15K.\r\n dings of dimensions {10, 20, 50, 100, 200, 400} and then triplet is predicted as positive. For each dimension-precision\r\n uniformly quantize the entity and relation embeddings for\r\n each TransE embedding to bits {1, 2, 4, 8, 16, 32} per entry pair, we set the thresholds on FB15K-95 embedding and use\r\n 11 the same thresholds for the FB15K embedding. We include\r\n in embedding. Weperform a hyperparameter sweep on\r\n the learning using dimension 50, and select the best learning results with threshold set independently for each embedding\r\n rate on the validation set for link prediction. We use this in Appendix D.6. As for classi\ufb01cation with downstream\r\n learning rate for all dimensions to minimize the impact of NLPtasks, we de\ufb01ne stability on the triplet classi\ufb01cation\r\n learning rate on our analysis. We take other training hy- task as the percentage prediction disagreement.\r\n perparameters from the TransE paper (Bordes et al., 2013) Results We\ufb01ndthatthestability-memory tradeoffs con-\r\n for the FB15K dataset, and use three seeds to train each tinue to hold for TransE embeddings on the link prediction\r\n dimension using the OpenKE repository (Han et al., 2018). and triplet classi\ufb01cation tasks: overall as the memory in-\r\n Evaluation Protocol For each dimension-precision, we creases, the instability decreases, and speci\ufb01cally, as the\r\n evaluate all pairs of embeddings trained on FB15K-95 and dimension and precision increases, the instability decreases.\r\n FB15Konthelinkprediction and triplet classi\ufb01cation tasks. In Figure 3 (left), we show for link prediction that as the\r\n For each test triplet, the link prediction task evaluates the memorypervector increases, the unstable-rank@10 mea-\r\n mean predicted rank of an observed triplet among all cor- sure decreases. Each line represents a different precision,\r\n rupted triplets. We measure instability on this task with whereeachpointonthelinerepresentsadifferentdimension.\r\n unstable-rank@10: the fraction of changes in rank greater Thus, we can also see that as the dimension increases, the\r\n than 10 between two embeddings across all test triplets. unstable-rank@10decreases,andasprecisionincreases,this\r\n Thetriplet classi\ufb01cation task was introduced in Socher et al. measure also decreases. When \ufb01tting a linear-log model to\r\n (2013a) and is a binary classi\ufb01cation task to determine the dimension-precision combinations for all memory bud-\r\n whether or not a triplet occurs in the knowledge graph. For gets, we \ufb01nd that increasing the memory 2\u00d7 decreases the\r\n each relation, a threshold T is determined based on the instability by 7% to 19% (relative). In Figure 3 (right), we\r\n R similarly show for triplet classi\ufb01cation that as the memory\r\n validation set, such that if d(e +r ,e ) \u2264 T then the\r\n h r t R per vector increases, the prediction disagreement between\r\n 11Thesamedimensionisusedforboththeentityandtherelation the embeddings trained on the two datasets decreases. Fi-\r\n embeddings. nally, as we saw with word embeddings, we observe that\r\n Understanding the Downstream Instability of Word Embeddings\r\n the effect of the dimension or precision on stability is more diction disagreement), evaluating how different parameters\r\n signi\ufb01cant at low memory regimes. of embeddings impact downstream instability with large-\r\n 6.2 Contextual Word Embeddings scaleWikipediaembeddingsovermultipledownstreamNLP\r\n tasks. Furthermore, we provide theoretical analysis which is\r\n Unlike pre-trained word embeddings, contextual word em- speci\ufb01c to the downstream instability setting to help explain\r\n beddings (Peters et al., 2018; Vaswani et al., 2017) extract our empirical observations.\r\n word representations dynamically with awareness of the More broadly, researchers have also studied the general\r\n input context. We \ufb01nd that the stability-memory trade-off problem of ML model instability in the context of online\r\n observed on pre-trained embeddings can still hold for con- training and incremental learning. Fard et al. (2016) study\r\n textual word embeddings, though with noisier trends: higher the problem of reducing the prediction churn between con-\r\n dimensionality and higher precision can demonstrate better secutively trained classi\ufb01ers by introducing a Monte Carlo\r\n downstream stability. We pre-train shallow, 3-layer versions stabilization operator as a form of regularization. Cotter\r\n of BERT(Devlin et al., 2019) on sub-sampled Wiki\u201917 and et al. (2016) further de\ufb01ne stability as a design goal for clas-\r\n Wiki\u201918 dumps (\u223c200 million tokens) as feature extrac- si\ufb01ers in real-world applications, along with goals such as\r\n tors with different transformer layer output dimensionalities, precision, recall, and fairness, and propose an algorithm to\r\n ranging from a quarter as large to 4\u00d7 as large as the hid- optimize for these multiple design goals. Other researchers\r\n den size in BERT (i.e., 768).12 To evaluate the effect\r\n BASE have also studied the problem of catastrophic forgetting\r\n of precision, we use uniform quantization to compress the whenmodelsareincrementally trained (Yang et al., 2019),\r\n output of the last transformer layer in the BERT models. which shares a similar goal of wanting to learn new infor-\r\n Finally, we measure the prediction disagreement between mation, while minimizing changes with respect to previous\r\n linear classi\ufb01ers trained on top of the Wiki\u201917 and Wiki\u201918 models. Astheseworksfocusonchangestothedownstream\r\n BERTmodels,withtheBERTmodelparameters\ufb01xed. model training to reduce instability, we believe these works\r\n Across four sentiment analysis tasks, we can observe re- are complementary to our work, which focuses on better un-\r\n duced instability with higher dimensional BERT embed- derstanding the instability introduced by word embeddings.\r\n dings (Figure 11a in Appendix D.7); however, the reduction Lastly, although the bias-variance tradeoff is a commonly\r\n in instability from increasing the dimension is noisier than used tool in ML to analyze model stability, there is an im-\r\n with pre-trained word embeddings. We hypothesize this portant difference in our setting. While the variance of a\r\n is due to the instability of the training of the BERT em- modelquanti\ufb01es the expected deviation of the model from\r\n bedding itself, which is a much more complex model than its mean (typically over randomness in the training sample),\r\n pre-trained word embeddings. We also observe that increas- in our work we analyze the disagreement between two sepa-\r\n ing the precision can decrease the downstream instability, rate models trained with different \ufb01xed data matrices on the\r\n such that using 1 or 2 bits for precision often demonstrates samerandomlabelvector.\r\n observable degradation in stability, but precisions higher\r\n than 4-bit have negligible in\ufb02uence on stability (Figure 11b 8 CONCLUSION\r\n in Appendix D.7). For more details on the training and\r\n evaluation, see Appendix D.7. Weperformed the \ufb01rst in-depth study of the downstream\r\n instability of word embeddings. In our study, we exposed a\r\n 7 RELATEDWORK novelstability-memorytradeoff, showingthatincreasingthe\r\n There have been many recent works studying word em- embeddingdimensionorprecision decreases downstream\r\n bedding instability (Hellrich & Hahn, 2016; Antoniak & instability. To better understand these empirical results, we\r\n Mimno,2018;Wendlandtetal., 2018; Pierrejean & Tanguy, introduced a new measure for embedding instability\u2014the\r\n 2018; Chugh et al., 2018; Hellrich et al., 2019); these works eigenspace instability measure\u2014which we theoretically re-\r\n havefocusedontheintrinsicinstabilityofwordembeddings, late to downstream prediction disagreement. We showed\r\n meaning the stability measured between the embedding ma- that this theoretically grounded embedding measure cor-\r\n trices without training a downstream model. In the work relates strongly with downstream instability, and can be\r\n of Wendlandt et al. (2018) they do consider a downstream used to select dimension-precision parameters, performing\r\n task (part-of-speech tagging), but focus on how the intrin- better than or competitively with other embedding mea-\r\n sic instability impacts the error of words on this task. In sures on minimizingdownstreaminstabilitywithouttraining\r\n contrast, we focus on the downstream instability (i.e., pre- the downstream tasks. Finally, we demonstrated that the\r\n stability-memory tradeoff extends to other types of embed-\r\n 12The recent 12-layer BERT model is pre-trained with 3 dings, including contextual word embeddings and knowl-\r\n BASE edgegraphembeddings. Wehopeourstudymotivatesfuture\r\n billion tokens from BooksCorpus (Zhu et al., 2015) and Wikipedia,\r\n and requires 16 TPU chips to train for 4 days. workonMLmodelinstabilityinmorecomplexpipelines.\r\n Understanding the Downstream Instability of Word Embeddings\r\n ACKNOWLEDGEMENTS Chugh, M., Whigham, P. A., and Dick, G. Stability of word\r\n We thank Charles Kuang, Shoumik Palkar, Fred Sala, embeddings using word2vec. In AI 2018: Advances in\r\n Paroma Varma, and the anonymous reviewers for their Arti\ufb01cial Intelligence, pp. 812\u2013818, 2018.\r\n valuable feedback. We gratefully acknowledge the sup- Cotter, A., Friedlander, M. P., Goh, G., and Gupta, M. R.\r\n port of DARPA under Nos. FA87501720095 (D3M), Satisfying real-world goals with dataset constraints. In\r\n FA86501827865 (SDH), and FA86501827882 (ASED); Advances in Neural Information Processing Systems\r\n NIH under No. U54EB020405 (Mobilize), NSF under (NeurIPS), pp. 2415\u20132423, 2016.\r\n Nos. CCF1763315(BeyondSparsity), CCF1563078 (Vol-\r\n ume to Velocity), and 1937301 (RTML); ONR under Covington, P., Adams, J., and Sargin, E. Deep neural net-\r\n No. N000141712266 (Unifying Weak Supervision); the works for youtube recommendations. In Proceedings\r\n Moore Foundation, NXP, Xilinx, LETI-CEA, Intel, IBM, of the 10th ACM Conference on Recommender Systems\r\n Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, (RecSys), pp. 191\u2013198, 2016.\r\n Accenture, Ericsson, Qualcomm, Analog Devices, the\r\n Okawa Foundation, American Family Insurance, Google Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT:\r\n Cloud, Swiss Re, NSF Graduate Research Fellowship under Pre-training of deep bidirectional transformers for lan-\r\n No. DGE-1656518, and members of the Stanford DAWN guage understanding. In Proceedings of the 2019 Confer-\r\n project: Teradata, Facebook, Google, Ant Financial, NEC, ence of the North American Chapter of the Association\r\n VMWare,andInfosys. The U.S. Government is authorized for Computational Linguistics: Human Language Tech-\r\n to reproduce and distribute reprints for Governmental pur- nologies (NAACL-HLT), pp. 4171\u20134186, 2019.\r\n poses notwithstanding any copyright notation thereon. Any\r\n opinions, \ufb01ndings, and conclusions or recommendations Fard, M. M., Cormier, Q., Canini, K. R., and Gupta,\r\n expressed in this material are those of the authors and do M. R. Launch and iterate: Reducing prediction churn.\r\n not necessarily re\ufb02ect the views, policies, or endorsements, In Advances in Neural Information Processing Systems\r\n either expressed or implied, of DARPA, NIH, ONR, or the (NeurIPS), pp. 3179\u20133187, 2016.\r\n U.S. Government.\r\n Gardner, M., Grus, J., Neumann, M., Tafjord, O., Dasigi, P.,\r\n REFERENCES Liu, N. F., Peters, M., Schmitz, M., and Zettlemoyer, L.\r\n AllenNLP: A deep semantic natural language processing\r\n Akbik, A., Blythe, D., and Vollgraf, R. Contextual string platform. In Proceedings of Workshop for NLP Open\r\n embeddings for sequence labeling. In Proceedings of the Source Software (NLP-OSS), pp. 1\u20136, 2018.\r\n International Conference on Computational Linguistics\r\n (COLING),pp.1638\u20131649,2018. Gordon, J. Introducing tensor\ufb02ow hub: A library for\r\n Andrews, M. Compressing word embeddings. In Inter- reusable machine learning modules in tensor\ufb02ow,\r\n national Conference on Neural Information Processing 2018. URLhttps://medium.com/tensorflow/\r\n (ICONIP), pp. 413\u2013422, 2016. introducing-tensorflow-hub-a-library-\r\n for-reusable-machine-learning-\r\n Antoniak, M. and Mimno, D. Evaluating the stability of modules-in-tensorflow-cdee41fa18f9.\r\n embedding-based word similarities. Transactions of the\r\n AssociationforComputationalLinguistics(TACL),6:107\u2013 Hamilton, W. L., Leskovec, J., and Jurafsky, D. Diachronic\r\n 119, 2018. word embeddings reveal statistical laws of semantic\r\n Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. En- change. In Proceedings of the 54th Annual Meeting of\r\n riching word vectors with subword information. Trans- the Association for Computational Linguistics (ACL), pp.\r\n actions of the Association for Computational Linguistics 1489\u20131501, 2016.\r\n (TACL), 5:135\u2013146, 2017. Han, X., Cao, S., Xin, L., Lin, Y., Liu, Z., Sun, M., and Li,\r\n Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., and J. Openke: An open toolkit for knowledge embedding.\r\n Yakhnenko, O. Translating embeddings for modeling In Proceedings of EMNLP, 2018.\r\n multi-relational data. In Advances in Neural Information He, X., Pan, J., Jin, O., Xu, T., Liu, B., Xu, T., Shi, Y.,\r\n Processing Systems (NeurIPS), pp. 2787\u20132795, 2013. Atallah, A., Herbrich, R., Bowers, S., and Candela, J. Q.\r\n Bullinaria, J. A. and Levy, J. P. Extracting semantic repre- Practical lessons from predicting clicks on ads at face-\r\n sentations from word co-occurrence statistics: A compu- book. In Proceedings of the Eighth International Work-\r\n tational study. Behavior Research Methods, 39:510\u2013526, shop on Data Mining for Online Advertising (ADKDD),\r\n 2007. pp. 5:1\u20135:9, 2014.\r\n Understanding the Downstream Instability of Word Embeddings\r\n Hellrich, J. and Hahn, U. Bad company\u2013neighborhoods in the North American Chapter of the Association for Com-\r\n neural embeddingspacesconsideredharmful. InProceed- putational Linguistics: Human Language Technologies\r\n ings of the International Conference on Computational (EMNLP-HLT),pp.2227\u20132237,2018.\r\n Linguistics (COLING), pp. 2785\u20132796, 2016. Pierrejean, B. and Tanguy, L. Predicting word embeddings\r\n Hellrich, J., Kampe, B., and Hahn, U. The in\ufb02uence of variability. In The Seventh Joint Conference on Lexi-\r\n down-sampling strategies on SVD word embedding sta- cal and Computational Semantics (*SEM), pp. 154\u2013159,\r\n bility. In Proceedings of the 3rd Workshop on Evaluating 2018.\r\n Vector Space Representations for NLP, pp. 18\u201326, 2019. \u00a8\r\n Schonemann,P.H. Ageneralizedsolutionoftheorthogonal\r\n Hermann,J. and Balso, M. D. Meet michelangelo: Uber\u2019s Procrustes problem. Psychometrika, 31(1):1\u201310, 1966.\r\n machine learning platform, 2017. URL https:// Sell, T. and Pienaar, W. Introducing Feast: an\r\n eng.uber.com/michelangelo/. open source feature store for machine learning,\r\n Jin, C., Kakade, S. M., and Netrapalli, P. Provable ef\ufb01- 2019. URL https://cloud.google.com/\r\n cient online matrix completion via non-convex stochastic blog/products/ai-machine-learning/\r\n gradient descent. In Advances in Neural Information introducing-feast-an-open-source-\r\n Processing Systems (NeurIPS), pp. 4527\u20134535, 2016. feature-store-for-machine-learning.\r\n Kim,Y. Convolutional neural networks for sentence classi\ufb01- Shiebler, D., Green, C., Belli, L., and\r\n cation. In Proceedings of the 2014 Conference on Empir- Tayal, A. Embeddings@Twitter, 2018.\r\n ical Methods in Natural Language Processing (EMNLP), URL https://blog.twitter.com/\r\n pp. 1746\u20131751, 2014. engineering/en us/topics/insights/\r\n 2018/embeddingsattwitter.html.\r\n \u00b4\r\n May, A., Zhang, J., Dao, T., and Re, C. On the down-\r\n stream performance of compressed word embeddings. Shu, R. and Nakayama, H. Compressing word embeddings\r\n In Advances in Neural Information Processing Systems via deep compositional code learning. In International\r\n (NeurIPS), 2019. Conference on Learning Representation (ICLR), 2018.\r\n Mikolov, T., Chen, K., Corrado, G. S., and Dean, J. Ef\ufb01cient Socher, R., Chen, D., Manning, C. D., and Ng, A. Y. Rea-\r\n estimation of word representations in vector space. arXiv soning with neural tensor networks for knowledge base\r\n preprint arXiv:1301.3781, 2013a. completion. In Advances in Neural Information Process-\r\n ing Systems (NeurIPS), pp. 926\u2013934, 2013a.\r\n Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean,\r\n J. Distributed representations of words and phrases and Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning,\r\n their compositionality. In AdvancesinNeuralInformation C. D., Ng, A., and Potts, C. Recursive deep models\r\n Processing Systems (NeurIPS), pp. 3111\u20133119, 2013b. for semantic compositionality over a sentiment treebank.\r\n In Proceedings of the 2013 Conference on Empirical\r\n Pang, B. and Lee, L. A sentimental education: Sentiment Methods in Natural Language Processing (EMNLP), pp.\r\n analysis using subjectivity summarization based on mini- 1631\u20131642, 2013b.\r\n mumcuts. In \u201dProceedings of the 42nd Annual Meeting Tjong Kim Sang, E. F. and De Meulder, F. Introduction\r\n of the Association for Computational Linguistics (ACL)\u201d, to the CoNLL-2003 shared task: Language-independent\r\n pp. 271\u2013278, 2004. named entity recognition. In Proceedings of the Sev-\r\n Pang, B. and Lee, L. Seeing stars: Exploiting class relation- enth Conference on Natural Language Learning at HLT-\r\n ships for sentiment categorization with respect to rating NAACL2003,pp.142\u2013147,2003.\r\n scales. In Proceedings of the 43rd Annual Meeting of Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones,\r\n the Association for Computational Linguistics (ACL), pp. L., Gomez, A. N., Kaiser, \u0141., and Polosukhin, I. Atten-\r\n 115\u2013124, 2005. tion is all you need. In Advances in Neural Information\r\n Pennington, J., Socher, R., and Manning, C. D. Glove: Processing Systems (NeurIPS), pp. 5998\u20136008, 2017.\r\n Global vectors for word representation. In Empirical Wendlandt, L., Kummerfeld, J., and Mihalcea, R. Factors\r\n Methods in Natural Language Processing (EMNLP), pp. in\ufb02uencing the surprising instability of word embeddings.\r\n 1532\u20131543, 2014. In Proceedings of the 2018 Conference of the North Amer-\r\n Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., ican Chapter of the Association for Computational Lin-\r\n Lee, K., and Zettlemoyer, L. Deep contextualized word guistics: Human Language Technologies (NAACL-HLT),\r\n representations. In Proceedings of the 2018 Conference of pp. 2092\u20132102, 2018.\r\n Understanding the Downstream Instability of Word Embeddings\r\n Wiebe, J., Wilson, T. S., and Cardie, C. Annotating expres-\r\n sions of opinions and emotions in language. Language\r\n Resources and Evaluation, 39:165\u2013210, 2005.\r\n Yang, Y., Zhou, D.-W., Zhan, D.-C., Xiong, H., and Jiang, Y.\r\n Adaptivedeepmodelsforincrementallearning: Consider-\r\n ing capacity scalability and sustainability. In Proceedings\r\n of the 25th ACM SIGKDD International Conference on\r\n Knowledge Discovery & Data Mining (KDD), pp. 74\u201382,\r\n 2019.\r\n Yin, Z. and Shen, Y. On the dimensionality of word em-\r\n bedding. In Advances in Neural Information Processing\r\n Systems (NeurIPS), pp. 887\u2013898, 2018.\r\n Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urta-\r\n sun, R., Torralba, A., and Fidler, S. Aligning books and\r\n movies: Towards story-like visual explanations by watch-\r\n ing movies and reading books. In Proceedings of the\r\n IEEE international conference on computer vision, pp.\r\n 19\u201327, 2015.\r\n Understanding the Downstream Instability of Word Embeddings\r\n A ARTIFACTAPPENDIX\r\n A.1 Abstract\r\n This artifact reproduces the memory-stability tradeoff and the embedding distance measure results for the sentiment analysis\r\n experiments on the word2vec CBOW and matrix completion embedding algorithms. It contains the pre-trained CBOW\r\n and MCembeddingsofsixdifferent dimensions, trained on the Wiki\u201917 and Wiki\u201918 datasets, and the scripts and data for\r\n training the sentiment analysis tasks on these embeddings. It can validate the results in Figures 1 and 2, and Tables 1, 2, and\r\n 3for the sentiment analysis tasks for the MC and CBOW embedding algorithms. We describe the speci\ufb01c steps to reproduce\r\n the SST-2 sentiment analysis results (which received the ACM badges), however, the steps can be easily modi\ufb01ed to validate\r\n the MR, Subj, and MPQA sentiment tasks.\r\n Ourexperimental pipeline consists of 3 main steps: (1) train and compress embeddings, (2) train downstream models and\r\n compute metrics, and (3) run analyses. Because step (1) is very computationally expensive (takes approximately 600 CPU\r\n hours to train all CBOW and MC embeddings using 56 threads), we provide the pre-trained embeddings (they must still be\r\n compressed). This artifact supports reproducing steps (1), (2), and (3), starting from the compression of the embeddings.\r\n Thefull artifact requires 1.1 TB of disk space for storing all embeddings and model output, and requires at least 1 GPU\r\n (tested on NVIDIA K80s) for training downstream models. We also provide a lightweight option to start from (3), which\r\n does not require training downstream models or space to store embeddings and can be run on a local machine. To do this,\r\n weprovide CSVsofthepre-computed embedding distance measures and downstream instabilities.\r\n A.2 Artifact check-list (meta-information)\r\n \u2022 Model: Linear bag-of-words model for sentiment analysis (included).\r\n 13\r\n \u2022 Dataset: SST-2 (included).\r\n \u2022 Run-timeenvironment: DebianGNU/Linux,orUbuntu16.04withCUDA(\u22659.0).\r\n \u2022 Hardware: Computenode(AmazonEC2p2.16xlargeorequivalent)withatleast1NVIDIAK80formodeltraining.\r\n \u2022 Metrics: Embedding distance measures and downstream instability (de\ufb01ned in Section 2).\r\n \u2022 Output: Reproduces SST-2 results in Figures 1 and 2, and in Tables 1, 2, and 3.\r\n \u2022 Experiments: Included shell scripts, Jupyter notebook for plotting.\r\n \u2022 Howmuchdiskspacerequired(approximately)?: 900GBforstoringallembeddings, 200GBforstoringSST-2modeland\r\n analysis results.\r\n \u2022 Howmuchtimeisneededtopreparework\ufb02ow(approximately)?: 30minutesforinstallingdependencies.\r\n \u2022 Howmuchtimeisneededtocomplete experiments (approximately)?: 17 CPU hours for embedding compression, 43 GPU\r\n hours for model training, 15 CPU hours for metric computation, and 1-3 minutes for analysis. Note embedding compression, model\r\n training, and metric computation are easily parallelizable.\r\n \u2022 Publicly available?: Yes.\r\n \u2022 Codelicenses (if publicly available)?: MIT License.\r\n A.3 Description\r\n A.3.1 Howtoaccess\r\n Our source code is publicly available on GitHub: https://github.com/HazyResearch/anchor-stability. Pre-trained\r\n embeddings are currently stored in a publicly accessible Google Cloud storage bucket (script to download from the bucket is provided in\r\n the GitHub repository in run get embs.sh).\r\n Wealsohavethesourcecodeandpre-trainedemebddingspermanentlyavailableathttps://doi.org/10.5281/zenodo.3687120,\r\n which obtained the ACM badges.\r\n 13Wedescribe how to modify the scripts for the other provided datasets of MR, Subj, MPQA in Section A.7.\r\n Understanding the Downstream Instability of Word Embeddings\r\n A.3.2 Hardwaredependencies\r\n WerecommendanAmazonEC2p2.16xlargeorequivalentfortheembeddingcompression,modeltraining, and metric computation steps.\r\n For a Base AMI, we suggest the Deep Learning AMI (Ubuntu 16.04) Version 26.0 (ami-025ed45832b817a35).\r\n A.3.3 Software dependencies\r\n Wetested our implementation on Ubuntu 16.04 with CUDA 9.0. We recommend using a conda environment or Python virtualenv, and we\r\n provide a requirements.txt\ufb01lewiththePythondependencies. Wetested our implementation with Python 3.6 and PyTorch 1.0.\r\n A.4 Installation\r\n Please see the https://github.com/HazyResearch/anchor-stability/blob/master/README.md \ufb01le for detailed\r\n installation instructions and scripts.\r\n A.5 Experimentwork\ufb02ow\r\n Weprovideshell scripts to run to reproduce each of the steps.14 Here we summarize the work\ufb02ow; please see the README.md for more\r\n detailed instructions and speci\ufb01c commands to run.\r\n 1. Obtain the pre-trained MC and CBOW embeddings trained on Wiki\u201917 and Wiki\u201918 and compress all embeddings to precisions\r\n b \u2208 {1,2,4,8,16,32}.\r\n 2. Train the downstream models on top of all of the compressed embeddings for the SST-2 task. After the models are done training,\r\n computetheembeddingdistancemeasuresanddownstreaminstabilitybetweenpairsofembeddingstrainedonWiki\u201917andWiki\u201918,\r\n and their corresponding pairs of models.\r\n 3. Run the analysis script to evaluate the Spearman correlations of the embedding distance measures with the downstream instabilities,\r\n and the selection criterion results for the tasks described in Section 5.2. Finally, graph the memory-stability tradeoff results with the\r\n Jupyter notebooks provided.\r\n A.6 Evaluation and expected result\r\n Step 3 in Section A.5 should reproduce the results for the CBOW and MC embeddings for the SST-2 sentiment analysis task in Figures 1\r\n and 2, as well as the results in Tables 1, 2, and 3, using the compressed embeddings, trained models, and measured instabilities generated\r\n in Steps 1 and 2. Note there might be slight variance in the k-NN results (+/- 0.03 for Spearman correlation and selection error).\r\n Using our provided CSVs \ufb01le (see the results directory), Step 3 should also reproduce the remaining analysis results for the MR, Subj,\r\n and MPQAsentimentanalysis tasks, as well as the CoNLL-2003 NER task found in Table 1, 2, and 3, as well as the linear-log trends\r\n described in Section 3.\r\n A.7 Experimentcustomization\r\n To reproduce the complete pipeline of results on the MR, Subj, and MPQA tasks, modify the run models.sh and\r\n run collect results.shscripttousetheMCandCBOWlearningrates(MC LRandCBOW LR)thatwefoundfromourgridsearch\r\n (AppendixC.3.1)forthenewtaskandupdatetheDATASETvariabletothenewtask. Thenpassthenewtasknametorun analysis.sh\r\n (e.g., with bash run analysis.sh mr).\r\n In terms of extending the results, the pre-trained embeddings we provide could be used to train more models to further measure\r\n the impact of the embedding instability on downstream instability. New embedding distance measures could also be added to\r\n anchor/embedding.pyandeasilyevaluatedagainstthemeasurespresentedinthispaperintermsoftheircorrelationwithdownstream\r\n instability.\r\n A.8 Methodology\r\n Submission, reviewing and badging methodology:\r\n \u2022 http://cTuning.org/ae/submission-20200102.html\r\n \u2022 http://cTuning.org/ae/reviewing-20200102.html\r\n \u2022 https://www.acm.org/publications/policies/artifact-review-badging\r\n 14If resource limited, to only reproduce the analysis results, we provide the CSVs of the embedding distance measures between pairs of\r\n embeddings and downstream instabilities between pairs of corresponding models.\r\n Understanding the Downstream Instability of Word Embeddings\r\n B EIGENSPACEINSTABILITY: THEORY\r\n Wepresent the proof of Proposition 1, which shows that the expected prediction disagreement between the linear regression models\r\n \u02dc \u02dc\r\n trained on embedding matrices X and X is equal to the eigenspace instability measure between X and X.\r\n n\u00d7d \u02dc n\u00d7k th\r\n Proposition 1. Let X \u2208 R , X \u2208 R be two full-rank embedding matrices, where x and x\u02dc correspond to the i rows of X and\r\n i i\r\n \u02dc n n\u00d7n\r\n Xrespectively. Let y \u2208 R be a random regression label vector with zero mean and covariance \u03a3 \u2208 R . Then the (normalized)\r\n \u02dc 15 \u02dc\r\n expected disagreement between the linear models fy and fy trained on label vector y using embedding matrices X and X respectively\r\n satis\ufb01es\r\n hP i\r\n n \u02dc 2\r\n E (f (x ) \u2212f (x\u02dc ))\r\n y i=1 y i y i \u02dc\r\n 2 =EI\u03a3(X,X). (2)\r\n E [kyk ]\r\n y\r\n T n\u00d7d \u02dc \u02dc \u02dc\u02dcT n\u00d7d \u02dc d th\r\n Proof. Let X = USV \u2208R and X = USV \u2208R be the SVDs of X and X respectively, and let x and x\u02dc in R be the i\r\n i i\r\n \u02dc d 2 \u2217 T \u22121 T\r\n rows of X and X. Recall that parameter vector w \u2208 R which minimizes kXw \u2212yk is given by w = (X X) X y(wherehere\r\n 2\r\n T T \u2217\r\n weusetheassumption that X is full-rank to know that X X is invertible). Thus, the linear regression model fy(x) = x w trained on\r\n n \u2217 T \u22121 T T \u22122 T T T n\r\n data matrix X with label vector y \u2208 R makes predictions Xw = X(X X) \u02dc X y = USV (VS V )VSU y=UU y\u2208R\r\n onthe n training points. So if we train linear model with data matrices X and X, using the same label vector y, these model will make\r\n T \u02dc \u02dcT\r\n predictions UU y and UU y on the n training points, respectively. Thus, the expected disagreement between the predictions made\r\n \u02dc\r\n using X vs. X, over the randomness in y, can be expressed as follows:\r\n \" n # h i\r\n X \u02dc 2 T \u02dc \u02dcT 2\r\n E (f (x ) \u2212f (x\u02dc )) = E kUU y\u2212UU yk\r\n y y i y i y\r\n i=1 \u0014 \u0015\r\n = E \u0010 T \u02dc \u02dcT \u0011T \u0010 T \u02dc \u02dcT \u0011\r\n y UU y\u2212UU y UU y\u2212UU y\r\n h T T T T \u02dc \u02dcT \u02dc \u02dcT T \u02dc \u02dcT T i\r\n = E y UU UU y+y UU UU y\u22122y UU UU y\r\n y\r\n h T \u0010 T \u02dc \u02dcT \u02dc \u02dcT T\u0011 i\r\n = E y UU +UU \u22122UU UU y\r\n y\r\n \u0014 \u0012 T \u0010 T \u02dc \u02dcT \u02dc \u02dcT T\u0011 \u0013\u0015\r\n = E tr y UU +UU \u22122UU UU y\r\n y\r\n \u0012\u0010 T \u02dc \u02dcT \u02dc \u02dcT T\u0011 h Ti\u0013\r\n = tr UU +UU \u22122UU UU E yy\r\n y\r\n \u0012\u0010 T \u02dc \u02dcT \u02dc \u02dcT T\u0011 \u0013 h Ti\r\n = tr UU +UU \u22122UU UU \u03a3 , where\u03a3=E yy\r\n y\r\n Furthermore, we can easily compute the expected norm of the label vector y.\r\n E \u0002kyk2\u0003 = E htr(yTy)i\r\n y y\r\n = E htr(yyT)i\r\n y\r\n = tr(E hyyTi)\r\n y\r\n = tr(\u03a3).\r\n Thus, we have successfully shown that\r\n hP i \u0012\u0010 T \u02dc \u02dcT \u02dc \u02dcT T\u0011 \u0013\r\n n \u02dc 2 tr UU +UU \u22122UU UU \u03a3\r\n E (f (x ) \u2212f (x\u02dc ))\r\n y i=1 y i y i\r\n E [kyk2] = tr(\u03a3)\r\n y\r\n \u02dc\r\n =: EI\u03a3(X,X),\r\n as desired.\r\n 15 T T \u22121 T \u02dc T \u02dcT \u02dc \u22121 \u02dcT\r\n f (x) = w x, for w = (X X) X y,andf (x\u02dc) = w\u02dc x\u02dc, for w\u02dc = (X X) X y.\r\n y y\r\n Understanding the Downstream Instability of Word Embeddings\r\n B.1 Ef\ufb01ciently Computing the Eigenspace Instability Measure\r\n T \u03b1 \u02dc \u02dcT \u03b1 2\u03b1 T\r\n Wenowdiscussanef\ufb01cientwayofcomputingtheeigenspaceinstability measure, assuming \u03a3 = (EE ) +(EE ) = VR V +\r\n \u02dc \u02dc2\u03b1\u02dcT \u02dc 16 T \u02dc \u02dc \u02dc \u02dc T\r\n VR V asdiscussedinSection4.1. Here,E andE correspondto\ufb01xedembeddingmatrices, whereE = VRW and E = VRW\r\n \u02dc\r\n are the SVDs of E and E respectively.\r\n Recall the de\ufb01nition of the eigenspace instability measure:\r\n \u0012 \u0013\r\n \u02dc 1 \u0010 T \u02dc \u02dcT \u02dc \u02dcT T\u0011\r\n EI\u03a3(X,X):= tr(\u03a3) tr UU +UU \u22122UU UU \u03a3 .\r\n Wenowshowthatbothtracesinthisexpression can be computed ef\ufb01ciently.\r\n \u0012\u0010 T \u02dc \u02dcT \u02dc \u02dcT T\u0011 \u0013 \u0012\u0010 T \u02dc \u02dcT \u02dc \u02dcT T\u0011\u0010 2\u03b1 T \u02dc \u02dc2\u03b1\u02dcT\u0011\u0013\r\n tr UU +UU \u22122UU UU \u03a3 = tr UU +UU \u22122UU UU VR V +VR V\r\n \u0010 \u03b1 T T \u03b1\u0011 \u0010 \u03b1 T \u02dc\u02dcT \u03b1\u0011 \u0010 \u03b1 T \u02dc\u02dcT T \u03b1\u0011\r\n = tr\u0010R V UU VR \u0011+tr\u0010R V UU VR \u0011\u22122tr\u0010R V UU UU VR \u0011+\r\n \u02dc\u03b1\u02dcT T \u02dc \u02dc\u03b1 \u02dc\u03b1\u02dcT \u02dc \u02dcT \u02dc \u02dc\u03b1 \u02dc\u03b1\u02dcT \u02dc \u02dcT T \u02dc \u02dc\u03b1\r\n tr R V UU VR +tr R V UU VR \u22122tr R V UU UU VR\r\n T \u03b1 2 \u02dcT \u03b1 2 \u0010 \u03b1 T\u02dc \u02dcT T \u03b1\u0011\r\n = kU VR kF +kU VR kF \u22122tr\u0010R (V U)(U U)(U V)R \u0011+\r\n T \u02dc \u02dc\u03b1 2 \u02dcT \u02dc \u02dc\u03b1 2 \u02dc\u03b1 \u02dcT \u02dc \u02dcT T \u02dc \u02dc\u03b1\r\n kU VR kF +kU VR kF \u22122tr R (V U)(U U)(U V)R . (3)\r\n \u0010 2\u03b1 T \u02dc \u02dc2\u03b1\u02dcT\u0011\r\n tr(\u03a3) = tr VR V +VR V\r\n \u0010 T 2\u03b1\u0011 \u0010\u02dcT\u02dc \u02dc2\u03b1\u0011\r\n = tr V VR +tr V VR\r\n 2\u03b1\u0001 \u0010\u02dc2\u03b1\u0011\r\n = tr R +tr R . (4)\r\n Wenownotethat the traces in Equation (4), and all the matrix multiplications in Equation (3) can be computed ef\ufb01ciently and with\r\n low-memory(noneedtoeverstoreannbynGrammatrix,forexample),assumingtheembeddingmatricesare\u201ctallandthin\u201d(large\r\n vocabulary, relatively low-dimensional). More speci\ufb01cally, the eigenspace instability measure can be computed in time O(nd2) and\r\n \u2032\r\n 2 \u02dc \u02dc n\u00d7d n\u00d7d \u2032\r\n memoryO(d ),wherewetakeX,X,E,E toallbeinR (or in R for d \u2264 d). Thus, even for large vocabulary n, the eigenspace\r\n instability measure can be computed relatively ef\ufb01ciently (assuming the dimension d isn\u2019t too large).\r\n C EXPERIMENTALSETUPDETAILS\r\n Wediscusstheexperimental protocols used for each of our experiments. In Appendix C.1, we discuss the training procedures for the word\r\n embeddings, and in Appendix C.2, we discuss how we compress and post-process the embeddings. In Appendix C.3, we describe the\r\n models, datasets, and training procedures used for the downstream tasks in our study, and in Appendix C.4, we discuss how we analyze\r\n the instability trends we observe on these tasks. Finally, in Appendix C.5 and Appendix C.6 we describe setup details for the extension\r\n experiments on knowledge graph and contextual word embeddings, respectively.\r\n C.1 WordEmbeddingTraining\r\n 17 18\r\n WeuseGoogle\u2019sCimplementationofword2vecCBOW ,theoriginalGloVeimplementation ,andourownC++implementationofMC\r\n to train word embeddings. For CBOW, we use the default learning rate. For GloVe, we use a learning rate of 0.01 (as the default of 0.05\r\n resulted in NaNs on 800-dimensional Wiki embeddings). For MC, since we are using our own implementation, we use a learning rate\r\n which we found to achieve low loss on Wiki\u201917. We include the full details on the hyperparameters used for both embedding algorithms\r\n in Table 4.\r\n C.2 WordEmbeddingCompressionandPost-Processing\r\n We now discuss some important implementation details for uniform quantization related to stability. We use the techniques and\r\n implementation from May et al. (2019). To minimize confounding factors with stability, we use deterministic rounding for each word.\r\n The bounds of the interval for uniform quantization are determined by computing an optimal clipping threshold which is based on\r\n \u02dc\r\n the distribution of the real numbers to be quantized. As we assume that embeddings X and X have similar distributions in terms of\r\n \u02dc\r\n their vector values, we use the same clipping threshold across embeddings X and X to avoid unnecessary sources of instability, and\r\n 16 \u02dc\r\n In our experiments, E and E are the highest-dimensional (d = 800), full-precision embeddings for Wiki\u201917 and Wiki\u201918, respectively.\r\n 17https://github.com/tmikolov/word2vec\r\n 18https://github.com/stanfordnlp/GloVe\r\n Understanding the Downstream Instability of Word Embeddings\r\n Table 4. Hyperparameters for embedding algorithms.\r\n Algorithm Hyperparameter Value\r\n Shared Training epochs 50\r\n Windowsize 15\r\n Minimumcount 5\r\n Threads 56\r\n CBOW Learning rate 0.05\r\n Negative samples 5\r\n GloVe Learning rate 0.01\r\n xmax 100\r\n \u03b1 0.75\r\n MC Learning rate 0.2\r\n LRdecayepochs 20\r\n Batch size 128\r\n Stopping tolerance 0.0001\r\n \u02dc\r\n wecomputetheclippingthreshold using embedding X. Finally, we apply orthogonal Procrustes to align embedding X to embedding\r\n Xbefore compressing the embeddings and training downstream models. Preliminary results indicated that this alignment decreased\r\n instability, particularly at high compression rates, and we use this technique throughout our experiments.\r\n C.3 DownstreamTasks\r\n Wediscuss the models, datasets, and training procedure we use for the sentiment analysis and NER tasks.\r\n C.3.1 Sentiment Analysis\r\n Weuseasimple,bag-of-wordsmodelforsentiment analysis. The goal of the task is to classify a sentence as positive or negative. For each\r\n sentence, the bag-of-words model averages the word embeddings of the words in the sentence and then passes the sentence embedding\r\n through a linear classi\ufb01er. This simple model allows us to study the impact of the embedding on the downstream task in a controlled\r\n setting, where the downstream model itself is expected to be fairly stable.\r\n Weusefourdatasets for the sentiment analysis task: SST-2, MR, Subj, and MPQA. These are the four largest binary classi\ufb01cation datasets\r\n used in Kim (2014).19 We use their given train/validation/test splits for SST-2. For MR, Subj, and MPQA, which do not have these splits,\r\n wetake10%ofthedataforthevalidation set, 10% for the test set, and use the remaining 80% for the training set.\r\n Wetunethelearningratefor each dataset and embedding algorithm. We use the 400-dimensional Wiki\u201917 embeddings to tune the learning\r\n rate in the grid of {1e-6, 1e-5, 0.0001, 0.001, 0.01, 0.1, 1}. We choose the learning rate which achieves the highest validation accuracy on\r\n average across three seeds for each dataset and report the selected values in Table 5a. To avoid choosing unstable learning rates, we also\r\n throw out learning rate values where the validation errors increase by 15% or greater between any consecutive epochs. We include the\r\n hyperparameters shared among all datasets in Table 5b.\r\n Table 5. (a) shows selected learning rates on the sentiment analysis tasks for each embedding algorithm. (b) shows training hyperparameters\r\n for the sentiment analysis datasets that are shared across embedding algorithms.\r\n (a) Tuned learning rates for the sentiment analysis datasets. (b) Shared training hyperparameters.\r\n Algorithm SST-2 MR Subj MPQA Hyperparameter Value\r\n CBOW 0.0001 0.001 0.0001 0.001 Optimizer Adam\r\n GloVe 0.01 0.01 0.01 0.001 Batch size 32\r\n MC 0.001 0.1 0.1 0.001 Training epochs 100\r\n C.3.2 NamedEntity Recognition\r\n Weusethesingle-layer, BiLSTM model from Akbik et al. (2018) for named entity recognition.20 We turn off the conditional random \ufb01eld\r\n (CRF)for computational ef\ufb01ciency and include a smaller subset of results with the CRF turned on in Appendix E.2.\r\n 19https://github.com/harvardnlp/sent-conv-torch/tree/master/data\r\n 20https://github.com/zalandoresearch/flair\r\n Understanding the Downstream Instability of Word Embeddings\r\n Weuse the standard English CoNLL-2003 dataset with the default setup for dataset splits (Tjong Kim Sang & De Meulder, 2003).\r\n 21\r\n Following Gardner et al. (2018), we ignore article divisions (denoted with \u201c-DOCSTART-\u201d) and do not consider them as sentences.\r\n Wetunethelearning rate per embedding algorithm, and otherwise follow the training hyperparameter settings of Akbik et al. (2018).\r\n Using the 400-dimensional Wiki\u201917 embeddings, we sweep the learning rate in the grid of {0.001, 0.01, 0.1, 1, 10}, and choose the one\r\n which achieves the highest validation micro F1-score on average across three seeds for each embedding algorithm. We train with vanilla\r\n SGDwithoutmomentumanduselearningdecaywithearly stopping if the learning rate becomes too small. We provide the selected\r\n learning rates in Table 6a and the hyperparameters shared across embeddings in Table 6b.\r\n Table 6. (a) shows selected learning rates for the NER task per embedding algorithm. (b) shows training hyperparameters shared across\r\n embedding algorithms for the NER task.\r\n (b) Shared training hyperparameters.\r\n Hyperparameter Value\r\n (a) Tuned learning rates for NER. Optimizer SGD\r\n Batch size 32\r\n CBOW GloVe MC Max. training epochs 150\r\n LSTMhiddensize 256\r\n 0.1 1.0 1.0 LSTMnum. layers 1\r\n Patience 3\r\n Anneal factor 0.5\r\n Worddropout 0.05\r\n Locked dropout 0.5\r\n C.4 Fitting Linear-Log Models to Trends\r\n Wedescribe in detail how we \ufb01t linear-log model to the memory, dimension, and precision trends in Section 3.3. To propose the simple\r\n rule of thumb relating stability and memory, we consider 10 tasks to form a data matrix for the linear-log model: 5 downstream tasks\r\n (the four sentiment tasks in our study and the NER task) for two embedding algorithms (CBOW and MC embeddings). Let P denote\r\n the number of Wiki\u201917/Wiki\u201918 pairs of embedding matrices from our experiments which correspond to a combination of dimension\r\n 3\r\n d, precision b, and random seed s (we consider 3 random seeds) such that the number of bits per row is less than our cutoff of 10\r\n 3 22 (t) P\u00d7(T+1) (t) P\r\n (bd < 10 ). For each task t (out of T = 10 total tasks), we construct a data matrix X \u2208R , and a label vector y \u2208R ,as\r\n follows: Each row in X(t) corresponds to one of the above P pairs of Wiki\u201917/Wiki\u201918 embedding matrices. For each of these embedding\r\n matrix pairs, we compute the memory m\u2032 in bits occupied per row of the embedding matrices, as well as the downstream prediction\r\n disagreement percentage y\u2032 \u2208 [0,100] between the models trained on those embeddings. We then set the corresponding row in X(t) to be\r\n \u2032 T+1 T\r\n [log (m ),e ] \u2208 R , where e \u2208 R is a binary vector with a one at index t and zeros everywhere else, and the corresponding entry\r\n 2 t t\r\n of y(t) to the prediction disagreement y\u2032; note that appending et to log2(m\u2032) allows us to learn a different bias term (i.e., y-intercept)\r\n per task. We then vertically concatenate all the X(t) matrices and label vectors y(t), to form a single data matrix X \u2208 RTP\u00d7(T+1) and\r\n TP\r\n label vector y \u2208 R . To \ufb01t our log-linear model, we use X and y to solve the least squares problem using the closed form solution,\r\n \u02c6 T \u22121 T \u02c6 T+1 \u02c6 \u02c6\r\n \u03b2 = (X X) X y. Given\u03b2 \u2208 R , for each task t we can extract the \ufb01tted log-linear trend: DI \u2248 \u03b2 \u2212 \u03b2 \u2217 log (m), where\r\n t t 0 2\r\n \u02c6 \u02c6 \u02c6 th \u02c6\r\n \u03b2 \u22481.3isthe\ufb01rstelementof\u03b2,and\u03b2 isthe(t+1) element of \u03b2. This implies that doubling the memory of the embeddings on\r\n 0 t\r\n average leads to a 1.3% reduction in downstream prediction disagreement.\r\n To \ufb01t the individual dimension and precision log-linear trends, we follow a protocol very similar to the above. For the dimension\r\n (respectively, precision) trend, the primary difference with the above protocol is that instead of having an independent y-intercept term per\r\n task, we have an independent y-intercept term for each combination of task and precision (resp., dimension). Furthermore, in the rows of\r\n the data matrices, instead of log2(\u00b7) of the memory m, we consider log2(\u00b7) of the dimension d (resp., precision b).\r\n Wealsousethelinear-logmodelforstability-memorytocomputetheminimumandmaximumrelativepercentagedecreasesindownstream\r\n instability when increasing the memory of word embeddings. In particular, our goal is to understand how much the 1.3% decrease in\r\n prediction disagreement is in relative terms. To do this, we consider the combination of downstream task and embedding algorithm which\r\n is most stable at high memory (task: Subj; embedding algorithm: CBOW), and the combination which is least stable at low memory\r\n (task: MR; embedding algorithm: MC). At these extreme points, the instability is approximately 2.2% and 25.9%, respectively. A 1.3%\r\n 1.3 \u0001\r\n absolute decrease in instability from 3.5% to 2.2% corresponds to a relative decrease of approximately 37% 3.5 \u2248 0.37 . Similarly, a\r\n 1.3 \u0001\r\n 1.3%absolute decrease in instability from 25.9% to 24.6% corresponds to a relative decrease of approximately 5% 25.9 \u2248 0.05 . Thus,\r\n weconcludethat this 1.3% absolute decrease in instability corresponds to a relative decrease in instability between 5% and 37%, across\r\n the tasks and embedding algorithms we consider.\r\n Werepeat the procedures above to \ufb01t a linear-log model to the stability-memory trend for knowledge graph embeddings in Section 6.1.\r\n 21https://github.com/allenai/allennlp\r\n 22In our case P = 63, because we have 3 random seeds, and 21 pairs of dimension d \u2208 {25,50,100,200,400,800} and precision\r\n 3\r\n b \u2208 {1,2,4,8,16,32} such that db < 10 .\r\n Understanding the Downstream Instability of Word Embeddings\r\n C.5 KnowledgeGraphEmbeddings\r\n WeusetheOpenKErepositorytogenerateknowledgegraphembeddings(Hanetal.,2018).23 We follow the training hyperparameters\r\n described in Bordes et al. (2013) for TransE embeddings for the FB15K dataset where available, and use default parameters from the\r\n OpenKErepository, otherwise. We modify the repository to follow the early stopping procedure and normalization of entity embeddings\r\n to follow the protocol of Bordes et al. (2013). We additionally sweep the learning rate in {1e-5, 0.0001, 0.001, 0.01, 0.1} using dimension\r\n 50 on the FB15K-95 dataset, and choose the learning rate which attains the lowest mean rank (i.e., highest quality) on the validation\r\n set for the link prediction task. We include the full hyperparameters in Table 7. We also note that unlike with word embeddings, we do\r\n not align embeddings with orthogonal Procrustes before compressing the embeddings with uniform quantization. We found alignment\r\n to result in a quality drop on knowledge graph embeddings, likely due to the fact that there are two sets of embeddings jointly learned\r\n (relation and entity embeddings) which require more advanced alignment techniques.\r\n Table 7. Hyperparameters for training TransE knowledge graph embeddings. Bolded values indicate we performed a grid search. Other\r\n values are from Bordes et al. (2013) and Han et al. (2018).\r\n Hyperparameter Value\r\n Optimizer SGD\r\n Max. training epochs 1000\r\n Num. batches 100\r\n Threads 8\r\n Early stopping patience 10\r\n Head/tail replacement strategy Uniform\r\n Entity negative rate 1\r\n Relation negative rate 0\r\n Margin \u03b3 1\r\n Distance d L1\r\n Learning rate 0.001\r\n C.6 Contextual Word Embeddings\r\n Tostudy the downstream instability of contextual word embeddings, we pre-train BERT Devlin et al. (2019) models and then use them as\r\n \ufb01xed feature extractors to train downstream task models. We use BERT without \ufb01ne-tuning parameters for downstream tasks because our\r\n goal is to isolate and study the instability resulting from the difference in pre-training corpora; this is in analogy to our study in Section 3\r\n onthe instability of conventional \ufb01xed pre-trained embeddings.\r\n Pre-training In the pre-training phase, we use Wikipedia dumps (the major component of the corpus used by Devlin et al. (2019)) to\r\n train the BERT models. We use Wiki\u20192017 and Wiki\u20192018 dumps respectively for pre-training to study the instability introduced by the\r\n change in corpora. We pre-train BERT models with 3 transformer layers on 10% subsampled articles from the Wikipedia dumps, which\r\n consists of approximately 200 million tokens. We use these shallower BERT model on the subsampled pre-training corpus to allow for\r\n computationally feasible training of BERT models with different transformer output dimensionality. As our corpus size are different from\r\n the one used by the original BERT model (Devlin et al., 2019), we \ufb01rst grid search the pre-training learning rate with the subsampled\r\n Wiki\u201917 corpus using the same transformer output dimensionality as the BERT . We then use the grid-searched optimal learning rate\r\n BASE 24\r\n to pre-train the BERT model with different transformer dimensionality for both Wiki\u201917 and Wiki\u201918 corpus.\r\n DownstreamEvaluation Toevaluatethedownstreaminstabilityofpre-trained BERT models, we take BERT model pairs with the\r\n samemodelcon\ufb01guration but trained on Wiki\u201917 and Wiki\u201918 respectively. We measure the percentage of disagreement in downstream\r\n task prediction of the BERT pairs as proxy for downstream instability. Speci\ufb01cally, we evaluate the instability on the sentiment analysis\r\n task using the SST, Subj, MR and MPQA datasets. In these tasks, we use linear bag-of-words models on top of the last transformer layer\r\n output; this output acts as the contextual word vector representation. To train the sentiment analysis task models, we \ufb01rst grid-search the\r\n learning rate using BERT with 768-dimensional transformer output for each dataset and choose the value with the highest validation\r\n accuracy.25 We then use the grid-searched learning rate to train the sentiment analysis models using different pre-trained BERT models.\r\n To ensure statistically meaningful results, we use three random seeds to pre-train BERT models and train the downstream sentiment\r\n analysis models. We otherwise use the same hyperparameters reported in Table 5b.\r\n D EXTENDEDEMPIRICALRESULTS\r\n Wenowpresentadditional experimental results to further validate the claims in this paper and provide deeper analysis of our results. We\r\n organize this section as follows:\r\n 23https://github.com/thunlp/OpenKE/tree/OpenKE-PyTorch\r\n 24Wefollow the experiment design from pre-trained word embeddings to use the same learning rate for pre-training BERT models with\r\n transformer con\ufb01gurations.\r\n 25Weusethedimensionality used for original BERT (Devlin et al., 2019).\r\n BASE\r\n Understanding the Downstream Instability of Word Embeddings\r\n \u2022 In Appendix D.1, we present additional results showing that the stability-memory trends (and individual dimension and precision\r\n trends) hold on sentiment analysis tasks.\r\n \u2022 In Appendix D.2, we evaluate another important property\u2013quality\u2014exploring the tradeoffs of quality with memory and stability for\r\n the tasks in our study.\r\n \u2022 In Appendix D.3, we discuss how we choose the additional hyperparameters required for both the k-NN measure and the eigenspace\r\n instability measure.\r\n \u2022 In Appendix D.4, we use visualizations to further analyze the relationship between the downstream instability and the embedding\r\n distance measures.\r\n \u2022 In Appendix D.5, we include additional results on sentiment analysis tasks for the evaluation of the embedding distance measures.\r\n Wealsoevaluate the worst-case performance of the embedding distance measures as selection criteria, showing that the eigenspace\r\n instability measure and k-NN measure remain the top-performing measures overall.\r\n \u2022 In Appendix D.6, we experiment with a modi\ufb01ed setup for the triplet classi\ufb01cation task, showing that the trends continue to hold,\r\n but the instability plateaus faster under this modi\ufb01cation.\r\n \u2022 In Appendix D.7, we include the \ufb01gures for the contextual word embedding results presented in Section 6.2.\r\n D.1 Stability-Memory Tradeoff\r\n Wevalidate that the stability-memory tradeoff holds on three more sentiment tasks (Subj, MR, and MPQA) for dimension and precision,\r\n \ufb01rst in isolation and then together. As always, we train embeddings and downstream models over three seeds, and the error bars indicate\r\n the standard deviation over these seeds. In Figure 4, we can see more evidence that as the dimension increases, the downstream instability\r\n often decreases, with the trends more consistent for lower precision embeddings. In Figure 5, we further validate that as the precision\r\n increases, the downstream instability decreases. Finally, in Figure 6, we show on all four sentiment tasks (SST-2, Subj, MR, and MPQA)\r\n that when jointly varying dimension and precision, the instability decreases as the memory increases.\r\n Subj MR MPQA\r\n 6 10.0 CBOW\r\n 15 GloVe\r\n MC\r\n 4 10 7.5\r\n 5.0\r\n % Disagreement2 % Disagreement5 % Disagreement2.5\r\n 25 50 Dimension100200400800 25 50 Dimension100200400800 25 50 Dimension100200400800\r\n (a) 32-bit Precision\r\n Subj MR MPQA\r\n 10.0 25 12.5\r\n 7.5 20 10.0\r\n 5.0 15 7.5 CBOW\r\n % Disagreement % Disagreement10 % Disagreement5.0GloVe\r\n 2.5 MC\r\n 25 50 Dimension100200400800 25 50 Dimension100200400800 25 50 Dimension100200400800\r\n (b) 1-bit Precision\r\n Figure 4. The effect of embedding dimension on the downstream instability of sentiment analysis tasks for CBOW, GloVe, and MC. We\r\n showtheresults at two different precisions: (top) 32-bit precision (uncompressed), and (bottom) 1-bit precision (32\u00d7 compressed).\r\n D.2 Quality Tradeoffs\r\n Wealsoevaluate the quality-memory tradeoffs and quality-stability tradeoffs for CBOW and MC embedding algorithms, \ufb01nding that like\r\n stability, the quality also increases with the embedding memory. In Figures 7 (a) and 8 (a), we show the quality-memory tradeoff across\r\n sentiment analysis and NER tasks and CBOW and MC embedding algorithms for different dimension-precision combinations. We see\r\n that the dimension tends to impact the quality signi\ufb01cantly more than the precision (i.e., the change in dimension for a \ufb01xed precision\r\n affects the quality more than the change in precision for a \ufb01xed dimension affects the quality). Recall that in contrast, for instability, we\r\n Understanding the Downstream Instability of Word Embeddings\r\n Subj 20 MR MPQA\r\n 10 CBOW\r\n 6 GloVe\r\n 15 8 MC\r\n 4 10 6\r\n % Disagreement2 % Disagreement % Disagreement4\r\n 5\r\n 1 2 Precision48 16 32 1 2 Precision48 16 32 1 2 Precision48 16 32\r\n Figure 5. The effect of embedding precision on the downstream instability of sentiment analysis tasks for CBOW, GloVe, and MC\r\n embedding algorithms with 100-dimensional embeddings.\r\n CBOW, SST-2 8 CBOW, Subj 17.5 CBOW, MR CBOW, MPQA\r\n 20 12\r\n 15.0\r\n 15 6 12.5 10\r\n 4 10.0 8\r\n 10 7.5 6\r\n % Disagreement5 % Disagreement2 % Disagreement5.0 % Disagreement4\r\n 2.5 2\r\n 102 103 104 102 10 3 10 4 10 2 1 03 104 102 103 104\r\n Memory (Bits / Word) Memory (Bits / Word) Memory (Bits / Word) Memory (Bits / Word)\r\n GloVe, SST-2 GloVe, Subj GloVe, MR 14 GloVe, MPQA\r\n 20.0\r\n 17.5 10 17.5 12\r\n 15.0 8 15.0 10\r\n 12.5 6 12.5 8\r\n 10.0 4 10.0 6\r\n % Disagreement7.5 % Disagreement % Disagreement7.5 % Disagreement4\r\n 5.0 2 5.0 2\r\n 102 103 104 102 10 3 10 4 10 2 1 03 104 102 103 104\r\n Memory (Bits / Word) Memory (Bits / Word) Memory (Bits / Word) Memory (Bits / Word)\r\n 30 MC, SST-2 12 MC, Subj MC, MR 14 MC, MPQA\r\n 30 b=1\r\n 25 10 25 12 b=2\r\n b=4\r\n 20 8 20 10 b=8\r\n b=16\r\n 15 6 15 8 b=32\r\n % Disagreement10 % Disagreement4 % Disagreement % Disagreement6\r\n 10\r\n 102 103 104 2 102 10 3 10 4 10 2 1 03 104 102 103 104\r\n Memory (Bits / Word) Memory (Bits / Word) Memory (Bits / Word) Memory (Bits / Word)\r\n Figure 6. The effect of embedding dimension and precision on the downstream instability of sentiment analysis tasks for CBOW, GloVe,\r\n and MCembeddingalgorithms.\r\n sawthat the precision actually had a slightly greater effect than the dimension in Section 3.3. In Figures 7 (b) and 8 (b) we also show the\r\n quality-stability tradeoffs. For many of the sentiment analysis tasks, there is not signi\ufb01cant evidence of a strong relationship between the\r\n two; however, for the NER task, we can clearly see that as the instability increases, the quality decreases. For several of the tasks (e.g.,\r\n CBOW,MR;CBOW,MPQA),wecanseethatfordifferentprecisions(i.e.,lines), the instability changes signi\ufb01cantly, but the quality is\r\n relatively constant. This aligns with the previous observation that the precision tends to impact the instability more than it does the quality.\r\n In a similar way, for different dimensions (i.e., points), we see that the quality can change signi\ufb01cantly while the instability may stay\r\n relatively constant, especially for higher precisions (e.g., CBOW, SST-2, CBOW, MPQA).\r\n Understanding the Downstream Instability of Word Embeddings\r\n CBOW, SST-2 CBOW, Subj 77.5 CBOW, MR CBOW, MPQA\r\n 80 92 85\r\n 90 75.0\r\n 75 88 72.5 80 b=1\r\n 86 70.0 b=2\r\n 70 67.5 75 b=4\r\n Test Accuracy Test Accuracy84 Test Accuracy Test Accuracy b=8\r\n 65 65.0 b=16\r\n 82 70 b=32\r\n 102 103 104 1 02 103 10 4 102 103 104 102 1 03 104\r\n Memory (Bits / Word) Memory (Bits / Word) Memory (Bits / Word) Memory (Bits / Word)\r\n MC, SST-2 MC, Subj MC, MR 85 MC, MPQA\r\n 80 90.0 75\r\n 75 87.5 80\r\n 85.0 70 b=1\r\n b=2\r\n 70 82.5 75 b=4\r\n Test Accuracy Test Accuracy Test Accuracy65 Test Accuracy b=8\r\n 80.0 b=16\r\n 65 70 b=32\r\n 102 103 104 1 02 103 10 4 102 103 104 102 1 03 104\r\n Memory (Bits / Word) Memory (Bits / Word) Memory (Bits / Word) Memory (Bits / Word)\r\n (a) Quality-memory tradeoffs\r\n CBOW, SST-2 CBOW, Subj 77.5 CBOW, MR CBOW, MPQA\r\n 80 92 85\r\n 90 75.0\r\n 75 88 72.5 80\r\n 86 70.0\r\n Test Accuracy70 Test Accuracy Test Accuracy67.5 Test Accuracy75\r\n 65 84 65.0\r\n 82 70\r\n 5 % Disagreement1015 2 % Disagreement4 6 5 % Disagreement10 15 4 % Disagreement6810 12\r\n MC, SST-2 MC, Subj MC, MR 85 MC, MPQA\r\n 80 90.0 75 b=1\r\n b=2\r\n 87.5 b=4 80\r\n 75 70 b=8\r\n 85.0 b=16\r\n b=32 75\r\n Test Accuracy70 Test Accuracy82.5 Test Accuracy65 Test Accuracy\r\n 80.0\r\n 65 10 15 20 25 4 6 8 10 10 15 20 25 70 6 8 10 12\r\n % Disagreement % Disagreement % Disagreement % Disagreement\r\n (b) Quality-stability tradeoffs\r\n Figure 7. Quality tradeoffs for the sentiment analysis tasks with CBOW (top) and MC (bottom) embeddings for varying dimension-\r\n precision combinations.\r\n CBOW, CoNLL-2003 MC, CoNLL-2003 CBOW, CoNLL-2003 MC, CoNLL-2003\r\n 85.0 85 85.0 85 b=1\r\n b=2\r\n 82.5 82.5 b=4\r\n b=8\r\n 80.0 80 80.0 80 b=16\r\n b=1 b=32\r\n F1 Score77.5 F1 Score75 b=2 F1 Score77.5 F1 Score75\r\n b=4\r\n 75.0 b=8 75.0\r\n b=16\r\n 72.5 70 b=32 72.5 70\r\n 102 103 104 102 103 104 4 6 8 10 4 6 8 10 12\r\n Memory (Bits / Word) Memory (Bits / Word) % Disagreement % Disagreement\r\n (a) Quality-memory tradeoffs (b) Quality-stability tradeoffs\r\n Figure 8. Quality tradeoffs for the NER task with CBOW and MC embeddings for varying dimension-precision combinations.\r\n Understanding the Downstream Instability of Word Embeddings\r\n D.3 Selecting Hyperparameters for Embedding Distance Measures\r\n Theeigenspace instability measure and the measure each have a single hyperparameter to tune. For the eigenspace instability measure, \u03b1\r\n determines how important the directions of the eigenvalues of high variance are. For the measure, k determines how many neighbors are\r\n compared for each query word. To tune these hyperparameters, we compute the Spearman correlation between the embedding distance\r\n measure and the downstream prediction disagreement on the validation datasets for the \ufb01ve tasks in our study and MC and CBOW\r\n embeddingalgorithms. In Table 8a we report the average Spearman correlation for different values of \u03b1 for the eigenspace instability\r\n measure where we see \u03b1 = 3 is the top-performing value. In Table 8b we report the average Spearman correlation for different values\r\n of k for the k-NN measure, where we see k = 5 is the top-performing value. Based on these results, we use \u03b1 = 3 and k = 5 for our\r\n experiments throughout the paper.\r\n Table 8. Average Spearman correlation \u03c1 values for different values of \u03b1 for the eigenspace instability measure (a) and k for the k-NN\r\n measure (b). Top value bolded.\r\n (a) \u03b1 for the eigenspace instability measure (b) k for the k-NN measure\r\n \u03b1 \u03c1 k \u03c1\r\n 0 -0.350 1 0.766\r\n 1 -0.067 2 0.777\r\n 2 0.498 5 0.785\r\n 3 0.751 10 0.782\r\n 4 0.748 50 0.774\r\n 5 0.741 100 0.763\r\n 6 0.738 500 0.703\r\n 7 0.739 1000 0.675\r\n 8 0.739\r\n D.4 Predictive Performance of the Eigenspace Instability Measure\r\n Wenowprovideadditionalresultsvalidating the strong relationship between the eigenspace instability measure and downstream instability.\r\n In addition to the Spearman correlation results we provide in Table 1, we visualize the downstream instability v. embedding distance\r\n measure results for the CoNLL-2003 NER task in Figure 9 with CBOW and MC embeddings, taking the average over three seeds. We\r\n see that k-NN measure and the eigenspace instability measure achieve strong correlations since the lines are generally monotonically\r\n increasing for both CBOW and MC embedding algorithms.\r\n D.5 EmbeddingDistanceMeasuresforDimension-PrecisionSelection\r\n We\ufb01rstinclude Spearman correlation results and selection task results for CBOW, GloVe, and MC on the two additional downstream\r\n tasks\u2013MRandMPQA\u2013inTables9a,9b,and9c,whereweseethattheeigenspaceinstability measure and the k-NN measure continue to\r\n outperform the other measures.\r\n Wealsoevaluate the the worst-case performance of the embedding distance measures when used as a selection criterion for dimension-\r\n precision parameters. First, on the easier task of choosing the more stable dimension-precision pair out of two choices, we de\ufb01ne the\r\n worst-case performance as the maximum increase in instability that may occur by using the embedding distance measure to choose\r\n the dimension-precision parameters (rather than the ground truth choice). On the more challenging task of choosing the most stable\r\n dimension-precision pair under a memory budget, we de\ufb01ne the worst-case performance as the worst-case absolute percentage error to the\r\n oracle parameters under a given memory budget. We see in Tables 10 and 11 that the eigenspace instability measure and k-NN measure\r\n are the top-performing measures overall across both tasks.\r\n D.6 KnowledgeGraphEmbeddings\r\n In Section 6.1, we showed that as the memory of the TransE embedding increases, the instability on link prediction and triplet classi\ufb01cation\r\n task decreases; we now experiment with a modi\ufb01ed setup for the triplet classi\ufb01cation experiments. In Figure 10, we use thresholds tuned\r\n per dataset (in Figure 3 (right) we use the same threshold on both the FB15K-95 and FB15K dataset) and see that the stability-memory\r\n tradeoffs are less pronounced for higher precisions.\r\n D.7 Contextual Word Embeddings\r\n Weinclude the plots for the contextual word embedding experiments with BERT embeddings in Figures 11a and 11b for dimension and\r\n precision, respectively. As discussed in Section 6.2, although noisier than the trends with pre-trained word embeddings, we see that\r\n generally as the dimension and precision increase, the downstream instability decreases.\r\n Understanding the Downstream Instability of Word Embeddings\r\n =0.80 =0.76 = 0.53 =0.01 =0.58\r\n 10 10 b=1 b=8 10 10 10\r\n b=2 b=16\r\n b=4 b=32\r\n 8 8 8 8 8\r\n % Disagreement6 % Disagreement6 % Disagreement6 % Disagreement6 % Disagreement6\r\n 4 4 4 4 4\r\n 0.00Eigenspace Instability0.050.100.21-k-NN0.40.6Semantic Displacement0.050.100.150.2001000000Pip Loss1-Eigenspace Overlap0.20.4\r\n (a) CBOW\r\n =0.78 =0.86 = 0.16 =0.11 =0.01\r\n 12 12 b=1 b=8 12 12 12\r\n b=2 b=16\r\n 10 10 b=4 b=32 10 10 10\r\n 8 8 8 8 8\r\n % Disagreement6 % Disagreement6 % Disagreement6 % Disagreement6 % Disagreement6\r\n 4 4 4 4 4\r\n 0.00Eigenspace Instability0.010.020.251-k-NN0.500.75Semantic Displacement0.10.25000Pip Loss10000 1-Eigenspace Overlap0.20.4\r\n (b) GloVe\r\n =0.83 =0.92 =0.32 =0.44 =0.31\r\n 12 12 b=1 b=8 12 12 12\r\n b=2 b=16\r\n 10 10 b=4 b=32 10 10 10\r\n 8 8 8 8 8\r\n % Disagreement6 % Disagreement6 % Disagreement6 % Disagreement6 % Disagreement6\r\n 4 4 4 4 4\r\n Eigenspace Instability0.020.040.060.251-k-NN0.500.75Semantic Displacement0.10.20.325005000Pip Loss7500100000.21-Eigenspace Overlap0.40.6\r\n (c) Matrix completion (MC)\r\n Figure 9. Downstream instability versus embedding distance measures for the NER task on the CoNLL-2003 dataset. \u03c1 is the Spearman\r\n correlation between the embedding distance measure and the downstream instability.\r\n 40 Triplet Classification\r\n b=1\r\n 35 b=2\r\n b=4\r\n 30 b=8\r\n 25 b=16\r\n b=32\r\n % Disagreement20\r\n 15\r\n 101 102 103 104\r\n Memory (Bits / Vector)\r\n Figure 10. Stability of triplet classi\ufb01cation when evaluating embeddings when tuning the threshold for the task per dataset.\r\n E ROBUSTNESSOFTRENDS\r\n Weexplore the robustness of our study by providing preliminary investigation on the effect of subword embedding algorithms, more\r\n complex downstream models, other sources of randomness introduced by the downstream model (e.g., model initialization and sampling\r\n order), \ufb01ne-tuning embeddings on downstream instability, and the downstream model learning rate.\r\n E.1 SubwordEmbeddings\r\n We experiment with fastText (Bojanowski et al., 2017) embeddings to evaluate if the stability-memory tradeoff holds on subword\r\n embedding methods. In Figure 12, we show that we can see that overall as the memory increases, the downstream instability decreases on\r\n the SST-2 and CoNLL-2003 tasks. However, the trend with respect to dimension is weaker on the SST-2 task at larger precisions.\r\n E.2 ComplexDownstreamModels\r\n In the main text, our primary downstream models are a simple linear bag-of-words model for sentiment analysis and a single layer\r\n BiLSTMforNER.WenowdemonstratethatcomplexdownstreammodelssuchasCNNsorBiLSTM-CRFscanstilldemonstratethe\r\n stability-memory tradeoffs, such that as the memory increases, the instability decreases. In Figure 13a, we show that when using a CNN\r\n Understanding the Downstream Instability of Word Embeddings\r\n Table 9. Extended results for Section 5 on MR and MPQA downstream tasks. Top-performing values are bolded.\r\n (a) Spearmancorrelationscoresbetweenembeddingdistancemeasuresanddownstreampredictiondisagreementacrossvarying\r\n dimension-precision pairs for the embedding. Downstream models are trained for sentiment analysis (MR, MPQA) tasks.\r\n DownstreamTask MR MPQA\r\n EmbeddingAlgorithm CBOW GloVe MC CBOW GloVe MC\r\n Eigenspace Instability 0.86 0.77 0.72 0.75 0.84 0.85\r\n 1\u2212k-NN 0.85 0.86 0.74 0.77 0.92 0.94\r\n Semantic Displacement 0.63 0.10 0.59 0.68 0.10 0.29\r\n PIP Loss -0.11 0.14 0.66 -0.38 0.16 0.42\r\n 1\u2212EigenspaceOverlap 0.66 -0.06 0.56 0.68 -0.07 0.27\r\n (b) Selection error when using embedding distance measures to predict the most stable embedding dimension-precision parame-\r\n ters on downstream tasks. Downstream models are trained for sentiment analysis (MR, MPQA) tasks.\r\n DownstreamTask MR MPQA\r\n EmbeddingAlgorithm CBOW GloVe MC CBOW GloVe MC\r\n Eigenspace Instability 0.14 0.20 0.24 0.22 0.18 0.17\r\n 1\u2212k-NN 0.15 0.15 0.22 0.21 0.12 0.11\r\n Semantic Displacement 0.26 0.48 0.29 0.24 0.49 0.40\r\n PIP Loss 0.52 0.43 0.25 0.64 0.42 0.33\r\n 1\u2212EigenspaceOverlap 0.27 0.53 0.29 0.25 0.55 0.41\r\n (c) Average difference (absolute percentage) to the oracle downstream instability when using embedding distance measures as\r\n the selection criteria for dimension and precision parameters under \ufb01xed memory budgets.\r\n DownstreamTask MR MPQA\r\n EmbeddingAlgorithm CBOW GloVe MC CBOW GloVe MC\r\n Eigenspace Instability 0.69 1.35 1.03 0.34 0.87 0.44\r\n 1\u2212k-NN 0.71 1.32 0.57 0.32 0.86 0.33\r\n Semantic Displacement 0.72 2.72 2.02 0.56 2.43 1.33\r\n PIP Loss 3.33 2.03 1.96 3.24 1.40 1.05\r\n 1\u2212EigenspaceOverlap 0.84 2.72 2.02 0.51 2.43 1.25\r\n High Precision 1.55 2.72 2.07 0.40 2.43 1.49\r\n LowPrecision 3.33 2.03 4.09 3.24 1.40 0.66\r\n for the SST-2 sentiment analysis task, embeddings with very low memory budgets result in high instability. The instability quickly\r\n 2 3\r\n plateaus for memory budgets greater than 10 for the CBOW embeddings, but continues to decrease until a memory budget of 10 for the\r\n MCembeddings. TheCNNarchitecturehasoneconvolutional layer, with kernels of widths 3, 4, and 5, and 100 output channels. The\r\n convolutional layer is followed by a ReLU layer, a max-pooling layer, and \ufb01nally a linear classi\ufb01cation layer. We sweep the learning rate\r\n in a grid of {1e-5, 0.0001, 0.001, 0.01, 0.1} and choose the best learning rate by validation accuracy. Selected learning rates are shown in\r\n Table 12a and shared hyperparameters are shown in Table 12b.\r\n WenowdemonstratethattheBiLSTM-CRFalsoissubjecttothestability-memorytradeoff,whereasthedimensionandprecisionincrease,\r\n the instability decreases (Figure 13b). We use the same hyperparameters as in Table 6b and repeat our setup for the BiLSTM with the\r\n CRFturnedonfortheCoNLL-2003NERtask. DuetotheCRFbeingcomputationallyexpensive, we train a representative subset of\r\n points (dimensions in {25, 100, 800} and precisions in {1, 4, 32}). For each embedding algorithm, we grid search the learning rate for\r\n the BiLSTM-CRFwith400-dimensional embeddings in {0.001, 0.01, 0.1, 1.0, 10.0} and \ufb01nd that a learning rate of 0.1 is best for both\r\n embedding algorithms.\r\n E.3 Sources of Randomness Downstream\r\n Westudytheimpactoftwosourcesofrandomnessinthedownstreammodeltraining\u2014themodelinitialization seed and the sampling\r\n seed\u2014onthedownstreaminstability. First, we \ufb01x the embeddings and vary the model initialization seed and sampling seed independently.\r\n Wevarythesamplingorder by shuf\ufb02ing the order of the batches in the training dataset. We compare the instability from these sources of\r\n randomness in the downstream model training to the instability from the embeddings. For each source of randomness downstream, we \ufb01x\r\n the embedding (using a single seed of the Wiki\u201917, full-precision, 400-dimensional embedding), and measure the instability between\r\n Understanding the Downstream Instability of Word Embeddings\r\n Table 10. Worst-case absolute percentage error when using each embedding distance measure to predict the most stable embedding\r\n parameters on downstream tasks over of all pairs of parameters. Downstream models are trained for sentiment (SST-2, Subj) and NER\r\n (CoNLL-2003)tasks. Lowest errors are bolded.\r\n DownstreamTask SST-2 Subj CoNLL-2003\r\n EmbeddingAlgorithm CBOW GloVe MC CBOW GloVe MC CBOW GloVe MC\r\n Eigenspace Instability 10.43 6.48 13.18 3.50 3.00 3.40 3.30 4.04 4.11\r\n 1\u2212k-NN 10.43 4.78 11.75 2.80 3.00 3.40 2.17 2.39 3.16\r\n Semantic Displacement 11.70 9.23 16.80 5.40 6.00 7.10 5.13 8.05 7.03\r\n PIP Loss 16.14 14.61 15.76 6.40 9.10 4.40 6.78 9.69 5.77\r\n 1\u2212EigenspaceOverlap 12.69 11.92 16.80 5.50 8.50 7.10 5.86 9.23 7.03\r\n Table 11. Worst-case absolute percentage error to the oracle downstream instability when using embedding distance measures as the\r\n selection criteria for dimension and precision parameters. Smallest errors are bolded.\r\n DownstreamTask SST-2 Subj CoNLL-2003\r\n EmbeddingAlgorithm CBOW GloVe MC CBOW GloVe MC CBOW GloVe MC\r\n Eigenspace Instability 3.08 4.78 11.37 1.80 2.50 2.40 0.84 1.96 1.73\r\n 1\u2212k-NN 3.02 4.78 11.37 1.80 2.50 2.60 1.29 1.96 1.01\r\n Semantic Displacement 2.47 5.82 13.95 1.80 2.70 3.10 1.73 3.07 3.92\r\n PIP Loss 7.96 6.37 13.95 3.30 3.10 3.10 2.02 2.00 3.92\r\n 1\u2212EigenspaceOverlap 10.43 5.82 13.95 1.80 2.70 3.10 1.29 3.07 3.92\r\n High Precision 10.43 5.82 13.95 1.80 2.70 3.10 2.03 3.07 3.92\r\n LowPrecision 7.96 6.37 5.33 3.30 3.10 4.60 2.02 2.00 2.33\r\n 16 SST-2 Subj MR 13 MPQA\r\n 17\r\n 6 16 12\r\n 14 11\r\n 5 15\r\n % Disagreement12 % Disagreement4 % Disagreement14 % Disagreement10\r\n 9\r\n 192 384Dimension76815363072192384Dimension76815363072192384Dimension76815363072192384Dimension76815363072\r\n (a) Effect of Dimension\r\n 14.0 SST-2 6 Subj MR MPQA\r\n 17 12\r\n 13.5 5 16\r\n 13.0 15 11\r\n % Disagreement12.5 % Disagreement4 % Disagreement14 % Disagreement10\r\n 12.01 2 4 8 16 32 1 2 4 8 16 32 131 2 4 8 16 32 9 1 2 4 8 16 32\r\n Precision Precision Precision Precision\r\n (b) Effect of Precision\r\n Figure 11. Downstream instability of BERT embeddings on sentiment analysis with varying (a) output dimension, and (b) precision.\r\n models trained with different random seeds. We repeat over three pairs of models and report the average. We see in Table 13 that across\r\n four sentiment analysis tasks using the linear bag-of-words models, the sampling order seed introduces comparable instability to the\r\n change in embedding training data with full-precision, 400-dimensional embeddings, while the model initialization seed often contributes\r\n less instability. We note that using smaller memory budgets for the embeddings introduces much greater instability from the change in\r\n embedding training data, however, as shown in Figure 6.\r\n In our experiments, we had also \ufb01xed the model initialization seeds and sampling order seeds to match that of the embedding, such that\r\n the seeds were the same between any two models we compared. We now remove this constraint, and vary the model initialization and\r\n sampling order seed of the model corresponding to the Wiki\u201918 embedding, such that no two models compared have the same seeds and\r\n otherwise repeat the experimental described in Section 3 and Appendix C.3. In Figure 14a, we see that the stability-memory tradeoffs\r\n continue to hold and the trends are very similar to when we \ufb01xed the seeds (Figure 2). We note that many of the instability values\r\n themselves, particularly for CBOW, are slightly higher in Figure 14a than they are in Figure 2, likely due to the additional instability from\r\n the change in downstream model initialization and sampling seeds.\r\n Understanding the Downstream Instability of Word Embeddings\r\n FT-SG, SST-2 FT-SG, CoNLL-2003\r\n 16 12 b=1\r\n b=2\r\n 14 10 b=4\r\n b=8\r\n 12 b=16\r\n 8 b=32\r\n % Disagreement10 % Disagreement6\r\n 8 4\r\n 6 102 1 03 104 102 10 3 104\r\n Memory (Bits / Word) Memory (Bits / Word)\r\n Figure 12. Downstream instability of fastText skipgram embeddings on SST-2 and CoNLL-2003 tasks.\r\n CNN, CBOW, SST-2 CNN, MC, SST-2 BiLSTM-CRF, CBOW, CoNLL-2003 BiLSTM-CRF, MC, CoNLL-2003\r\n 24 b=1 9 12 b=1\r\n 16 22 b=2 b=4\r\n 20 b=4 8 10 b=32\r\n 14 b=8\r\n 18 b=16 7 8\r\n 12 16 b=32\r\n % Disagreement % Disagreement14 % Disagreement6 % Disagreement6\r\n 10 12\r\n 10 5 4\r\n 8\r\n 102 103 104 102 103 104 102 103 104 102 103 104\r\n Memory (Bits / Word) Memory (Bits / Word) Memory (Bits / Word) Memory (Bits / Word)\r\n (a) SST-2 task with a CNN. (b) NERtaskwithaBiLSTM-CRF.\r\n Figure 13. Downstream instability of CBOW and MC embeddings on more complex downstream models.\r\n Table 12. (a) shows selected learning rates for the CNN architecture for sentiment analysis task on the SST-2 dataset per embedding\r\n algorithm. (b) shows training hyperparameters for the CNN architecture for sentiment analysis.\r\n (b) Shared training hyperparameters for the CNN.\r\n (a) Tuned learning rates.\r\n Hyperparameter Value\r\n CBOW MC Optimizer Adam\r\n 0.0001 0.001 Batch size 32\r\n Training epochs 100\r\n Dropout 0.5\r\n Table 13. Using \ufb01xed 400-dimensional Wiki\u201917 embeddings and a linear bag-of-words model for sentiment analysis, we vary the model\r\n initialization seed and sampling order seed to measure their effect on downstream instability, compared to changing the embedding training\r\n data. Values represent the average percentage disagreement between models, and the largest instability is bolded for each embedding\r\n algorithm and task combination.\r\n DownstreamTask SST-2 MR Subj MPQA\r\n EmbeddingAlgorithm CBOW MC CBOW MC CBOW MC CBOW MC\r\n ModelInitialization Seed 3.48 7.08 2.44 9.28 1.10 4.53 1.45 4.30\r\n Sampling Order Seed 8.99 5.96 5.87 10.09 0.57 6.13 5.59 1.92\r\n EmbeddingTraining Data 6.59 8.66 4.00 11.22 1.50 3.40 3.30 4.78\r\n E.4 Effect of Fine-tuning Embeddings Downstream\r\n Westudytheimpactof\ufb01ne-tuning the embeddings downstream and \ufb01nd that the stability-memory tradeoff becomes noisier, but continues\r\n to hold under \ufb01ne-tuning, and \ufb01ne-tuning can dramatically help to decrease the downstream instability. In Figure 14b, we show that as the\r\n memoryincreases, the instability generally decreases for both CBOW and MC embeddings, even when we allow the embeddings to be\r\n updated (i.e., \ufb01ne-tuned) when training the downstream models. We note that we do not compress the embeddings during training in these\r\n experiments, therefore the memory denotes the memory required to store the embedding prior to training. To perform the \ufb01ne-tuning\r\n experiments, we follow the procedure described in Appendix C.3, and perform an additional learning rate sweep per embedding algorithm\r\n with \ufb01ne-tuning in the grid {1e-5, 0.0001, 0.001, 0.01, 0.1, 1, 10}. We found the optimal learning rate for both algorithms on the SST-2\r\n sentiment analysis task with \ufb01ne-tuning to be 0.0001. We also see that overall the instability decreases with \ufb01ne-tuning compared to \ufb01xing\r\n Understanding the Downstream Instability of Word Embeddings\r\n the embeddings (as we did in Figure 2). We note that the learning rate for the downstream model with MC and \ufb01ne-tuning is smaller than\r\n with \ufb01xed embeddings, which may also contribute to the reduced instability; however, from Figure 15, the reduction in instability with\r\n \ufb01ne-tuning still appears greater than that which can be achieved from a small change in learning rate alone.\r\n 20 CBOW, SST-2 MC, SST-2 10 CBOW, SST-2 MC, SST-2\r\n 18 b=1 6 b=1\r\n 25 b=2 8 b=2\r\n 16 b=4 5 b=4\r\n 14 b=8 b=8\r\n 20 b=16 6 4 b=16\r\n 12 b=32 b=32\r\n 10 15 3\r\n % Disagreement % Disagreement % Disagreement4 % Disagreement\r\n 8 2\r\n 6 10 2\r\n 1\r\n 102 103 104 102 103 104 102 103 104 102 103 104\r\n Memory (Bits / Word) Memory (Bits / Word) Memory (Bits / Word) Memory (Bits / Word)\r\n (a) Relaxed seed constraint. (b) Fine-tuned embeddings.\r\n Figure 14. Downstream instability of the SST-2 sentiment analysis task for different CBOW and MC embedding dimension-precision\r\n combinations when (a) we relax the constraint of having the model and sampling order seed be the same between models, and (b)\r\n embeddings are \ufb01ne-tuned. The memory indicates the embedding memory prior to training the downstream model, and embeddings are\r\n full-precision during downstream model training.\r\n E.5 Effect of Downstream Learning Rate\r\n Wenowstudytheimpactofthedownstreammodellearningrateontheinstability, showing that the learning rate of the downstream model\r\n is another factor that impacts the downstream instability. In Figure 15, we show the instability of CBOW and MC embeddings on the\r\n SST-2andMRsentimentanalysistaskswhendifferent learning rates are used for the downstream linear model. We mark the optimal\r\n learning rate by validation accuracy with a red star. We see that very small learning rates and very large learning rates tend to be the most\r\n unstable for both 100 and 400-dimensional embeddings. Moreover, the optimal learning rates do not signi\ufb01cantly increase the instability\r\n compared to the other learning rates in our sweep. Since we see that the learning rate further contributes to the instability, we \ufb01x the\r\n learning rate over different precisions and dimensions in our main study to have a controlled setting to study the impact of dimension and\r\n precision on instability.\r\n CBOW, SST-2 CBOW, MR\r\n 25 dim=100 50 dim=100\r\n 20 dim=400 dim=400\r\n 40\r\n 15 30\r\n 10 20\r\n % Disagreement5 % Disagreement10\r\n 0 105 10 3 10 1 10 5 10 3 10 1\r\n Learning Rate Learning Rate\r\n MC, SST-2 MC, MR\r\n 16 dim=100\r\n 14 60 dim=400\r\n 12 50\r\n 10 40\r\n 8 30\r\n % Disagreement6 dim=100 % Disagreement20\r\n 4 dim=400 10\r\n 105 10 3 10 1 10 5 10 3 10 1\r\n Learning Rate Learning Rate\r\n Figure 15. Downstream instability of CBOW and MC on the SST-2 and MR sentiment analysis tasks with various learning rates.\r\n", "award": [], "sourceid": 104, "authors": [{"given_name": "Megan", "family_name": "Leszczynski", "institution": "Stanford University"}, {"given_name": "Avner", "family_name": "May", "institution": "Stanford University"}, {"given_name": "Jian", "family_name": "Zhang", "institution": "SambaNova Systems"}, {"given_name": "Sen", "family_name": "Wu", "institution": "Stanford University"}, {"given_name": "Christopher", "family_name": "Aberger", "institution": "SambaNova Systems and Stanford University"}, {"given_name": "Christopher", "family_name": "Re", "institution": "Stanford University"}]}