{"title": "What is the State of Neural Network Pruning?", "book": "Proceedings of Machine Learning and Systems", "page_first": 129, "page_last": 146, "abstract": "Neural network pruning---the task of reducing the size of a network by removing parameters---has been the subject of a great deal of work in recent years. We provide a meta-analysis of the literature, including an overview of approaches to pruning and consistent findings in the literature. After aggregating results across 81 papers and pruning hundreds of models in controlled conditions, our clearest finding is that the community suffers from a lack of standardized benchmarks and metrics.\nThis deficiency is substantial enough that it is hard to compare pruning techniques to one another or determine how much progress the field has made over the past three decades.\nTo address this situation, we identify issues with current practices, suggest concrete remedies, and introduce ShrinkBench, an open-source framework to facilitate standardized evaluations of pruning methods.", "full_text": " WHATISTHESTATEOFNEURALNETWORKPRUNING?\r\n Davis Blalock*1 Jose Javier Gonzalez Ortiz*1 Jonathan Frankle1 John Guttag1\r\n ABSTRACT\r\n Neural network pruning\u2014the task of reducing the size of a network by removing parameters\u2014has been the\r\n subject of a great deal of work in recent years. We provide a meta-analysis of the literature, including an overview\r\n of approaches to pruning and consistent \ufb01ndings in the literature. After aggregating results across 81 papers\r\n and pruning hundreds of models in controlled conditions, our clearest \ufb01nding is that the community suffers\r\n from a lack of standardized benchmarks and metrics. This de\ufb01ciency is substantial enough that it is hard to\r\n compare pruning techniques to one another or determine how much progress the \ufb01eld has made over the past\r\n three decades. To address this situation, we identify issues with current practices, suggest concrete remedies, and\r\n introduce ShrinkBench, anopen-sourceframeworktofacilitatestandardizedevaluationsofpruningmethods. We\r\n use ShrinkBench to compare various pruning techniques and show that its comprehensive evaluation can prevent\r\n commonpitfalls when comparing pruning methods.\r\n 1 INTRODUCTION networks without reducing accuracy, and many pruning\r\n Much of the progress in machine learning in the past methods outperform random pruning. However, our cen-\r\n decade has been a result of deep neural networks. Many tral \ufb01nding is that the state of the literature is such that our\r\n of these networks, particularly those that perform the best motivating questions are impossible to answer. Few papers\r\n (Huangetal., 2018), require enormous amounts of compu- comparetooneanother,andmethodologiesaresoinconsis-\r\n tation and memory. These requirements not only increase tent between papers that we could not make these compar-\r\n infrastructure costs, but also make deployment of net- isons ourselves. For example, a quarter of papers compare\r\n works to resource-constrained environments such as mo- to no other pruning method, half of papers compare to at\r\n bile phones or smart devices challenging (Han et al., 2015; most one other method, and dozens of methods have never\r\n Sze et al., 2017; Yang et al., 2017). been compared to by any subsequent work. In addition,\r\n no dataset/network pair appears in even a third of papers,\r\n Onepopular approach for reducing these resource require- evaluation metrics differ widely, and hyperparameters and\r\n ments at test time is neural network pruning, which entails other counfounders vary or are left unspeci\ufb01ed.\r\n systematically removing parameters from an existing net- Most of these issues stem from the absence of standard\r\n work. Typically, the initial network is large and accurate, datasets, networks, metrics, and experimental practices. To\r\n and the goal is to produce a smaller network with simi- helpenablemorecomparablepruningresearch,weidentify\r\n lar accuracy. Pruning has been used since the late 1980s speci\ufb01c impediments and pitfalls, recommend best prac-\r\n (Janowsky, 1989; Mozer & Smolensky, 1989a;b; Karnin, tices, and introduce ShrinkBench, a library for standard-\r\n 1990), but has seen an explosion of interest in the past ized evaluation of pruning. ShrinkBench makes it easy to\r\n decade thanks to the rise of deep neural networks. adhere to the best practices we identify, largely by provid-\r\n For this study, we surveyed 81 recent papers on pruning ingastandardizedcollectionofpruningprimitives,models,\r\n in the hopes of extracting practical lessons for the broader datasets, and training routines.\r\n community. For example: which technique achieves the Ourcontributions are as follows:\r\n best accuracy/ef\ufb01ciency tradeoff? Are there strategies that\r\n work best on speci\ufb01c architectures or datasets? Which 1. A meta-analysis of the neural network pruning litera-\r\n high-level design choices are most effective? ture based on comprehensively aggregating reported re-\r\n Thereareindeedseveralconsistent results: pruning param- sults from 81 papers.\r\n eters based on their magnitudes substantially compresses 2. A catalog of problems in the literature and best prac-\r\n tices for avoiding them. These insights derive from an-\r\n *Equal contribution 1MIT CSAIL, Cambridge, MA, USA. alyzing existing work and pruning hundreds of models.\r\n Correspondence to: Davis Blalock . 3. ShrinkBench, an open-source library for evaluating\r\n Proceedings of the 3rd MLSys Conference, Austin, TX, USA, neural network pruning methods available at\r\n 2020. Copyright 2020 by the author(s). https://github.com/jjgo/shrinkbench.\r\n WhatistheStateofNeuralNetworkPruning?\r\n 2 OVERVIEWOFPRUNING Algorithm 1 Pruning and Fine-Tuning\r\n Before proceeding, we \ufb01rst offer some background on neu- Input: N, the number of iterations of pruning, and\r\n ral network pruning and a high-level overview of how ex- X,thedataset on which to train and \ufb01ne-tune\r\n isting pruning methods typically work. 1: W \u2190initialize()\r\n 2: W \u2190trainToConvergence(f(X;W))\r\n |W|\r\n 2.1 De\ufb01nitions 3: M \u21901\r\n 4: for i in 1 to N do\r\n Wede\ufb01neaneuralnetworkarchitecture as a function fam- 5: M\u2190prune(M,score(W))\r\n ily f(x;\u00b7). The architecture consists of the con\ufb01guration of 6: W\u2190fineTune(f(X;M\u2299W))\r\n the network\u2019s parameters and the sets of operations it uses 7: end for\r\n to produce outputs from inputs, including the arrangement 8: return M,W\r\n of parameters into convolutions, activation functions, pool- network, which\u2014although smaller in terms of parameter-\r\n ing, batch normalization, etc. Example architectures in- count\u2014may not be arranged in a fashion conducive to\r\n clude AlexNet and ResNet-56. We de\ufb01ne a neural network speedups using modern libraries and hardware. Other\r\n model as a particular parameterization of an architecture, methods consider parameters in groups (structured prun-\r\n i.e., f(x;W) for speci\ufb01c parameters W. Neural network ing), removing entire neurons, \ufb01lters, or channels to ex-\r\n pruning entails taking as input a model f(x;W) and pro- ploit hardware and software optimized for dense computa-\r\n ducing a new model f(x;M \u2299 W\u2032). Here W\u2032 is set of\r\n |W\u2032| tion (Li et al., 2016; He et al., 2017).\r\n parameters that may be different from W, M \u2208 {0,1}\r\n is a binary mask that \ufb01xes certain parameters to 0, and \u2299 is Scoring. It is common to score parameters based on their\r\n the elementwise product operator. In practice, rather than absolute values, trained importance coef\ufb01cients, or contri-\r\n using an explicit mask, pruned parameters of W are \ufb01xed butions to network activations or gradients. Some prun-\r\n to zero or removed entirely. ing methods compare scores locally, pruning a fraction of\r\n the parameters with the lowest scores within each struc-\r\n 2.2 High-Level Algorithm tural subcomponentofthenetwork(e.g.,layers)(Hanetal.,\r\n There are many methods of producing a pruned model 2015). Others consider scores globally, comparing scores\r\n f(x;M\u2299W\u2032)fromaninitiallyuntrainedmodelf(x;W ), to one another irrespective of the part of the network in\r\n 0 which the parameter resides (Lee et al., 2019b; Frankle &\r\n where W0 is sampled from an initialization distribution D. Carbin, 2019).\r\n Nearly all neural network pruning strategies in our survey\r\n derive from Algorithm 1 (Han et al., 2015). In this algo- Scheduling. Pruning methods differ in the amount of the\r\n rithm, the network is \ufb01rst trained to convergence. After- network to prune at each step. Some methods prune all\r\n wards, each parameter or structural element in the network desired weights at once in a single step (Liu et al., 2019).\r\n is issued a score, and the network is pruned based on these Othersprunea\ufb01xedfractionofthenetworkiterativelyover\r\n scores. Pruning reduces the accuracy of the network, so several steps (Han et al., 2015) or vary the rate of pruning\r\n it is trained further (known as \ufb01ne-tuning) to recover. The according to a more complex function (Gale et al., 2019).\r\n process of pruning and \ufb01ne-tuning is often iterated several Fine-tuning. For methods that involve \ufb01ne-tuning, it is\r\n times, gradually reducing the network\u2019s size. most common to continue to train the network using the\r\n Many papers propose slight variations of this algorithm. trained weights from before pruning. Alternative propos-\r\n For example, some papers prune periodically during train- als include rewinding the network to an earlier state (Fran-\r\n ing (Gale et al., 2019) or even at initialization (Lee et al., kle et al., 2019) and reinitializing the network entirely (Liu\r\n 2019b). Others modify the network to explicitly include et al., 2019).\r\n additional parameters that encourage sparsity and serve as\r\n a basis for scoring the network after training (Molchanov 2.4 Evaluating Pruning\r\n et al., 2017). Pruningcanaccomplishmanydifferentgoals,includingre-\r\n 2.3 Differences Betweeen Pruning Methods ducingthestoragefootprintoftheneuralnetwork,thecom-\r\n putational cost of inference, the energy requirements of in-\r\n Within the framework of Algorithm 1, pruning methods ference, etc. Each of these goals favors different design\r\n vary primarily in their choices regarding sparsity structure, choices and requires different evaluation metrics. For ex-\r\n scoring, scheduling, and \ufb01ne-tuning. ample, when reducing the storage footprint of the network,\r\n Structure. Some methods prune individual parameters all parameters can be treated equally, meaning one should\r\n (unstructured pruning). Doing so produces a sparse neural evaluate the overall compression ratio achieved by prun-\r\n ing. However, when reducing the computational cost of\r\n WhatistheStateofNeuralNetworkPruning?\r\n inference, different parameters may have different impacts. racy. In fact, for small amounts of compression, pruning\r\n For instance, in convolutional layers, \ufb01lters applied to spa- can sometimes increase accuracy (Han et al., 2015; Suzuki\r\n tially larger inputs are associated with more computation et al., 2018). This basic \ufb01nding has been replicated in a\r\n than those applied to smaller inputs. large fraction of the papers in our corpus.\r\n Regardless of the goal, pruning imposes a tradeoff between Along the same lines, it has been repeatedly shown that, at\r\n model ef\ufb01ciency and quality, with pruning increasing the least for large amounts of pruning, many pruning methods\r\n former while (typically) decreasing the latter. This means outperform random pruning (Yu et al., 2018; Gale et al.,\r\n that a pruning method is best characterized not by a single 2019; Frankle et al., 2019; Mariet & Sra, 2015; Suau et al.,\r\n modelithaspruned,butbyafamilyofmodelscorrespond- 2018; He et al., 2017). Interestingly, this does not always\r\n ing to different points on the ef\ufb01ciency-quality curve. To hold for small amounts of pruning (Morcos et al., 2019).\r\n quantify ef\ufb01ciency, most papers report at least one of two Similarly, pruning all layers uniformly tends to perform\r\n metrics. The \ufb01rst is the number of multiply-adds (often worse than intelligently allocating parameters to different\r\n referred to as FLOPs) required to perform inference with layers (Gale et al., 2019; Han et al., 2015; Li et al., 2016;\r\n the pruned network. The second is the fraction of param- Molchanov et al., 2016; Luo et al., 2017) or pruning glob-\r\n eters pruned. To measure quality, nearly all papers report ally (Lee et al., 2019b; Frankle & Carbin, 2019). Lastly,\r\n changes in Top-1 or Top-5 image classi\ufb01cation accuracy. whenholdingthenumberof\ufb01ne-tuningiterationsconstant,\r\n Asothershavenoted(Lebedevetal.,2014;Figurnovetal., manymethods produce pruned models that outperform re-\r\n 2016; Louizos et al., 2017; Yang et al., 2017; Han et al., training from scratch with the same sparsity pattern (Zhang\r\n 2015; Kim et al., 2015; Wen et al., 2016; Luo et al., 2017; et al., 2015; Yu et al., 2018; Louizos et al., 2017; He et al.,\r\n Heetal.,2018b),thesemetricsarefarfromperfect. Param- 2017; Luo et al., 2017; Frankle & Carbin, 2019) (at least\r\n eter and FLOP counts are a loose proxy for real-world la- withalargeenoughamountofpruning(Suauetal.,2018)).\r\n tency, throughout, memory usage, and power consumption. Retraining from scratch in this context means training a\r\n Similarly, image classi\ufb01cation is only one of the countless fresh, randomly-initialized model with all weights clamped\r\n tasks to which neural networks have been applied. How- to zero throughout training, except those that are nonzero\r\n ever, because the overwhelming majority of papers in our in the pruned model.\r\n corpus focus on these metrics, our meta-analysis necessar- Another consistent \ufb01nding is that sparse models tend to\r\n ily does as well. outperform dense ones for a \ufb01xed number of parameters.\r\n 3 LESSONS FROM THE LITERATURE Lee et al. (2019a) show that increasing the nominal size\r\n of ResNet-20 on CIFAR-10 while sparsifying to hold the\r\n After aggregating results from a corpus of 81 papers, we number of parameters constant decreases the error rate.\r\n identi\ufb01ed a number of consistent \ufb01ndings. In this section, Kalchbrenner et al. (2018) obtain a similar result for audio\r\n we provide an overview of our corpus and then discuss synthesis, as do Grayetal.(2017)foravarietyofadditional\r\n these \ufb01ndings. tasks across various domains. Perhaps most compelling of\r\n all are the many results, including in Figure 1, showing that\r\n 3.1 Papers Used in Our Analysis pruned models can obtain higher accuracies than the origi-\r\n nal modelsfromwhichtheyarederived. Thisdemonstrates\r\n Our corpus consists of 79 pruning papers published since that sparse models can not only outperform dense counter-\r\n 2010 and two classic papers (LeCun et al., 1990; Hassibi parts with the same number of parameters, but sometimes\r\n et al., 1993) that have been compared to by a number of dense models with even more parameters.\r\n recent methods. We selected these papers by identifying\r\n popular papers in the literature and what cites them, sys- 3.3 Pruning vs Architecture Changes\r\n tematically searching through conference proceedings, and\r\n tracing the directed graph of comparisons between prun- One current unknown about pruning is how effective it\r\n ing papers. This last procedure results in the property that, tends to be relative to simply using a more ef\ufb01cient archi-\r\n barring oversights on our part, there is no pruning paper tecture. These options are not mutually exclusive, but it\r\n in our corpus that compares to any pruning paper outside may be useful in guiding one\u2019s research or development\r\n of our corpus. Additional details about our corpus and its efforts to know which choice is likely to have the larger\r\n construction can be found in Appendix A. impact. Along similar lines, it is unclear how pruned mod-\r\n els from different architectures compare to one another\u2014\r\n 3.2 HowEffective is Pruning? i.e., to what extent does pruning offer similar bene\ufb01ts\r\n Oneofthe clearest \ufb01ndings about pruning is that it works. across architectures? To address these questions, we plot-\r\n More precisely, there are various methods that can sig- tedthereportedaccuraciesandcompression/speeduplevels\r\n ni\ufb01cantly compress models with little or no loss of accu- of pruned models on ImageNet alongside the same metrics\r\n WhatistheStateofNeuralNetworkPruning?\r\n 1 Speed and Size Tradeoffs for Original and Pruned Models\r\n for different architectures with no pruning (Figure 1). We\r\n 2 85\r\n plot results within a family of models as a single curve.\r\n Figure 1 suggests several conclusions. First, it reinforces 80\r\n the conclusion that pruning can improve the time or space\r\n vs accuracy tradeoff of a given architecture, sometimes 75\r\n evenincreasingtheaccuracy. Second,itsuggeststhatprun- 70\r\n ing generally does not help as much as switching to a better Top 1 Accuracy (%)\r\n architecture. Finally, it suggests that pruning is more effec- 65\r\n tive for architectures that are less ef\ufb01cient to begin with.\r\n 96\r\n 4 MISSING CONTROLLED COMPARISONS 94\r\n 92\r\n While there do appear to be a few general and consistent 90\r\n \ufb01ndings in the pruning literature (see the previous section), 88\r\n by far the clearest takeaway is that pruning papers rarely Top 5 Accuracy (%)86\r\n make direct and controlled comparisons to existing meth- 84\r\n ods. This lack of comparisons stems largely from a lack 106 107 108 109 1010\r\n of experimental standardization and the resulting fragmen- Number of Parameters Number of FLOPs\r\n tation in reported results. This fragmentation makes it dif- MobileNet-v2 (2018) ResNet (2016) VGG (2014) EfficientNet (2019)\r\n MobileNet-v2 Pruned ResNet Pruned VGG Pruned\r\n \ufb01cult for even the most committed authors to compare to Figure 1: Size and speed vs accuracy tradeoffs for dif-\r\n morethanafewexisting methods. ferent pruning methods and families of architectures.\r\n 4.1 OmissionofComparison Pruned models sometimes outperform the original ar-\r\n chitecture, but rarely outperform a better architecture.\r\n Many papers claim to advance the state of the art, but Ignoring Recent Methods Even when considering only\r\n don\u2019t compare to other methods\u2014including many pub- post-2010 approaches, there are still virtually no methods\r\n lished ones\u2014that make the same claim. that have been shown to outperform all existing \u201cstate-of-\r\n Ignoring Pre-2010s Methods There was already a rich the-art\u201d methods. This follows from the fact, depicted in\r\n body of work on neural network pruning by the mid 1990s the top plot of Figure 2, that there are dozens of modern\r\n (see, e.g., Reed\u2019s survey (Reed, 1993)), which has been al- papers\u2014including many af\ufb01rmed through peer review\u2014\r\n mostcompletelyignoredexceptforLecun\u2019sOptimalBrain that have never been compared to by any later study.\r\n Damage(LeCunet al., 1990) and Hassibi\u2019s Optimal Brain A related problem is that papers tend to compare to few\r\n Surgeon (Hassibi et al., 1993). Indeed, multiple authors existing methods. In the lower plot of Figure 2, we see\r\n haverediscoveredexistingmethodsoraspectsthereof,with that more than a fourth of our corpus does not compare\r\n Hanetal. (2015) reintroducing the magnitude-based prun- to any previously proposed pruning method, and another\r\n ing of Janowsky (1989), Lee et al. (2019b) reintroducing fourth compares to only one. Nearly all papers compare to\r\n the saliency heuristic of Mozer & Smolensky (1989a), and three or fewer. This might be adequate if there were a clear\r\n He et al. (2018a) reintroducing the practice of \u201creviving\u201d progression of methods with one or two \u201cbest\u201d methods at\r\n previously pruned weights described in Tresp et al. (1997). any given time, but this is not the case.\r\n 1Since manypruningpapersreportonlychangeinaccuracyor 4.2 Dataset and Architecture Fragmentation\r\n amount of pruning, without giving baseline numbers, we normal- Among81papers, we found results using 49 datasets, 132\r\n ize all pruning results to have accuracies and model sizes/FLOPs architectures, and 195 (dataset, architecture) combinations.\r\n as if they had begun with the same model. Concretely, this means As shown in Table 1, even the most common combination\r\n multiplying the reported fraction of pruned size/FLOPs by a stan- 3\r\n dardizedinitial value. This value is set to the median initial size or of dataset and architecture\u2014VGG-16 on ImageNet (Deng\r\n number of FLOPs reported for that architecture across all papers. et al., 2009)\u2014is used in only 22 out of 81 papers. More-\r\n Thisnormalizationschemeisnotperfect,butdoeshelpcontrolfor over, three of the top six most common combinations in-\r\n different methods beginning with different baseline accuracies. volve MNIST (LeCun et al., 1998a). As Gale et al. (2019)\r\n 2The Ef\ufb01cientNet family is given explicitly in the original pa-\r\n per (Tan & Le, 2019), the ResNet family consists of ResNet- andothers have argued, using larger datasets and models is\r\n 18, ResNet-34, ResNet-50, etc., and the VGG family consists of essential whenassessinghowwellamethodworksforreal-\r\n VGG-{11, 13, 16, 19}. There are no pruned Ef\ufb01cientNets since 3We adopt the common practice of referring to the\r\n Ef\ufb01cientNet was published too recently. Results for non-pruned ILSVRC2012training and validation sets as \u201cImageNet.\u201d\r\n modelsaretakenfrom(Tan&Le,2019)and(Biancoetal.,2018).\r\n WhatistheStateofNeuralNetworkPruning?\r\n Number of Papers Comparing to a Given Paper NumberofPapers\r\n 32 (Dataset, Architecture) Pair using Pair\r\n 28 ImageNet VGG-16 22\r\n 24 ImageNet ResNet-50 15\r\n 20\r\n 16 MNIST LeNet-5-Caffe 14\r\n 12 CIFAR-10 ResNet-56 14\r\n Number of papers8 MNIST LeNet-300-100 12\r\n 4\r\n compared to this many times0 MNIST LeNet-5 11\r\n 0 3 6 9 12 15 18 ImageNet CaffeNet 10\r\n Compared to by this many other papers\r\n Number of Papers a Given Paper Compares To CIFAR-10 CIFAR-VGG(Torch) 8\r\n 21 ImageNet AlexNet 8\r\n 18 ImageNet ResNet-18 6\r\n 15 ImageNet ResNet-34 6\r\n 12 CIFAR-10 ResNet-110 5\r\n 9 CIFAR-10 PreResNet-164 4\r\n 6 CIFAR-10 ResNet-32 4\r\n Number of papers that3\r\n compare to this many others0 Table 1: All combinations of dataset and architecture\r\n 0 2 4 6 8 10 used in at least 4 out of 81 papers.\r\n Compares to this many other papers\r\n Peer-Reviewed Other ods are nearby on the x-axis, it is not clear whether one\r\n Figure 2: Reported comparisons between papers. meaningfully outperforms another since neither reports a\r\n world networks. MNIST results may be particularly un- standarddeviationorothermeasureofcentraltendency. Fi-\r\n likely to generalize, since this dataset differs signi\ufb01cantly nally, most papers in our corpus do not report any results\r\n fromotherpopulardatasetsforimageclassi\ufb01cation. Inpar- with any of these common con\ufb01gurations.\r\n ticular, its images are grayscale, composed mostly of zeros, 4.4 Incomplete Characterization of Results\r\n andpossible to classify with over 99% accuracy using sim-\r\n ple models (LeCun et al., 1998b). If all papers reported a wide range of points in their trade-\r\n off curves across a large set of models and datasets, there\r\n 4.3 Metrics Fragmentation might be some number of direct comparisons possible be-\r\n As depicted in Figure 3, papers report a wide variety of tween any given pair of methods. As we see in the upper\r\n metrics and operating points, making it dif\ufb01cult to com- half of Figure 4, however, most papers use at most three\r\n pare results. Each column in this \ufb01gure is one (dataset, ar- (dataset, architecture) pairs; and as we see in the lower half,\r\n chitecture) combination taken from the four most common they use at most three\u2014and often just one\u2014point to char-\r\n 4 acterize each curve. Combined with the fragmentation in\r\n combinations , excluding results on MNIST. Each row is experimental choices, this means that different methods\u2019\r\n one pair of metrics. Each curve is the ef\ufb01ciency vs accu- results are rarely directly comparable. Note that the lower\r\n 5\r\n racy tradeoff obtained by one method. Methods are color- half restricts results to the four most common (dataset, ar-\r\n coded by year. chitecture) pairs.\r\n It is hard to identify any consistent trends in these plots,\r\n aside from the existence of a tradeoff between ef\ufb01ciency 4.5 ConfoundingVariables\r\n and accuracy. A given method is only present in a small Evenwhencomparisonsincludethesamedatasets,models,\r\n subset of plots. Methods from later years do not consis- metrics, and operating points, other confounding variables\r\n tently outperform methods from earlier years. Methods still make meaningful comparisons dif\ufb01cult. Some vari-\r\n within a plot are often incomparable because they report ables of particular interest include:\r\n results at different points on the x-axis. Even when meth-\r\n \u2022 Accuracy and ef\ufb01ciency of the initial model\r\n 4We combined the results for AlexNet and CaffeNet, which\r\n is a slightly modi\ufb01ed version of AlexNet (caf, 2016), since many \u2022 Data augmentation and preprocessing\r\n authors refer to the latter as \u201cAlexNet,\u201d and it is often unclear \u2022 Random variations in initialization, training, and \ufb01ne-\r\n which model was used.\r\n 5Since what counts as one method can be unclear, we consider tuning. This includes choice of optimizer, hyperparam-\r\n all results from one paper to be one method except when two or eters, and learning rate schedule.\r\n more named methods within the paper report using at least one \u2022 Pruning and \ufb01ne-tuning schedule\r\n identical x-coordinate (i.e., when the paper\u2019s results can\u2019t be plot-\r\n ted as one curve). \u2022 Deep learning library. Different libraries are known to\r\n WhatistheStateofNeuralNetworkPruning?\r\n VGG-16 on ImageNet Alex/CaffeNet on ImageNet ResNet-50 on ImageNet ResNet-56 on CIFAR-10\r\n 4 0.5\r\n 0 0\r\n 2 1 0.0\r\n 1\r\n 0 2 0.5\r\n Change in2 2 3\r\n 1.0\r\n Top-1 Accuracy (%) 3 4\r\n 4\r\n 0 1 2 3 4 1 2 3 4 0 1 2 3 4 0 2 4\r\n Log2(Compression Ratio) Log2(Compression Ratio) Log2(Compression Ratio) Log2(Compression Ratio)\r\n 0\r\n 2 0\r\n 1 1\r\n 0 2 1\r\n Change in1 3 2\r\n Top-5 Accuracy (%)2 4\r\n 3\r\n 0 1 2 3 0 2 4 0.0 0.5 1.0 1.5 2.0\r\n Log2(Compression Ratio) Log2(Compression Ratio) Log2(Compression Ratio)\r\n 4 0.0 0\r\n 0.5 1 0\r\n 2\r\n 1.0 2 1\r\n 0 1.5\r\n Change in 3 2\r\n 2 2.0\r\n Top-1 Accuracy (%) 2.5 4 3\r\n 4\r\n 2 4 6 1 2 3 2 3 1 2 3\r\n Theoretical Speedup Theoretical Speedup Theoretical Speedup Theoretical Speedup\r\n 2 0 0\r\n 0 2\r\n 4 1\r\n 2\r\n Change in4 6\r\n 2\r\n Top-5 Accuracy (%)6 8\r\n 10\r\n 2 4 6 8 10 2 4 6 2 3\r\n Theoretical Speedup Theoretical Speedup Theoretical Speedup\r\n Collins 2014 Kim 2016 Lin 2017 Dubey 2018, AP+Coreset-KPeng 2018 Choi 2019\r\n Han 2015 Srinivas 2016 Luo 2017 Dubey 2018, AP+Coreset-SSuau 2018, PFA-En Gale 2019, Magnitude-v2\r\n Zhang 2015 Wen 2016 Srinivas 2017 He, Yang 2018 Suau 2018, PFA-KL Kim 2019\r\n Figurnov 2016 Alvarez 2017 Yang 2017 He, Yang 2018, Fine-TuneSuzuki 2018 Liu 2019, Scratch-B\r\n Guo 2016 He 2017 Carreira-Perpinan 2018 He, Yihui 2018 Yamamoto 2018 Luo 2019\r\n Han 2016 He 2017, 3C Ding 2018 Huang 2018 Yu 2018 Peng 2019, CCP\r\n Hu 2016 Li 2017 Dubey 2018, AP+Coreset-A Lin 2018 Zhuang 2018 Peng 2019, CCP-AC\r\n Figure 3: Fragmentation of results. Shown are all self-reported results on the most common (dataset, architecture)\r\n combinations. Each column is one combination, each row shares an accuracy metric (y-axis), and pairs of rows\r\n share a compression metric (x-axis). Up and to the right is always better. Standard deviations are shown for He\r\n 2018 on CIFAR-10, which is the only result that provides any measure of central tendency. As suggested by the\r\n legend, only 37 out of the 81 papers in our corpus report any results using any of these con\ufb01gurations.\r\n yield different accuracies for the same architecture and both used the same code as the methods to which it com-\r\n dataset (Northcutt, 2019; Nola, 2016) and mayhavesub- pares and reports enough measurements to average out ran-\r\n tly different behaviors (Vryniotis, 2018). domvariations. This is exceptionally rare, with Gale et al.\r\n \u2022 Subtle differences in code and environment that may (2019) and Liu et al. (2019) being arguably the only ex-\r\n not be easily attributable to any of the above variations amples. Moreover, neither of these papers introduce novel\r\n (Crall, 2018; Jogeshwar, 2017; unr, 2017). pruning methods per se but are instead inquiries into the\r\n ef\ufb01cacy of existing methods.\r\n In general, it is not clear that any paper can succeed in ac- Many papers attempt to account for subsets of these con-\r\n counting for all of these confounders unless that paper has founding variables. A near universal practice in this re-\r\n WhatistheStateofNeuralNetworkPruning?\r\n Number of (Dataset, Architecture) Pairs Used Pruning ResNet-50 with Unstructured Magnitude-Based Pruning\r\n 21\r\n 18 76\r\n 15 74\r\n 12\r\n 9 72\r\n 6 70\r\n Number of papersusing this many pairs3\r\n 0 Top 1 Accuracy (%)68\r\n 2 4 6 8 10 12 14 16 18 20\r\n Number of pairs Pruning ResNet-50 with All Other Methods\r\n Number of Points used to Characterize Tradeoff Curve 76\r\n 27\r\n 24 74\r\n 21\r\n 18 72\r\n 15\r\n 12 70\r\n 9\r\n Number of curves6 Top 1 Accuracy (%)68\r\n using this many points3\r\n 0\r\n 1 2 3 4 5 6 7 8 9 106 107\r\n Number of points Number of Parameters\r\n Peer-Reviewed Other Frankle 2019, PruneAtEpoch=15 Dubey 2018, AP+Coreset-K\r\n Figure 4: Number of results reported by each paper, Frankle 2019, PruneAtEpoch=90 Dubey 2018, AP+Coreset-S\r\n Frankle 2019, ResetToEpoch=10 Gale 2019, SparseVD\r\n excluding MNIST. Top) Most papers report on three or Frankle 2019, ResetToEpoch=R Huang 2018\r\n fewer (dataset, architecture) pairs. Bottom) For each Gale 2019, Magnitude Lin 2018\r\n pair used, most papers characterize their tradeoff be- Gale 2019, Magnitude-v2 Liu 2019, Scratch-B\r\n Liu 2019, Magnitude Luo 2017\r\n tween amount of pruning and accuracy using a single Alvarez 2017 Yamamoto 2018\r\n point in the ef\ufb01ciency vs accuracy curve. In both plots, Dubey 2018, AP+Coreset-A Zhuang 2018\r\n the pattern holds even for peer-reviewed papers. Figure 5: Pruning ResNet-50 on ImageNet. Methods in\r\n gard is reporting change in accuracy relative to the original the upper plot all prune weights with the smallest mag-\r\n model,inadditiontoorinsteadofrawaccuracy. Thishelps nitudes, but differ in implementation, pruning sched-\r\n to control for the accuracy of the initial model. However, as ule, and\ufb01ne-tuning. Thevariationcausedbythesevari-\r\n wedemonstrateinSection7,thisisnotsuf\ufb01cienttoremove ables is similar to the variation across different pruning\r\n initial model as a confounder. Certain initial models can be methods, whose results are shown in the lower plot. All\r\n pruned more or less ef\ufb01ciently, in terms of the accuracy vs results are taken from the original papers.\r\n compression tradeoff. This holds true even with identical 5 FURTHERBARRIERSTOCOMPARISON\r\n pruning methods and all other variables held constant.\r\n Thereareatleasttwomoreempiricalreasonstobelievethat In the previous section, we discussed the fragmentation of\r\n confounding variables can have a signi\ufb01cant impact. First, datasets, models, metrics, operating points, and experimen-\r\n as one can observe in Figure 3, methods often introduce tal details, and how this fragmentation makes evaluating\r\n changes in accuracy of much less than 1% at reported op- the ef\ufb01cacy of individual pruning methods dif\ufb01cult. In this\r\n erating points. This means that, even if confounders have section, we argue that there are additional barriers to com-\r\n only a tiny impact on accuracy, they can still have a large paring methods that stem from common practices in how\r\n impact on which method appears better. methods and results are presented.\r\n Second, as shown in Figure 5, existing results demonstrate 5.1 Architecture Ambiguity\r\n that different training and \ufb01ne-tuning settings can yield It is often dif\ufb01cult, or even impossible, to identify the exact\r\n nearly as much variability as different methods. Specif- architecture that authors used. Perhaps the most prevalent\r\n ically, consider 1) the variability introduced by differ- example of this is when authors report using some sort of\r\n ent \ufb01ne-tuning methods for unstructured magnitude-based ResNet(Heetal.,2016a;b). Becausetherearetwodifferent\r\n pruning (Figure 6 top) and 2) the variability introduced by variations of ResNets, introduced in these two papers, say-\r\n entirely different pruning methods (Figure 6 bottom). The ing that one used a \u201cResNet-50\u201d is insuf\ufb01cient to identify a\r\n variability between \ufb01ne-tuning methods is nearly as large particular architecture. Some authors do appear to deliber-\r\n as the variability between pruning methods. ately point out the type of ResNet they use (e.g., (Liu et al.,\r\n 2017; Dong et al., 2017)). However, given that few papers\r\n WhatistheStateofNeuralNetworkPruning?\r\n even hint at the possibility of confusion, it seems unlikely times never made clear. Even when reporting FLOPs,\r\n that all authors are even aware of the ambiguity, let alone which is nominally a consistent metric, different authors\r\n that they have cited the corresponding paper in all cases. measure it differently (e.g., (Molchanov et al., 2016) vs\r\n Perhaps the greatest confusion is over VGG networks (Si- (Wang&Cheng,2016)),thoughmostoftenpapersentirely\r\n monyan&Zisserman,2014). Manypapersdescribeexper- omit their formula for computing FLOPs. We found up\r\n imenting on \u201cVGG-16,\u201d \u201cVGG,\u201dor\u201cVGGNet,\u201dsuggesting to a factor of four variation in the reported FLOPs of dif-\r\n a standard and well-known architecture. In many cases, ferent papers for the same architecture and dataset, with\r\n what is actually used is a custom variation of some VGG (Yang et al., 2017) reporting 371 MFLOPs for AlexNet on\r\n model, with removed fully-connected layers (Changpinyo ImageNet, (Choi et al., 2019) reporting 724 MFLOPs, and\r\n et al., 2017; Luo et al., 2017), smaller fully-connected lay- (Han et al., 2015) reporting 1500 MFLOPs.\r\n ers (Lee et al., 2019b), or added dropout or batchnorm (Liu 6 SUMMARYANDRECOMMENDATIONS\r\n et al., 2017; Lee et al., 2019b; Peng et al., 2018; Molchanov\r\n et al., 2017; Ding et al., 2018; Suau et al., 2018). In the previous sections, we have argued that existing work\r\n In somecases, papers simply fail to make clear what model tends to\r\n they used (even for non-VGG architectures). For exam- \u2022 makeit dif\ufb01cult to identify the exact experimental setup\r\n ple, one paper just states that their segmentation model and metrics,\r\n \u201cis composed from an inception-like network branch and a \u2022 use too few (dataset, architecture) combinations,\r\n DenseNet network branch.\u201d Another paper attributes their\r\n VGGNet to (Parkhi et al., 2015), which mentions three \u2022 report too few points in the tradeoff curve for any given\r\n VGG networks. Liu et al. (2019) and Frankle & Carbin combination, and no measures of central tendency,\r\n (2019) have circular references to one another that can no \u2022 omit comparison to many methods that might be state-\r\n longer be resolved because of simultaneous revisions. One of-the-art, and\r\n paper mentions using a \u201cVGG-S\u201d from the Caffe Model \u2022 fail to control for confounding variables.\r\n Zoo, but as of this writing, no model with this name ex- These problems often make it dif\ufb01cult or impossible to as-\r\n ists there. Perhaps the most confusing case is the Lenet- sess the relative ef\ufb01cacy of different pruning methods. To\r\n 5-Caffe reported in one 2017 paper. The authors are to enable direct comparison between methods in the future,\r\n be commended for explicitly stating not only that they use wesuggest the following practices:\r\n Lenet-5-Caffe, but their exact architecture. However, they\r\n describe an architecture with an 800-unit fully-connected \u2022 Identify the exact sets of architectures, datasets, and\r\n layer, while examination of both the Caffe .prototxt metrics used, ideally in a structured way that is not scat-\r\n \ufb01les (Jia et al., 2015a;b) and associated blog post (Jia et al., tered throughout the results section.\r\n 2016) indicates that no such layer exists in Lenet-5-Caffe. \u2022 Use at least three (dataset, architecture) pairs, including\r\n modern, large-scale ones. MNIST and toy models do\r\n 5.2 Metrics Ambiguity notcount. AlexNet,CaffeNet,andLenet-5arenolonger\r\n It can also be dif\ufb01cult to know what the reported metrics modernarchitectures.\r\n mean. For example, many papers include a metric along \u2022 For any given pruned model, report both compression\r\n the lines of \u201cPruned%\u201d. In some cases, this means frac- ratio and theoretical speedup. Compression ratio is de-\r\n tion of the parameters or FLOPs remaining (Suau et al., \ufb01ned as the original size divided by the new size. The-\r\n 2018). In other cases, it means the fraction of parameters or oretical speedup is de\ufb01ned as the original number of\r\n FLOPsremoved (Han et al., 2015; Lebedev & Lempitsky, multiply-adds divided by the new number. Note that\r\n 2016;Yaoetal.,2018). Thereisalsowidespreadmisuseof there is no reason to report only one of these metrics.\r\n the term \u201ccompression ratio,\u201d which the compression liter- \u2022 ForImageNetandothermany-classdatasets,reportboth\r\n aturehaslongusedtomean originalsize (Siedelmannetal., Top-1 and Top-5 accuracy. There is again no reason to\r\n compressed size\r\n 2015; Zukowski et al., 2006; Zhao et al., 2015; Lindstrom, report only one of these.\r\n 2014; Ratanaworabhan et al., 2006; Blalock et al., 2018), \u2022 Whatever metrics one reports for a given pruned model,\r\n but many pruning authors de\ufb01ne (usually without making also report these metrics for an appropriate control (usu-\r\n the formula explicit) as 1 \u2212 compressedsize. ally the original model before pruning).\r\n original size\r\n Reported \u201cspeedup\u201d values present similar challenges. \u2022 Plot the tradeoff curve for a given dataset and architec-\r\n These values are sometimes wall time, sometimes original ture, alongside the curves for competing methods.\r\n number of FLOPs divided by pruned number of FLOPs, \u2022 When plotting tradeoff curves, use at least 5 operating\r\n sometimes a more complex formula relating these two points spanning a range of compression ratios. The set\r\n quantities (Dong et al., 2017; He et al., 2018a), and some- of ratios {2,4,8,16,32} is a good choice.\r\n WhatistheStateofNeuralNetworkPruning?\r\n \u2022 Report and plot means and sample standard deviations, complexmethods(Hanetal.,2015;2016;Galeetal.,2019;\r\n instead of one-off measurements, whenever feasible. Frankleetal.,2019). Gradient-basedmethodsarelesscom-\r\n \u2022 Ensure that all methods being compared use identical mon,butaresimpletoimplementandhaverecentlygained\r\n libraries, data loading, and other code to the greatest ex- popularity (Lee et al., 2019b;a; Yu et al., 2018). Random\r\n tent possible. pruning is a common straw man that can serve as a useful\r\n debuggingtool. Note that these baselines are not reproduc-\r\n Wealsorecommendthatreviewersdemandamuchgreater tions of any of these methods, but merely inspired by their\r\n level of rigor when evaluating papers that claim to offer a pruning heuristics.\r\n better method of pruning neural networks.\r\n 7.3 Avoiding Pruning Pitfalls with Shrinkbench\r\n 7 SHRINKBENCH Using the described baselines, we pruned over 800 net-\r\n 7.1 OverviewofShrinkBench workswithvaryingdatasets, networks, compression ratios,\r\n initial weights and random seeds. In doing so, we identi-\r\n To make it as easy as possible for researchers to put our \ufb01ed various pitfalls associated with experimental practices\r\n suggestions into practice, we have created an open-source that are currently common in the literature but are avoided\r\n library for pruning called ShrinkBench. ShrinkBench pro- byusing ShrinkBench.\r\n videsstandardizedandextensiblefunctionalityfortraining, We highlight several noteworthy results below. For addi-\r\n pruning, \ufb01ne-tuning, computing metrics, and plotting, all tional experimental results and details, see Appendix D.\r\n using a standardized set of pretrained models and datasets. One standard deviation bars across three runs are shown\r\n ShrinkBench is based on PyTorch (Paszke et al., 2017) and for all CIFAR-10 results.\r\n is designed to allow easy evaluation of methods with ar- Metrics are not Interchangeable. As discussed previ-\r\n bitrary scoring functions, allocation of pruning across lay- ously, it is commonpracticetoreporteitherreductioninthe\r\n ers, and sparsity structures. In particular, given a callback number of parameters or in the number of FLOPs. If these\r\n de\ufb01ning how to compute masks for a model\u2019s parameter metrics are extremely correlated, reporting only one is suf-\r\n tensors at a given iteration, ShrinkBench will automati- \ufb01cient to characterize the ef\ufb01cacy of a pruning method. We\r\n cally apply the pruning, update the network according to a foundaftercomputingthesemetricsforthesamemodelun-\r\n standard training or \ufb01ne-tuning setup, and compute metrics der many different settings that reporting one metric is not\r\n across many models, datasets, random seeds, and levels of suf\ufb01cient. While these metrics are correlated, the correla-\r\n pruning. We defer discussion of ShrinkBench\u2019s implemen- tion is different for each pruning method. Thus, the relative\r\n tation and API to the project\u2019s documentation. performanceofdifferentmethodscanvarysigni\ufb01cantlyun-\r\n 7.2 Baselines der different metrics (Figure 6).\r\n WeusedShrinkBench to implement several existing prun- ResNet-18 on ImageNet\r\n ing heuristics, both as examples of how to use our library 0.70\r\n and as baselines that new methods can compare to: 0.65\r\n 0.60\r\n \u2022 Global Magnitude Pruning - prunes the weights with 0.55\r\n the lowest absolute value anywhere in the network. Accuracy0.50Global Weight\r\n \u2022 LayerwiseMagnitudePruning-foreachlayer,prunes Layer Weight\r\n 0.45 Global Gradient\r\n the weights with the lowest absolute value. Layer Gradient\r\n \u2022 Global Gradient Magnitude Pruning - prunes the 0.40 1 2 4 8 16 1 2 4 8 16 32\r\n weightswiththelowestabsolutevalueof(weight\u00d7gra- Compression Ratio Theoretical Speedup\r\n dient), evaluated on a batch of inputs. Figure 6: Top 1 Accuracy for ResNet-18 on ImageNet\r\n \u2022 Layerwise Gradient Magnitude Pruning - for each for several compression ratios and their corresponding\r\n layer, prunes the weights the lowest absolute value of theoretical speedups. Global methods give higher accu-\r\n (weight \u00d7 gradient), evaluated on a batch of inputs. racy than Layerwise ones for a \ufb01xed model size, but the\r\n \u2022 Random Pruning - prunes each weight independently reverse is true for a \ufb01xed theoretical speedup.\r\n with probability equal to the fraction of the network to Results Vary Across Models, Datasets, and Pruning\r\n be pruned. Amounts Many methods report results on only a small\r\n number of datasets, models, amounts of pruning, and ran-\r\n Magnitude-based approaches are common baselines in the domseeds. Iftherelativeperformanceofdifferentmethods\r\n literature and have been shown to be competitive with more tends to be constant across all of these variables, this may\r\n WhatistheStateofNeuralNetworkPruning?\r\n not be problematic. However, our results suggest that this Absolute Relative\r\n performance is not constant. 0.9 0.0\r\n Figure 7 shows the accuracy for various compression ra- 0.1\r\n 0.8 0.2\r\n tios for CIFAR-VGG (Zagoruyko, 2015) and ResNet-56\r\n on CIFAR-10. In general, Global methods are more accu- Accuracy0.7Global A Accuracy0.3\r\n rate than Layerwise methods and Magnitude-based meth- 0.6 Global B 0.4\r\n ods are more accurate than Gradient-based methods, with Layer A\r\n random performing worst of all. However, if one were to Layer B 0.5\r\n look only at CIFAR-VGG for compression ratios smaller 0.5 1 2 4 8 16 32 64 1 2 4 8 16 32 64\r\n than 10, one could conclude that Global Gradient outper- Compression Ratio Compression Ratio\r\n forms all other methods. Similarly, while Global Gradient Figure 8: Global and Layerwise Magnitude Pruning on\r\n consistently outperforms Layerwise Magnitude on CIFAR- two different ResNet-56 models. Even with all other\r\n VGG,theoppositeholdsonResNet-56(i.e.,theorangeand variables held constant, different initial models yield\r\n green lines switch places). different tradeoff curves. This may cause one method\r\n to erroneously appear better than another. Controlling\r\n Moreover, we found that for some settings close to the for initial accuracy does not \ufb01x this.\r\n drop-off point (such as Global Gradient, compression 16), We also found that the common practice of examining\r\n different random seeds yielded signi\ufb01cantly different re- changes in accuracy is insuf\ufb01cient to correct for initial\r\n sults (0.88 vs 0.61 accuracy) due to the randomness in model as a confounder. Even when reporting changes, one\r\n minibatch selection. This is illustrated by the large verti- pruning method can arti\ufb01cially appear better than another\r\n cal error bar in the left subplot. by virtue of beginning with a different model. We see this\r\n CIFAR-VGG ResNet-56 on the right side of Figure 8, where Layerwise Magnitude\r\n 0.9 with Weights B appears to outperform Global Magnitude\r\n 0.8 with Weights A, even though the former never outperforms\r\n the latter when initial model is held constant.\r\n 0.7 Global Weight\r\n Accuracy Layer Weight 8 CONCLUSION\r\n 0.6 Global Gradient\r\n Layer Gradient Considering the enormous interest in neural network prun-\r\n Random\r\n 0.5 1 2 4 8 16 32 1 2 4 8 16 32 ing over the past decade, it seems natural to ask simple\r\n Compression Ratio Compression Ratio questions about the relative ef\ufb01cacy of different pruning\r\n Figure7: Top1AccuracyonCIFAR-10forseveralcom- techniques. Althoughafewbasic\ufb01ndingsaresharedacross\r\n pression ratios. Global Gradient performs better than the literature, missing baselines and inconsistent experi-\r\n GlobalMagnitudeforCIFAR-VGGonlowcompression mental settings make it impossible to assess the state of\r\n ratios, but worse otherwise. Global Gradient is con- the art or con\ufb01dently compare the dozens of techniques\r\n sistently better than Layerwise Magnitude on CIFAR- proposed in recent years. After carefully studying the\r\n VGG,butconsistently worse on ResNet-56. literature and enumerating numerous areas of incompa-\r\n rability and confusion, we suggest concrete remedies in\r\n Using the Same Initial Model is Essential. As men- the form of a list of best practices and an open-source\r\n tioned in Section 4.5, many methods are evaluated using library\u2014ShrinkBench\u2014to help future research endeavors\r\n different initial models with the same architecture. To as- to produce the kinds of results that will harmonize the lit-\r\n sess whether beginning with a different model can skew erature and make our motivating questions easier to an-\r\n the results, we created two different models and evaluated swer. Furthermore,ShrinkBenchresultsonvariouspruning\r\n Global vs Layerwise Magnitude pruning on each with all techniques evidence the need for standardized experiments\r\n other variables held constant. whenevaluating neural network pruning methods.\r\n To obtain the models, we trained two ResNet-56 networks ACKNOWLEDGEMENTS\r\n using Adam until convergence with \u03b7 = 10\u22123 and \u03b7 =\r\n \u22124\r\n 10 . We\u2019ll refer to these pretrained weights as Weights We thank Luigi Celona for providing the data used in\r\n Aand Weights B, respectively. As shown on the left side (Bianco et al., 2018) and Vivienne Sze for helpful discus-\r\n of Figure 8, the different methods appear better on differ- sion. This research was supported by the Qualcomm Inno-\r\n ent models. With Weights A, the methods yield similar vation Fellowship, the \u201cla Caixa\u201d Foundation Fellowship,\r\n absolute accuracies. With Weights B, however, the Global Quanta Computer, and Wistron Corporation.\r\n method is more accurate at higher compression ratios.\r\n WhatistheStateofNeuralNetworkPruning?\r\n REFERENCES Frankle, J. and Carbin, M. The lottery ticket hypothesis:\r\n What\u2019s the advantage of the reference caffenet in com- Finding sparse, trainable neural networks. In 7th Inter-\r\n parison with the alexnet? https://github.com/ national Conference on Learning Representations, ICLR\r\n BVLC/caffe/issues/4202, 5 2016. Accessed: 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe-\r\n 2019-07-22. view.net, 2019. URL https://openreview.net/\r\n forum?id=rJl-b3RcF7.\r\n Keras exported model shows very low accuracy in Frankle, J., Dziugaite, G. K., Roy, D. M., and Carbin, M.\r\n tensor\ufb02ow serving. https://github.com/ The lottery ticket hypothesis at scale. arXiv preprint\r\n keras-team/keras/issues/7848, 9 2017. arXiv:1903.01611, 2019.\r\n Accessed: 2019-07-22.\r\n Bianco, S., Cadene, R., Celona, L., and Napoletano, P. Gale, T., Elsen, E., and Hooker, S. The state of sparsity in\r\n Benchmark analysis of representative deep neural net- deep neural networks, 2019.\r\n workarchitectures. IEEEAccess, 6:64270\u201364277, 2018. Gray, S., Radford, A., and Kingma, D. P. Gpu kernels for\r\n Blalock, D., Madden, S., and Guttag, J. Sprintz: Time se- block-sparse weights. arXiv preprint arXiv:1711.09224,\r\n ries compression for the internet of things. Proceedings 2017.\r\n of the ACM on Interactive, Mobile, Wearable and Ubiq- Han, S., Pool, J., Tran, J., and Dally, W. Learning both\r\n uitous Technologies, 2(3):93, 2018. weights and connections for ef\ufb01cient neural network. In\r\n Changpinyo, S., Sandler, M., and Zhmoginov, A. The Advances in neural information processing systems, pp.\r\n power of sparsity in convolutional neural networks. 1135\u20131143, 2015.\r\n arXiv preprint arXiv:1702.06257, 2017. Han, S., Mao, H., and Dally, W. J. Deep compression:\r\n Choi,Y.,El-Khamy,M.,andLee,J. Jointlysparseconvolu- Compressing deep neural network with pruning, trained\r\n tional neural networksindualspatial-winograddomains. quantization and huffman coding. In Bengio, Y. and Le-\r\n arXiv preprint arXiv:1902.08192, 2019. Cun, Y. (eds.), 4th International Conference on Learn-\r\n ing Representations, ICLR 2016, San Juan, Puerto Rico,\r\n Crall, J. Accuracy of resnet50 is much higher than May 2-4, 2016, Conference Track Proceedings, 2016.\r\n reported! https://github.com/kuangliu/ URLhttp://arxiv.org/abs/1510.00149.\r\n pytorch-cifar/issues/45, 2018. Accessed: Hassibi, B., Stork, D. G., and Wolff, G. J. Optimal brain\r\n 2019-07-22. surgeon and general network pruning. In IEEE inter-\r\n Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and national conference on neural networks, pp. 293\u2013299.\r\n Fei-Fei, L. Imagenet: A large-scale hierarchical image IEEE, 1993.\r\n database. In 2009 IEEE conference on computer vision He,K.,Zhang,X.,Ren,S.,andSun,J. Deepresiduallearn-\r\n andpattern recognition, pp. 248\u2013255. Ieee, 2009. ing for image recognition. In Proceedings of the IEEE\r\n Ding,X.,Ding,G.,Han,J.,andTang,S. Auto-balanced\ufb01l- conference on computer vision and pattern recognition,\r\n ter pruning for ef\ufb01cient convolutional neural networks. pp. 770\u2013778, 2016a.\r\n In Thirty-Second AAAI Conference on Arti\ufb01cial Intelli- He, K., Zhang, X., Ren, S., and Sun, J. Identity mappings\r\n gence, 2018. in deep residual networks. In European conference on\r\n Dong, X., Huang, J., Yang, Y., and Yan, S. More is less: A computer vision, pp. 630\u2013645. Springer, 2016b.\r\n morecomplicated network with less inference complex- He, Y., Zhang, X., and Sun, J. Channel pruning for accel-\r\n ity. In Proceedings of the IEEE Conference on Computer erating very deep neural networks. In Proceedings of the\r\n Vision and Pattern Recognition, pp. 5840\u20135848, 2017. IEEEInternational Conference on Computer Vision, pp.\r\n Dubey, A., Chatterjee, M., and Ahuja, N. Coreset-based 1389\u20131397, 2017.\r\n neural network compression. In Proceedings of the Eu- He, Y., Kang, G., Dong, X., Fu, Y., and Yang, Y. Soft\r\n ropean Conference on Computer Vision (ECCV), pp. \ufb01lter pruning for accelerating deep convolutional neural\r\n 454\u2013470, 2018. networks. In IJCAI International Joint Conference on\r\n Figurnov, M., Ibraimova, A., Vetrov, D. P., and Kohli, P. Arti\ufb01cial Intelligence, 2018a.\r\n Perforatedcnns: Acceleration through elimination of re- He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., and Han,\r\n dundant convolutions. In Advances in Neural Informa- S. Amc: Automl for model compression and accelera-\r\n tion Processing Systems, pp. 947\u2013955, 2016. tion on mobile devices. In Proceedings of the European\r\n WhatistheStateofNeuralNetworkPruning?\r\n Conference on Computer Vision (ECCV), pp. 784\u2013800, Lebedev, V. and Lempitsky, V. Fast convnets using group-\r\n 2018b. wise brain damage. In Proceedings of the IEEE Confer-\r\n ence on Computer Vision and Pattern Recognition, pp.\r\n Huang, Y., Cheng, Y., Chen, D., Lee, H., Ngiam, J., Le, 2554\u20132564, 2016.\r\n Q. V., and Chen, Z. Gpipe: Ef\ufb01cient training of gi- Lebedev, V., Ganin, Y., Rakhuba, M., Oseledets, I.,\r\n ant neural networks using pipeline parallelism. arXiv and Lempitsky, V. Speeding-up convolutional neu-\r\n preprint arXiv:1811.06965, 2018. ral networks using \ufb01ne-tuned cp-decomposition. arXiv\r\n Huang, Z. and Wang, N. Data-driven sparse structure se- preprint arXiv:1412.6553, 2014.\r\n lection for deep neural networks. In Proceedings of the LeCun, Y., Denker, J. S., and Solla, S. A. Optimal brain\r\n European Conference on Computer Vision (ECCV), pp. damage. In Advances in neural information processing\r\n 304\u2013320, 2018. systems, pp. 598\u2013605, 1990.\r\n Janowsky, S. A. Pruning versus clipping in neural net- LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al.\r\n works. Physical Review A, 39(12):6600\u20136603, June Gradient-based learning applied to document recogni-\r\n 1989. ISSN 0556-2791. doi: 10.1103/PhysRevA.39. tion. Proceedings of the IEEE, 86(11):2278\u20132324,\r\n 6600. URL https://link.aps.org/doi/10. 1998a.\r\n 1103/PhysRevA.39.6600.\r\n LeCun, Y., Cortes, C., and Burges, C. The mnist database\r\n Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, of handwritten digits, 1998b. Accessed: 2019-09-6.\r\n J., Girshick, R., Guadarrama, S., and Darrell, T. lenet. Lee, N., Ajanthan, T., Gould, S., and Torr, P. H. S.\r\n https://github.com/BVLC/caffe/blob/ A Signal Propagation Perspective for Pruning Neu-\r\n master/examples/mnist/lenet.prototxt, ral Networks at Initialization. arXiv:1906.06307 [cs,\r\n 2 2015a. stat], June 2019a. URL http://arxiv.org/abs/\r\n Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, 1906.06307. arXiv: 1906.06307.\r\n J., Girshick, R., Guadarrama, S., and Darrell, T. lenet- Lee, N., Ajanthan, T., and Torr, P. H. S. Snip: single-\r\n train-test. https://github.com/BVLC/caffe/ shot network pruning based on connection sensitivity.\r\n blob/master/examples/mnist/lenet_ In 7th International Conference on Learning Represen-\r\n train_test.prototxt,22015b. tations, ICLR 2019, New Orleans, LA, USA, May 6-\r\n Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., 9, 2019. OpenReview.net, 2019b. URL https://\r\n Long, J., Girshick, R., Guadarrama, S., and openreview.net/forum?id=B1VZqjAcYX.\r\n Darrell, T. Training lenet on mnist with caffe. Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf,\r\n https://caffe.berkeleyvision.org/ H.P. Pruning\ufb01ltersforef\ufb01cientconvnets. arXivpreprint\r\n gathered/examples/mnist.html, 5 2016. arXiv:1608.08710, 2016.\r\n Accessed: 2019-07-22. Lindstrom, P. Fixed-rate compressed \ufb02oating-point arrays.\r\n Jogeshwar, A. Validating resnet50. https://github. IEEEtransactionsonvisualizationandcomputergraph-\r\n com/keras-team/keras/issues/8672, 12 ics, 20(12):2674\u20132683, 2014.\r\n 2017. Accessed: 2019-07-22. Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., and Zhang, C.\r\n Kalchbrenner, N., Elsen, E., Simonyan, K., Noury, S., Learning ef\ufb01cient convolutional networks through net-\r\n Casagrande, N., Lockhart, E., Stimberg, F., Oord, A. work slimming. In Proceedings of the IEEE Interna-\r\n v. d., Dieleman, S., and Kavukcuoglu, K. Ef\ufb01cient neu- tional Conference on Computer Vision, pp. 2736\u20132744,\r\n ral audio synthesis. arXiv preprint arXiv:1802.08435, 2017.\r\n 2018. Liu, Z., Sun, M., Zhou, T., Huang, G., and Darrell, T. Re-\r\n Karnin, E. D. A simple procedure for pruning back- thinking the value of network pruning. In 7th Interna-\r\n propagation trained neural networks. IEEE transactions tional Conference on Learning Representations, ICLR\r\n onneural networks, 1(2):239\u2013242, 1990. 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe-\r\n view.net, 2019. URL https://openreview.net/\r\n Kim, Y.-D., Park, E., Yoo, S., Choi, T., Yang, L., and forum?id=rJlnB3C5Ym.\r\n Shin, D. Compression of deep convolutional neural net- Louizos, C., Ullrich, K., and Welling, M. Bayesian com-\r\n worksforfast and low power mobile applications. arXiv pression for deep learning. In Advances in Neural Infor-\r\n preprint arXiv:1511.06530, 2015. mation Processing Systems, pp. 3288\u20133298, 2017.\r\n WhatistheStateofNeuralNetworkPruning?\r\n Luo, J.-H., Wu, J., and Lin, W. Thinet: A \ufb01lter level Ratanaworabhan, P., Ke, J., and Burtscher, M. Fast loss-\r\n pruning method for deep neural network compression. less compression of scienti\ufb01c \ufb02oating-point data. In\r\n In Proceedings of the IEEE international conference on Data Compression Conference (DCC\u201906), pp. 133\u2013142.\r\n computer vision, pp. 5058\u20135066, 2017. IEEE, 2006.\r\n Mariet, Z. and Sra, S. Diversity networks: Neural network Reed, R. Pruning algorithms-a survey. IEEE Trans-\r\n compression using determinantal point processes. arXiv actions on Neural Networks, 4(5):740\u2013747, Septem-\r\n preprint arXiv:1511.05077, 2015. ber 1993. ISSN 10459227. doi: 10.1109/72.\r\n Molchanov, D., Ashukha, A., and Vetrov, D. Variational 248452. URL http://ieeexplore.ieee.org/\r\n dropout sparsi\ufb01es deep neural networks. In Proceed- document/248452/.\r\n ings of the 34th International Conference on Machine Siedelmann, H., Wender, A., and Fuchs, M. High speed\r\n Learning-Volume 70, pp. 2498\u20132507. JMLR. org, 2017. lossless image compression. In German Conference on\r\n Molchanov, P., Tyree, S., Karras, T., Aila, T., and Kautz, J. Pattern Recognition, pp. 343\u2013355. Springer, 2015.\r\n Pruning convolutional neural networks for resource ef\ufb01- Simonyan, K. and Zisserman, A. Very deep convolu-\r\n cient inference. arXiv preprint arXiv:1611.06440, 2016. tional networks for large-scale image recognition. arXiv\r\n Morcos,A.S.,Yu,H.,Paganini,M.,andTian,Y.Oneticket preprint arXiv:1409.1556, 2014.\r\n to win them all: generalizing lottery ticket initializations Suau, X., Zappella, L., and Apostoloff, N. Network com-\r\n across datasets and optimizers. arXiv:1906.02773 [cs, pression using correlation analysis of layer responses.\r\n stat], June 2019. URL http://arxiv.org/abs/ 2018.\r\n 1906.02773. arXiv: 1906.02773.\r\n Suzuki, T., Abe, H., Murata, T., Horiuchi, S., Ito, K.,\r\n Mozer, M. C. and Smolensky, P. Skeletonization: A tech- Wachi, T., Hirai, S., Yukishima, M., and Nishimura,\r\n nique for trimming the fat from a network via relevance T. Spectral-pruning: Compressing deep neural network\r\n assessment. In Advances in neural information process- via spectral analysis. arXiv preprint arXiv:1808.08558,\r\n ing systems, pp. 107\u2013115, 1989a. 2018.\r\n Mozer, M. C. and Smolensky, P. Using Relevance to Sze, V., Chen, Y.-H., Yang, T.-J., and Emer, J. Ef\ufb01cient\r\n Reduce Network Size Automatically. Connection processing of deep neural networks: A tutorial and sur-\r\n Science, 1(1):3\u201316, January 1989b. ISSN 0954-0091, vey. arXiv preprint arXiv:1703.09039, 2017.\r\n 1360-0494. doi: 10.1080/09540098908915626. URL\r\n https://www.tandfonline.com/doi/full/ Tan, M. and Le, Q. V. Ef\ufb01cientnet: Rethinking model scal-\r\n 10.1080/09540098908915626. ing for convolutional neural networks. arXiv preprint\r\n arXiv:1905.11946, 2019.\r\n Nola, D. Keras doesn\u2019t reproduce caffe example code\r\n accuracy. https://github.com/keras-team/ Tresp, V., Neuneier, R., and Zimmermann, H.-G. Early\r\n keras/issues/4444,112016. Accessed: 2019-07- brain damage. In Advances in neural information pro-\r\n 22. cessing systems, pp. 669\u2013675, 1997.\r\n Northcutt, C. Towards reproducibil- Vryniotis, V. Change bn layer to use moving mean/var\r\n ity: Benchmarking keras and pytorch. if frozen. https://github.com/keras-team/\r\n https://l7.curtisnorthcutt.com/towards-reproducibility- keras/pull/9965,42018. Accessed: 2019-07-22.\r\n benchmarking-keras-pytorch, 2 2019. Accessed: Wang, P. and Cheng, J. Accelerating convolutional neural\r\n 2019-07-22. networks for mobile applications. In Proceedings of the\r\n Parkhi, O. M., Vedaldi, A., Zisserman, A., et al. Deep face 24th ACM international conference on Multimedia, pp.\r\n recognition. In bmvc, volume 1, pp. 6, 2015. 541\u2013545. ACM, 2016.\r\n Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., Wen, W., Wu, C., Wang, Y., Chen, Y., and Li, H. Learn-\r\n DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and ing structured sparsity in deep neural networks. In Ad-\r\n Lerer, A. Automatic differentiation in pytorch. 2017. vances in neural information processing systems, pp.\r\n Peng, B., Tan, W., Li, Z., Zhang, S., Xie, D., and Pu, S. 2074\u20132082, 2016.\r\n Extreme network compression via \ufb01lter group approxi- Yamamoto,K.andMaeno,K. Pcas: Pruningchannelswith\r\n mation. In Proceedings of the European Conference on attention statistics. arXiv preprint arXiv:1806.05382,\r\n ComputerVision (ECCV), pp. 300\u2013316, 2018. 2018.\r\n WhatistheStateofNeuralNetworkPruning?\r\n Yang, T.-J., Chen, Y.-H., and Sze, V. Designing energy-\r\n ef\ufb01cient convolutional neural networks using energy-\r\n aware pruning. In Proceedings of the IEEE Conference\r\n on Computer Vision and Pattern Recognition, pp. 5687\u2013\r\n 5695, 2017.\r\n Yao, Z., Cao, S., and Xiao, W. Balanced sparsity\r\n for ef\ufb01cient dnn inference on gpu. arXiv preprint\r\n arXiv:1811.00206, 2018.\r\n Yu, R., Li, A., Chen, C.-F., Lai, J.-H., Morariu, V. I., Han,\r\n X., Gao, M., Lin, C.-Y., and Davis, L. S. Nisp: Pruning\r\n networksusingneuronimportancescorepropagation. In\r\n Proceedings of the IEEE Conference on Computer Vi-\r\n sion and Pattern Recognition, pp. 9194\u20139203, 2018.\r\n Zagoruyko, S. 92.45% on cifar-10 in torch. https://\r\n torch.ch/blog/2015/07/30/cifar.html, 7\r\n 2015. Accessed: 2019-07-22.\r\n Zhang, X., Zou, J., He, K., and Sun, J. Accelerating very\r\n deepconvolutionalnetworksforclassi\ufb01cationanddetec-\r\n tion. IEEEtransactionsonpatternanalysisandmachine\r\n intelligence, 38(10):1943\u20131955, 2015.\r\n Zhao, W. X., Zhang, X., Lemire, D., Shan, D., Nie, J.-Y.,\r\n Yan, H., and Wen, J.-R. A general simd-based approach\r\n to accelerating compression algorithms. ACM Transac-\r\n tions on Information Systems (TOIS), 33(3):15, 2015.\r\n Zukowski, M., Heman, S., Nes, N., and Boncz, P. Super-\r\n scalar ram-cpucachecompression. InDataEngineering,\r\n 2006. ICDE\u201906. Proceedings of the 22nd International\r\n Conference on, pp. 59\u201359. IEEE, 2006.\r\n WhatistheStateofNeuralNetworkPruning?\r\n A CORPUSANDDATACLEANING B CHECKLISTFOREVALUATINGA\r\n Weselected the 81 papers used in our analysis in the fol- PRUNING METHOD\r\n lowing way. First, we conducted an ad hoc literature For any pruning technique proposed, check if:\r\n search, \ufb01nding widely cited papers introducing pruning\r\n methods and identifying other pruning papers that cited \u2022 It is contextualized with respect to magnitude prun-\r\n themusingGoogleScholar. Wethenwentthroughthecon- ing, recently-published pruning techniques, and prun-\r\n ference proceedings from the past year\u2019s NeurIPS, ICML, ing techniques proposed prior to the 2010s.\r\n CVPR, ECCV, and ICLR and added all relevant papers\r\n (though it is possible we had false dismissals if the title \u2022 The pruning algorithm, constituent subroutines (e.g.,\r\n and abstract did not seem relevant to pruning). Finally, score, pruning, and \ufb01ne-tuning functions), and hyper-\r\n during the course of cataloging which papers compared to parameters are presented in enough detail for a reader\r\n which others, we added to our corpus any pruning paper to reimplement and match the results in the paper.\r\n that at least one existing paper in our corpus purported to \u2022 All claims about the technique are appropriately\r\n compare to. We included both published papers and un- restricted to only the experiments presented (e.g.,\r\n published ones of reasonable quality (typically on arXiv). CIFAR-10, ResNets, image classi\ufb01cation tasks, etc.).\r\n Since we make strong claims about the lack of compar-\r\n isons, we included in our corpus \ufb01ve papers whose meth- \u2022 There is a link to downloadable source code.\r\n ods technically do not meet our de\ufb01nition of pruning but\r\n are similar in spirit and compared to by various pruning For all experiments, check if you include:\r\n papers. In short, we included essentially every paper intro-\r\n ducing a method of pruning neural networks that we could \u2022 A detailed description of the architecture with hyper-\r\n \ufb01nd, taking care to capture the full directed graph of papers parameters in enough detail to for a reader to reimple-\r\n and comparisons between them. ment it and train it to the same performance reported\r\n Because different papers report slightly different metrics, in the paper.\r\n particularly with respect to model size, we converted re-\r\n ported results to a standard set of metrics whenever possi- \u2022 If the architecture is not novel: a citation for the ar-\r\n ble. For example, we converted reported Top-1 error rates chitecture/hyperparameters and a description of any\r\n to Top-1 accuracies, and fractions of parameters pruned to differences in architecture, hyperparameters, or per-\r\n compression ratios. Note that it is not possible to con- formance in this paper.\r\n vert between size metrics and speedup metrics, since the \u2022 A detailed description of the dataset hyperparameters\r\n amount of computation associated with a given parameter (e.g., batch size and augmentation regime) in enough\r\n can depend on the layer in which it resides (since convo- detail for a reader to reimplement it.\r\n lutional \ufb01lters are reused at many spatial positions). For\r\n simplicity and uniformity, we only consider self-reported \u2022 Adescription of the library and hardware used.\r\n results except where stated otherwise.\r\n Wealsodidnotattempt to capture all reported metrics, but For all results, check if:\r\n instead focused only on model size reduction and theoret-\r\n ical speedup, since 1) these are by far the most commonly \u2022 Dataispresentedacrossarangeofcompressionratios,\r\n reported and, 2) there is already a dearth of directly compa- including extreme compression ratios at which the ac-\r\n rable numbers even for these common metrics. This is not curacy of the pruned network declines substantially.\r\n entirely fair to methods designed to optimize other metrics, \u2022 Dataspeci\ufb01estherawaccuracyofthenetworkateach\r\n such as power consumption (Louizos et al., 2017; Yang point.\r\n et al., 2017; Han et al., 2015; Kim et al., 2015), memory\r\n bandwidth usage (Peng et al., 2018; Kim et al., 2015), or \u2022 Data includes multiple runs with separate initializa-\r\n \ufb01ne-tuning time (Dubey et al., 2018; Yamamoto & Maeno, tions and random seeds.\r\n 2018;Huang&Wang,2018;Heetal.,2018a),andwecon-\r\n sider this a limitation of our analysis. \u2022 Dataincludesclearly de\ufb01nederror bars and a measure\r\n Lastly, as a result of relying on reading of hundreds of of central tendency (e.g., mean) and variation (e.g.,\r\n pages of dense technical content, we are con\ufb01dent that we standard deviation).\r\n have made some number of isolated errors. We therefore \u2022 Data includes FLOP-counts if the paper makes argu-\r\n welcome correction by email and refer the reader to the ments about ef\ufb01ciency and performance due to prun-\r\n arXivversionofthispaperforthemostup-to-daterevision. ing.\r\n WhatistheStateofNeuralNetworkPruning?\r\n For all pruning results presented, check if there is a com- \u2022 Epochs: 30\r\n parison to: \u2022 Optimizer: Adam\r\n \u2022 Arandompruningbaseline. \u2022 Initial Learning Rate: 3 \u00d7 10\u22124\r\n \u2022 Learning rate schedule: Fixed\r\n \u2013 Aglobal random pruning baseline.\r\n \u2013 Arandompruning baseline with the same layer- All reported ImageNet experiments used the following\r\n wise pruning proportions as the proposed tech- \ufb01netuning setup\r\n nique. \u2022 Batch size: 256\r\n \u2022 Amagnitudepruning baseline. \u2022 Epochs: 20\r\n \u2013 Aglobaloruniformlayerwiseproportionmagni- \u2022 Optimizer: SGD with Nesterov Momentum (0.9)\r\n tude pruning baseline. \u2022 Initial Learning Rate: 1 \u00d7 10\u22123\r\n \u2013 Amagnitudepruningbaselinewiththesamelay- \u2022 Learning rate schedule: Fixed\r\n erwise pruning proportions as the proposed tech-\r\n nique. D ADDITIONALRESULTS\r\n \u2022 Other relevant state-of-the-art techniques, including: Here we include the entire set of results obtained with\r\n \u2013 Adescription of how the comparisons were pro- ShrinkBench. For CIFAR10, results are included for\r\n duced (data taken from paper, reimplementation, CIFAR-VGG, ResNet-20, ResNet-56 and ResNet-110.\r\n or reuse of code from the paper) and any differ- Standard deviations across three different random runs are\r\n ences or uncertainties between this setting and plotted as error bars. For ImageNet, results are reported for\r\n the setting used in the main experiments. ResNet-18.\r\n C EXPERIMENTALSETUP\r\n For reproducibility purposes, ShrinkBench \ufb01xes ran-\r\n dom seeds for all the dependencies (PyTorch, NumPy,\r\n Python).\r\n C.1 Pruning Methods\r\n For the reported experiments, we did not prune the clas-\r\n si\ufb01er layer preceding the softmax. ShrinkBench supports\r\n pruning said layer as an option to all proposed pruning\r\n strategies. For both Global and Layerwise Gradient Mag-\r\n nitude Pruning a single minibatch is used to compute the\r\n gradients for the pruning. Three independent runs using\r\n different random seeds were performed for every CIFAR10\r\n experiment. We found some variance across methods that\r\n relied on randomness, such as random pruning or gradient\r\n based methods that use a sampled minibatch to compute\r\n the gradients with respect to the weights.\r\n C.2 Finetuning Setup\r\n Pruning was performed from the pretrained weights and\r\n \ufb01xed from there forwards. Early stopping is implemented\r\n during \ufb01netuning. Thus if the validation accuracy repeat-\r\n edly decreases after some point we stop the \ufb01netuning pro-\r\n cess to prevent over\ufb01tting.\r\n AllreportedCIFAR10experimentsusedthefollowing\ufb01ne-\r\n tuning setup:\r\n \u2022 Batch size: 64\r\n WhatistheStateofNeuralNetworkPruning?\r\n 0.95 CIFAR-VGG on CIFAR-10 0.95 CIFAR-VGG on CIFAR-10\r\n 0.90 0.90\r\n 0.85 0.85\r\n Accuracy0.80Global Weight Accuracy0.80\r\n Layer Weight\r\n 0.75 Global Gradient 0.75\r\n Layer Gradient\r\n Random\r\n 0.70 1 2 4 8 16 32 0.70 1 2 4 8 16 3232\r\n Compression Ratio Theoretical Speedup\r\n Figure9: Accuracyforseverallevelsofcompression Figure 10: Accuracy vs theoretical speedup for\r\n for CIFAR-VGGonCIFAR-10 CIFAR-VGGonCIFAR-10\r\n 0.95 ResNet-20 on CIFAR-10 0.95 ResNet-20 on CIFAR-10\r\n 0.90 0.90\r\n 0.85 0.85\r\n Accuracy0.80Global Weight Accuracy0.80Global Weight\r\n Layer Weight Layer Weight\r\n 0.75 Global Gradient 0.75 Global Gradient\r\n Layer Gradient Layer Gradient\r\n Random Random\r\n 0.70 1 2 4 8 16 32 0.70 1 2 4 8 16 3232\r\n Compression Ratio Theoretical Speedup\r\n Figure 11: Accuracy for several levels of compres- Figure 12: Accuracy vs theoretical speedup for\r\n sion for ResNet-20 on CIFAR-10 ResNet-20 on CIFAR-10\r\n 0.95 ResNet-56 on CIFAR-10 0.95 ResNet-56 on CIFAR-10\r\n 0.90 0.90\r\n 0.85 0.85\r\n Accuracy0.80Global Weight Accuracy0.80Global Weight\r\n Layer Weight Layer Weight\r\n 0.75 Global Gradient 0.75 Global Gradient\r\n Layer Gradient Layer Gradient\r\n Random Random\r\n 0.70 1 2 4 8 16 32 0.70 1 2 4 8 16 3232\r\n Compression Ratio Theoretical Speedup\r\n Figure 13: Accuracy for several levels of compres- Figure 14: Accuracy vs theoretical speedup for\r\n sion for ResNet-56 on CIFAR-10 ResNet-56 on CIFAR-10\r\n WhatistheStateofNeuralNetworkPruning?\r\n 0.95 ResNet-110 on CIFAR-10 0.95 ResNet-110 on CIFAR-10\r\n 0.90 0.90\r\n 0.85 0.85\r\n Accuracy0.80Global Weight Accuracy0.80Global Weight\r\n Layer Weight Layer Weight\r\n 0.75 Global Gradient 0.75 Global Gradient\r\n Layer Gradient Layer Gradient\r\n Random Random\r\n 0.70 1 2 4 8 16 32 0.70 1 2 4 8 16 3232\r\n Compression Ratio Theoretical Speedup\r\n Figure 15: Accuracy for several levels of compres- Figure 16: Accuracy vs theoretical speedup for\r\n sion for ResNet-110 on CIFAR-10 ResNet-110 on CIFAR-10\r\n ResNet-18 on ImageNet ResNet-18 on ImageNet\r\n 0.70 0.70\r\n 0.65 0.65\r\n 0.60 0.60\r\n 0.55 0.55\r\n Top 1 Accuracy0.50Global Weight Top 1 Accuracy0.50Global Weight\r\n Layer Weight Layer Weight\r\n 0.45 Global Gradient 0.45 Global Gradient\r\n Layer Gradient Layer Gradient\r\n 0.40 1 2 4 8 16 0.40 1 2 4 8 16 32\r\n Compression Ratio Theoretical Speedup\r\n Figure 17: Accuracy for several levels of compres- Figure 18: Accuracy vs theoretical speedup for\r\n sion for ResNet-18 on ImageNet ResNet-18 on ImageNet\r\n", "award": [], "sourceid": 73, "authors": [{"given_name": "Davis", "family_name": "Blalock", "institution": "MIT"}, {"given_name": "Jose Javier", "family_name": "Gonzalez Ortiz", "institution": "MIT"}, {"given_name": "Jonathan", "family_name": "Frankle", "institution": "MIT"}, {"given_name": "John", "family_name": "Guttag", "institution": "MIT"}]}