{"title": "A Systematic Methodology for Analysis of Deep Learning Hardware and Software Platforms", "book": "Proceedings of Machine Learning and Systems", "page_first": 30, "page_last": 43, "abstract": "Training deep learning models is compute-intensive and there is an industry-wide trend towards hardware and software specialization to improve performance. To systematically compare deep learning systems, we introduce a methodology comprised of a set of analysis techniques and parameterized end-to-end models for fully connected, convolutional, and recurrent neural networks. This methodology can be applied to analyze various hardware and software systems, and is intended to complement traditional methods. We demonstrate its utility by comparing two generations of specialized platforms (Google's Cloud TPU v2/v3), three heterogeneous platforms (Google TPU, Nvidia GPU, and Intel CPU), and specialized software stacks (TensorFlow and CUDA).", "full_text": " ASYSTEMATICMETHODOLOGYFORANALYSISOFDEEPLEARNING\r\n HARDWAREANDSOFTWAREPLATFORMS\r\n YuEmmaWang1 Gu-YeonWei1 DavidBrooks1\r\n ABSTRACT\r\n Training deep learning models is compute-intensive and there is an industry-wide trend towards hardware and\r\n software specialization to improve performance. To systematically compare deep learning systems, we introduce a\r\n methodology comprised of a set of analysis techniques and parameterized end-to-end models for fully connected,\r\n convolutional, and recurrent neural networks. This methodology can be applied to analyze various hardware and\r\n software systems, and is intended to complement traditional methods. We demonstrate its utility by comparing\r\n twogenerations of specialized platforms (Google\u2019s Cloud TPU v2/v3), three heterogeneous platforms (Google\r\n TPU,NvidiaGPU,andIntelCPU),andspecializedsoftware stacks (TensorFlow and CUDA).\r\n 1 INTRODUCTION et al., 2016) and Transformer (Vaswani et al., 2017) can lead\r\n With the end of Moore\u2019s law, academic and industrial re- to misleading conclusions. For example, Transformer is a\r\n search efforts have shifted from general-purpose processors large FC modelthattrains3.5\u00d7fasterontheTensorProcess-\r\n to domainspeci\ufb01carchitectures(DSAs)(Hennessy&Patter- ing Unit (TPU) than on a GPU, yet focusing on this single\r\n son, 2011). Deep learning, which has revolutionized many model would not reveal the severe TPU memory bandwidth\r\n application domains (Silver et al., 2017; Huang et al., 2017; bottleneck that arises with FCs larger than 4k nodes.\r\n Amodeietal., 2016; Wu et al., 2016), is a promising \ufb01eld We propose a comprehensive performance evaluation\r\n for DSAs (Dean et al., 2018). New customized training methodology that combines parameterized deep learning\r\n hardware, software stacks, and optimization tools are being benchmarks with systematic analysis techniques. We in-\r\n developed to support ever more sophisticated deep learning troduce ParaDnn, a tool that generates thousands of pa-\r\n models. Thus there is a great need to concurrently develop rameterized multi-layer models, including fully-connected\r\n a systematic and scienti\ufb01c methodology for comprehensive models, convolutional neural networks, and recurrent neural\r\n performance analysis of hardware and software systems networks, with model parameter sizes that vary by almost\r\n customized for deep learning. \ufb01ve orders of magnitude, far beyond the range of existing\r\n Therapidevolutionofdeeplearningmodelsandcorrespond- benchmarks. Systematic analysis techniques then learn the\r\n ing hardware and software platforms requires new analysis sensitivity of performance to model hyperparameters and\r\n techniques that go beyond simply running today\u2019s well- explore various dimensions of the design space. We show\r\n known deep learning models on individual platforms. A that this parameterized analysis methodology complements\r\n systematic methodology must expose interactions between the use of real-world workloads (e.g., MLPerf), leading to\r\n hardware and software platforms across the spectrum of insights that traditional approaches either cannot expose or\r\n modelattributes (e.g., hyperparameters), so that the result- cannot fully explain.\r\n ing insights can be applied to future models. The methodol- Weconductcasestudies in three diverse performance eval-\r\n ogy itself needs a fast development cycle to rapidly target uation scenarios: homogeneous platforms, heterogeneous\r\n newplatforms, and it should include large enough models platforms, and software stacks. We hope to motivate re-\r\n to stress the limits of emerging platforms. searchers to apply our methodology to other platforms. In\r\n Recent analysis efforts have been limited to relatively small Section 4, we analyze and compare two generations of ho-\r\n collections of seemingly arbitrary DNN models (Mattson mogeneous specialized platforms, TPU v2 and v3. Our\r\n et al., 2019; Adolf et al., 2016; Chen et al., 2012; Tao methodology provides insights for designing and upgrading\r\n et al., 2018). The development of such suites is very time- MLaccelerators in production-scale systems. In Section 5,\r\n consuming. It took half a year to release MLPerf v0.6, and weperformcrosscomparisonofthree architectures (CPU,\r\n months to add a new model. Even so, the shelf life of such GPU,andTPU)thatspanthecontinuumbetweengeneral\r\n modelsisseldommorethanacoupleofyears. Moreover,us- purpose processors and specialized accelerators, and the\r\n ing a collection of individual models such as ResNet-50 (He methodology reveals individual strengths and weaknesses\r\n of each platform. In Section 6, we explore the performance\r\n 1John A. Paulson School of Engineering and Applied Sciences, evolution of specialized software stacks, TensorFlow and\r\n Harvard University, Cambridge, MA, USA. Correspondence to: CUDA.Table1summarizesfourteenobservations and in-\r\n YuEmmaWang. sights as examples enabled by our methodology.\r\n Proceedings of the 3rd MLSys Conference, Austin, TX, USA, 2020. While our analysis methodology is able to reveal optimiza-\r\n Copyright 2020 by the author(s). tion opportunities in current system designs, optimization\r\n ASystematicMethodologyforAnalysisofDeepLearningHardwareandSoftwarePlatforms\r\n Observation Proof Insight/Explanation\r\n 1. TPU exploits the parallelism from batch size and the model width. Fig 2 Todesign/upgrade new specialized systems,\r\n 2. Many operations are bottlenecked by TPU memory bandwidth. Fig 3 architects need to consider interactions between\r\n 3. TPU suffers from large inter-chip communication overhead. Fig 4 the operation mix from key workloads (arithmetic\r\n 4. Smaller CNN models are more bottlenecked by CPU hosts. Fig 5 intensity) and system con\ufb01gurations (FLOPS,\r\n 5. TPU v3 speeds up compute-bound MatMuls by 2.3\u00d7, memorybandwidth/capacity, intra-chip/host-device\r\n memory-boundonesby3\u00d7,andlargeembeddingsby>3\u00d7. Fig 6 interconnect). TPU serves as a great example.\r\n 6. The largest FC models prefer CPU due to memory constraints. Fig 7 Needformodelparallelism on GPU and TPU.\r\n 7. Models with large batch size prefer TPU. Fig 8 Large batches pack well on systolic arrays;\r\n Those with small batch size prefer GPU. Fig 10 warp scheduling is \ufb02exible for small batches.\r\n 8. Smaller FC models prefer TPU and larger FC models prefer GPU. Fig 8 FCneedsmorememorybandwidthpercore(GPU).\r\n 9. TPU speedup over GPU increases with larger CNNs. Fig 10 TPUarchitecture is highly optimized for large CNNs.\r\n 10. TPU achieves up to 3\u00d7 FLOPS utilization compared to GPU. Fig 11 TPUisoptimizedfor both CNN and RNN models.\r\n 11. GPUperformance scales better with RNN embedding size than TPU. Fig 10 GPUismore\ufb02exibletoparallelize non-MatMuls.\r\n 12. Within seven months, the software stack specialized for TPU It is easier to optimize for certain models\r\n wasimprovedbyupto2.5\u00d7(CNN),7\u00d7(FC),and9.7\u00d7(RNN). Fig 12 than to bene\ufb01t all models at once.\r\n 13. Quantization from 32 bits to 16 bits Smaller data types save memory traf\ufb01c and enable\r\n signi\ufb01cantly improves TPU and GPU performance. Fig 12 larger batch sizes, resulting in super-linear speedups.\r\n 14. TensorFlow and CUDA teams provide substantial performance There is huge potential to optimize compilers\r\n improvements in each update. Fig 12 even after the hardware has been shipped.\r\n Table 1: A summary of major observations and insights enabled by our analysis methodology.\r\n details are beyond the scope of this paper. Our analysis Variable Layer Nodes Input Output Batch Size\r\n Min 4 32 2000 200 64\r\n focuses on training, not inference. We do not study accu- Max 128 8192 8000 1000 16384\r\n racy or the performance of multi-GPU/TPU systems. Such Inc \u00d72 \u00d72 +2000 +200 \u00d72\r\n studies might yield different conclusions, and we leave these (a) Fully Connected Models\r\n extensions to future work, as each deserves in-depth study. Variable Block Filter Image Output Batch Size\r\n Section 7 discusses the limitations in detail. Min 1 16 200 500 64\r\n Max 8 6 300 1500 1024\r\n 2 METHODOLOGY Inc +1 \u00d72 +50 +500 \u00d72\r\n Current deep learning (DL) performance analysis methods (b) Conv. Neural Nets: Residual and Bottleneck Blocks\r\n have limitations in terms of the insights they are able to Variable Layer Embed Length Vocab Batch Size\r\n reveal. They often leverage two distinct types of bench- Min 1 100 10 2 16\r\n mark suites: real-world suites such as MLPerf (Mattson Max 13 900 90 1024 1024\r\n et al., 2019), Fathom (Adolf et al., 2016), BenchNN (Chen Inc +4 +400 +40 \u00d74 \u00d74\r\n et al., 2012), and BenchIP (Tao et al., 2018), and micro- (c) Recurrent Neural Networks: RNN, LSTM, GRU\r\n benchmarksuites, such as DeepBench (Research, 2017) and Table 2: The ranges of the hyperparameters and dataset variables\r\n BenchIP.Eachreal-worldsuitecontainsahandfulofpopular (italic) chosen in this paper.\r\n DLmodelsspanningavariety of model architectures. Such limited to those cases, and one goal of this paper is to moti-\r\n suites have a long development cycle. Their shelf-life is un- vate application of our methodology to new platforms.\r\n knownsincetheyonlycontaintoday\u2019sdeeplearningmodels,\r\n which may become obsolete as DL models evolve rapidly. 2.1 ParaDnn\r\n Further, they fail to reveal deep insights into interactions We\ufb01rstintroduce ParaDnn, a tool that generates parameter-\r\n between DL model attributes and hardware performance, ized end-to-end models to run on target platforms. ParaDnn\r\n since the benchmarks are sparse points in the vast space creates models encompassing fully-connected models (FC),\r\n of deep learning models. Micro-benchmark suites exercise convolutional neural networks (CNN), and recurrent neu-\r\n basic operations (e.g., matrix multiplication or convolution) ral networks (RNN). The models are parameterizable, so\r\n in neural networks, but they cannot simulate complex depen- ParaDnn models are equal to or greater in size compared\r\n dencies between different operations in end-to-end models. to today\u2019s real-world models. For example, a single end-to-\r\n Tocomplement existing performance analysis methods, we end CNNmodelfromParaDnncontainsamixtureofmany\r\n introduce a systematic methodology, composed of a tool, different layers with different sizes of convolution, batch\r\n ParaDnn, and a set of analysis methods. ParaDnn has the normalization, pooling, and FC layers. The complexity of\r\n advantages of the above approaches, with the goal of provid- ParaDnnworkloadsiscomparabletothatofreal-worldmod-\r\n ing large \u201cend-to-end\u201d models covering current and future els (e.g., ResNet-50 and Transformer), as will be shown in\r\n applications, and parameterizing the models to explore a Figure 1. Insights about hardware performance sensitivity\r\n much larger design space of DNN model attributes. Our to model attributes allow interpolating and extrapolating\r\n methodology can stress the upper and lower bounds of hard- to future models of interest. These insights could not be\r\n ware and software systems in various dimensions, including discovered with either the small point space exploration of\r\n \ufb02oating-point computation capability, memory bandwidth, the real-world suites or microbenchmarks, which do not\r\n inter-chip bandwidth, and host-device balance. For cross- capture inter-operation dependencies as ParaDnn does. The\r\n platform comparisons, the methodology can also discover model types of ParaDnn cover 95% of Google\u2019s TPU work-\r\n cases favoring one platform over another and describe the loads (Jouppi et al., 2017), all of Facebook\u2019s deep learning\r\n DLhyperparameters of such cases. With ParaDnn, such models(Hazelwoodetal.,2018;Guptaetal.,2019;Naumov\r\n studies can be conducted comprehensively, quickly, and et al., 2019), and eight out of nine MLPerf models (Mattson\r\n conveniently. The utility of parameterized analysis is not et al., 2019), with the exception of minigo, the reinforcement\r\n learning model.\r\n ASystematicMethodologyforAnalysisofDeepLearningHardwareandSoftwarePlatforms\r\n Fully-Connected Models FC models comprise multiple ResidualFC\r\n fully-connected layers. The architecture is BottleneckRNN\r\n Input \u2192[Layer[Node]]\u2192Output, LSTMGRU\r\n where [Layer] means the number of layers is variable. We TransformerRetinaNet\r\n can sweep the number of layers, the number of nodes per ResNet-50DenseNet\r\n layer, and the numbers input and output units of the datasets. SqueezeNetMobileNet\r\n ConvolutionalNeuralNetworksCNNmodelsareresidual 105 10 6 1 07 108\r\n networks. The architecture of ParaDnn CNNs is # Trainable Parameters\r\n Input\u2192[Residual/Bottleneck Block]\u00d74\u2192FC\u2192Output. Figure 1: The numbers of trainable parameters for all models.\r\n Aresidual network contains four groups of blocks (He et al., all ParaDnn workloads on platforms under study to collect\r\n 2016). Each can be a residual block or a bottleneck block, performance metrics of interest. All analysis methods and\r\n followed by a fully-connected layer. Residual blocks have results distinguish ParaDnn from suites of individual mod-\r\n twoconvolutional layers and two batch normalization lay- els, because the real-world suites do not support sensitivity\r\n ers, while bottleneck blocks have three of each. ParaDnn analysis, and ParaDnn covers a much larger design space.\r\n treats the minimum number of \ufb01lters as a variable, and it Heat Map With ParaDnn, we can measure performance\r\n doubles in every group. An input image is square with three sensitivity to hyperparameters. Heat maps are an intuitive\r\n channels, represented by its length. approach to understand the design space of DL models. A\r\n Recurrent Neural Networks RNNs are comprised of mul- heat map uses colors to show how a performance metric\r\n tiple layers of basic RNN, LSTM, or GRU cells: of interest responds to model hyperparameters (on x- and\r\n Input \u2192[RNN/LSTM/GRUCell]\u2192Output. y-axes). The rate of color change across the map re\ufb02ects the\r\n Eachtokenoftheinputsequenceisembeddedwithina\ufb01xed sensitivity of performance to hyperparameters.\r\n length vector, and the length of the vector is the embedding Quanti\ufb01cationwithLinearRegressionTable2shows\ufb01ve\r\n size. We sweep the number of layers and the embedding hyperparameters of each model type under study, and a heat\r\n size. The variables in the dataset include the maximum map can only visualize two hyperparameters. Also, ob-\r\n length per input sequence and vocabulary size. serving sensitivity via heat maps is more qualitative than\r\n RangeofHyperparametersandDatasetsWechoosethe quantitative. We propose to use linear regression (LR) to\r\n range of hyperparameters and datasets to cover the real quantify the sensitivity. We train a LR model using hyperpa-\r\n models (Section 2.2), and make sure the design space is rameters to predict performance, and use the weights of the\r\n tractable. Table 2 summarizes how hyperparameters are hyperparameters as a measure of sensitivity. Other metrics\r\n swept. We focus on large batches, and extremely small including T- and F-test may be used for this purpose (Hogg\r\n batches may lead to different conclusions. By default, this et al., 2005), but they only report positive values of im-\r\n paper uses CNNs with bottleneck blocks and basic RNNs. portance. LR reports the signs of the weights, indicating\r\n positive or negative correlations. Note that this LR model is\r\n 2.2 Real-World Models not for prediction. Section 4.1 presents a detailed example.\r\n In addition to ParaDnn, we study six real-world models. We Roo\ufb02ineModelRoo\ufb02inemodelsareusefultostudymem-\r\n showthat ParaDnn and those models are complementary\u2014 ory and computation bottlenecks (Williams et al., 2009;\r\n ParaDnn explores a larger design space, and real models Jouppi et al., 2017). A roo\ufb02ine represents the upper bound\r\n represent several currently popular design points. of \ufb02oating-point operations per second (FLOPS) for work-\r\n This work focuses on TensorFlow, the native framework loads with different compute intensity. Roo\ufb02ine model\r\n for TPU. We include two of the three workloads in Ten- analysis shows that ParaDnn\u2019s models range from extremely\r\n sorFlow from MLPerf (Mattson et al., 2019), i.e., Trans- bandwidth-bound to compute-bound. Such a range is hard\r\n former (Vaswani et al., 2017) and ResNet-50 (He et al., to achieve with existing real models, especially to reach the\r\n 2016). We also select other real-world workloads (Repos- limits of TPUs. Section 4.2 presents details.\r\n itory, 2018), including RetinaNet (Lin et al., 2017a), In addition to roo\ufb02ine models, we study the design space in\r\n DenseNet (Huang et al., 2017), MobileNet (Howard et al., other dimensions as well, by visualizing ParaDnn and real-\r\n 2017), and SqueezeNet (Iandola et al., 2016). We refer to world models on scatter plots with various x- and y-axes.\r\n themasreal workloads/models. Section 4.4 studies FLOPS and data infeed time. Section 5\r\n Figure 1 shows the numbers of trainable parameters across studies model size and speedup.\r\n all workloads to quantify the sizes of the models. The BoxPlotsWeuseboxplotstosummarizetheperformance\r\n ParaDnn workloads are shown as ranges and the real work- of each ParaDnn model type. Box plots show that the per-\r\n loads as dots. ParaDnn covers a large range of models, from formance of a ParaDnn model type spans a large range, and\r\n 10ktonearlyabillionparameters. Transformeristhelargest they highlight the risk of overly optimizing hardware and\r\n real FC, and RetinaNet is the largest real CNN. The small software systems for certain models.\r\n models, SqueezeNet and MobileNet, are typical of models 3 HARDWAREPLATFORMS\r\n targeting mobile applications. Ourselection of hardware re\ufb02ects the latest con\ufb01gurations\r\n 2.3 Analysis Methods widely available in cloud platforms at paper submission\r\n ParaDnn enables a set of analysis methods that can quan- time. Platform speci\ufb01cations are summarized in Table 3.\r\n tify, compare, and visualize the DL design space in var- CPUPlatformTheCPUisann1-standard-32instancefrom\r\n ious dimensions. We apply those methods after running Google Cloud Platform with Skylake architecture. It has 16\r\n ASystematicMethodologyforAnalysisofDeepLearningHardwareandSoftwarePlatforms\r\n Mem MemBdw Peak FLO PS% FL OP S% FLO PS %\r\n Platform Unit Version (GB) (GB/s) FLOPS 6 50 6 4 20.0\r\n CPU 1 VM Skylake 120 16.6 2TSP\u2020 7 40 40 17.5\r\n GPU 1 Pkg V100 16 900 125T 8 7 6 15.0\r\n TPU 1 Board v2 64 2400 180T 9 30 8 30 12.5\r\n * 10 20 8 10.0\r\n TPUv3 (8 cores) v3 128 3600 420T 11 20 9 7.5\r\n \u2020 2FMA\u00d732Single-Precision\u00d716Cores\u00d72GHz=2SPTFLOPS 12 10 5.0\r\n * Log2(Batch Size)13 Log2(Batch Size)1010Log2(Batch Size)102.5\r\n Estimated based on empirical results (Section 4.5). 14 5 6 7 89 101112 0 16 32 64 100 500 900\r\n Table 3: Hardware platforms under study. Log2(# Nodes) Filters Embeddingsizes\r\n coresand32threads. Ithaslargememory(120GB)andlow- (a) FC (b) CNN (c) RNN\r\n est peak \ufb02ops (2 TFLOPS) among the three. GeekBench 4 BatchNode BatchFilter EmbeddingBatch\r\n produced the bandwidth measurement. OutputLayer BlockInput MaxlengthLayer\r\n GPUPlatformTheGPUisanNVIDIAV100inaDGX-1 Input Output Vocab\r\n GPUplatformthat contains 8 V100 packages (SXM2) con- 1.0LR Weights0.50.00.51.01.0LR Weights0.50.00.51.01.0LR Weights0.50.00.51.0\r\n nected via 300 GB/s NVlink 2.0 interconnect. We currently (d) FC (e) CNN (f) RNN\r\n measure the performance of a single SXM2 node. One node Figure 2: (a)\u2013(c) ParaDnn\u2019s FLOPS utilization and (d)\u2013(f) its\r\n has 16 GB of memory and 900 GB/s memory bandwidth. A sensitivity to hyperparameters.\r\n V100has640tensorcoresandisabletorunmixedprecision motivated by the observation that with conventional CPUs,\r\n training using \ufb02oat16 to compute and \ufb02oat32 to accumulate, Google would have had to double their datacenter footprint\r\n making its peak performance 125 TFLOPS. to meet the internal demand for DL workloads. Google has\r\n TPUPlatform We use Cloud TPU v2 instances to which been using TPUs for their large-scale production systems,\r\n we were given academic access in February 2018. Each including Search, Translate, and Gmail. Analyzing the ar-\r\n TPUboardcontains four TPU packages (the default con\ufb01g- chitecture of such systems can provide valuable insights\r\n uration) (Dean, 2017). One package contains 2 cores and into future deep learning accelerator design.\r\n one core has one matrix unit (MXU). A Cloud TPU v2 plat- In this section, we use our methodology to study the per-\r\n form supports 180TFLOPS at peak. Memory size is 8GB formance characteristics of TPU v2 and v3 (Dean, 2017;\r\n per core, or 64GB per board, with 2400GB/s overall mem- Google, 2018), with a focus on v2, from the computation\r\n ory bandwidth. TPU v2 supports mixed precision training capability of the core to system balance. We show that\r\n using b\ufb02oat16 and \ufb02oat32. TPU v3 has twice the number of ParaDnncanreveal system bottlenecks in a more compre-\r\n MXUsandtwicetheHBMcapacitypercoreofv2(Google, hensive way than real-world models by probing upper and\r\n 2018). Its memory bandwidth has not been disclosed, but lower system limits. Based on such observations, we dis-\r\n empirical results show that it has increased by 1.5\u00d7. TPU cuss possible steps to improve TPU performance, which can\r\n v3hasapeakof420TFLOPS,2.3\u00d7greaterthanv2. be generalized to other deep learning accelerator systems.\r\n This is the \ufb01rst research paper to study TPU v2/v3, which Observations 1 to 5 in Table 1 summarize our key \ufb01ndings.\r\n supports training, while TPU v1 only runs inference (Jouppi 4.1 FLOPSUtilization\r\n et al., 2017). To enable training, TPU v2 supports more WeuseourmethodologytostudytheTPU\u2019s\ufb02oating-point\r\n operations than matrix multiplication such as gradient and operations per second (FLOPS) utilization, which is the\r\n various optimizer operations. It also carries more pressure ratio of workload average FLOPS to platform peak FLOPS,\r\n on the memory system, since weights are accessed a second measuring how ef\ufb01ciently the computation capacity of a\r\n time in the backward pass. Also, TPU v2 has scalar/vector platform is used. We measure the FLOPS of ParaDnn mod-\r\n units, which do not exist in v1. TPU v2 has MXUs of size els sweeping hyperparameters listed in Table 2. To visualize\r\n 128\u00d7128with32-or16-bitdatatypes;v1has256\u00d7256and FLOPS,weuseheatmaps.\r\n 8 bits. HeatMapsFigures2(a)\u2013(c)present heat maps of FLOPS\r\n Understanding TPUMemorySizeTheTPUimplements utilization for FC, CNN, and RNN ParaDnn models. For\r\n data parallelism by splitting each batch of training data each model type, we choose two hyperparameters (as de-\r\n evenly among the 8 cores. Every TPU core keeps a whole scribed below) that affect FLOPS utilization the most,\r\n copy of the model. Therefore memory size per core deter- sweeping their ranges to create a map grid while keeping\r\n mines the maximum model supported (Sec 5.1), while total other hyperparameters \ufb01xed. FLOPS utilization of all three\r\n memorydetermines the maximum batch size (Sec 5.2). model types increases with batch size, indicating that the\r\n Comparison Rationale One V100 package and one TPU TPUiscapableofleveraging the parallelism within a batch.\r\n board (4 packages) are the minimal units available. On FLOPSutilization of FCs also increases with node count\r\n CloudTPU,distributionofcomputationacrossitsfourpack- per FC layer; that of CNNs also increases with \ufb01lter count;\r\n ages happens automatically, while multi-GPU performance and that of RNNs, with embedding size. So the TPU also\r\n depends largely on user\u2019s implementation. Conclusions exploits parallelism within the widths of the models.\r\n here do not apply to systems with multiple GPUs or TPU Quantifying with Linear Regression To quantify these\r\n boards (Chao & Saeta, 2019). effects, we use the weights of a linear regression (LR) model.\r\n 4 TPUPERFORMANCEIMPLICATIONS For FC, the LR model is\r\n AstheendofDennardscalingandMoore\u2019slawhasslowed FLOPS= w \u00d7layer+w \u00d7node+\r\n the performance improvement of general-purpose micropro- 0 1\r\n w \u00d7input+w \u00d7output+w \u00d7batchsize,\r\n cessors (Dean et al., 2018), the design of DSAs is becoming 2 3 4\r\n more and more relevant. The TPU is a prominent exam- where w \u2013w are hyperparameter weights. When training\r\n 0 4\r\n ple (Jouppi et al., 2017; Dean, 2017). Its development was the LR model, we normalize weights to the same scale, so\r\n ASystematicMethodologyforAnalysisofDeepLearningHardwareandSoftwarePlatforms\r\n 5 node-128 105\r\n that weight re\ufb02ects importance. For example, a positive w 10 node-512 Op % of\r\n 1 node-2048 Op Name\r\n node-8192 104 Transformer\r\n value shows that node count affects performance positively. Transformer |\r\n Small FC 3 Fused MatMul |66.0%\r\n 104 10 |\r\n Loop Fusion | 7.0%\r\n Figures 2(d)\u2013(f) show the LR weights of the model hyper- |\r\n 2 CrossReplicaSum 3.9%|\r\n GFLOPS 10 |\r\n parameters. Batch size and model width have the highest Input Fusion | 9.0%\r\n |\r\n 3 |\r\n 10 1 RMSProp N/A |\r\n absoluteweights,shownonthex-andy-axesinFigures2(a)\u2013 bs: 51216k 10 |\r\n (c). Figure 2(d) shows that the FLOPS utilization of FCs is 100 101 102 101 100 101 102 103\r\n largely affected by batch size and node count, while layer Floating Ops/Byte Floating Ops/Byte\r\n count, and output and input unit counts do not matter as (a) FC b\ufb02oat16 (b) FC Op Breakdown\r\n much. Similarly, Figure 2(e) shows that \ufb01lter count and filter-16 105\r\n 105 filter-32\r\n batch size are most important for CNNs. For RNNs, utiliza- filter-64 104 Op % of\r\n tion is most affected by batch and embedding sizes. ResNet-50 Op Name ResNet-50\r\n Small CNN 103 |\r\n 4 Fused MatMul 85.2%|\r\n 10 |\r\n TakeawaysParaDnnenablessystematic study of hyperpa- 2 Loop Fusion | 9.0%\r\n GFLOPS 10 |\r\n MaxPoolGrad 2.9% |\r\n |\r\n rameter sensitivity and shows that it is natural for a ML sys- |\r\n 101 CrossReplicaSum 1.1%|\r\n |\r\n temtoutilize parallelism arising from large batch size and 103\r\n model width. It is especially intuitive to map batch size and 100 101 102 10 1 100 101 102\r\n modelwidthtothetwodimensionsofsystolicarrays. Mean- Floating Ops/Byte Floating Ops/Byte\r\n while, parallelism opportunities opened by large numbers of (c) CNN b\ufb02oat16 (d) CNNOpBreakdown\r\n layers remain to be explored via model parallelism (Dean Figure 3: TPU roo\ufb02ines for FCs and CNNs. (a) and (c): ParaDnn\r\n and real-world models. (b) and (d): their operation breakdown.\r\n et al., 2012; Jia et al., 2018) and pipelining (Blog, 2019). large batch sizes increase the computation per weight/acti-\r\n 4.2 Roo\ufb02ineModelAnalysis vation transferred, i.e, the arithmetic intensity. Speci\ufb01cally,\r\n The computation capacity of the TPU\u2019s core is only one FCs with \u2265 2k nodes per layer and \u2265 8k batch size are\r\n source of its performance. Memory bandwidth also has a compute-bound. Transformer is close to compute-bound\r\n signi\ufb01cant impact. In this section, we apply the roo\ufb02ine and it uses 4k batch size, so it overlaps with FCs having 4k\r\n model (Williams et al., 2009) to ParaDnn FCs and CNNs batch sizes. Figure 3(c) shows that models close to ResNet-\r\n to analyze the TPU\u2019s computation and memory bandwidth. 50 are compute-bound, while a majority of the CNNs are\r\n We omit RNN models because the TPU pro\ufb01ler reports bottlenecked by memory bandwidth. The CNNs\u2019 higher\r\n incorrect numbers for RNN memory bandwidth. FLOPScomesfromhigherarithmetic intensity caused by\r\n TheRoo\ufb02ineModelFigure3showstheroo\ufb02ineplots. The more\ufb01lters. Whenmemorybandwidthisthebottleneck,the\r\n y-axis is FLOPS and the x-axis is arithmetic intensity, i.e., waytoincrease FLOPS is to increase arithmetic intensity.\r\n \ufb02oating-point operations per byte transferred from memory. Figures 3(b) and 3(d) show the TensorFlow operations that\r\n Theroo\ufb02ine (the red line in Figure 3) has of a slanted part take more than 1% of the workload execution time and more\r\n and a horizontal part. It represents the highest achievable than 0 FLOPS. The arithmetic intensity of such operations\r\n FLOPSatagivenarithmetic intensity. Any data point (x,y) can be as low as 0.125.1 The TensorFlow breakdown in\r\n on the slanted part has x = memory bandwidth. The hor- Figure 3 is generated after operation fusion, which com-\r\n y\r\n izontal part is the hardware peak FLOPS. A workload or bines and executes several operations together for higher\r\n operation (a point in Figure 3) close to the slanted roo\ufb02ine is ef\ufb01ciency. In Figures 3(b) and 3(d), the only compute-bound\r\n memory-bound; one close to the horizontal part is compute- operation is large fused MatMul (MatMul fused with other\r\n bound. A workload or operation not close to the roo\ufb02ine operations), so a compute-bound model needs large Mat-\r\n stresses neither memory interconnect nor compute units. Muls. Other operations are closer to the slanted line, con-\r\n Figures 3(a) and 3(c) show all the ParaDnn FCs and CNNs strained by memory bandwidth. Transformer and ResNet-\r\n (dots) plus Transformer and ResNet-50 (stars). Figures 3(b) 50 are compute-bound (Figures 3(a) and 3(c)) because they\r\n and 3(d) show all the operation breakdowns. The triangles have compute-bound MatMuls (Figures 3(b) and 3(d)).\r\n in Figures 3(a) and 3(c) are selected memory-bound models. Real-WorldModelAnalysisParaDnnandreal-worldmod-\r\n Thedesign space shown with roo\ufb02ine models indicates that els are complementary. By analyzing ParaDnn, we ex-\r\n ParaDnn is a superset of the real-world models. ParaDnn plore the design space and reach the limits of platforms.\r\n models span a much larger range in the design space, from Analyzing real-world models puts the design space study\r\n extremely memory-bound to compute-bound. Therefore into realistic context by highlighting popular representa-\r\n performance analysis with ParaDnn can comprehensively tive designs. The tables in Figure 3 show the operation\r\n test the limits of platforms in both extremes. An exception breakdown of Transformer and ResNet-50, and indicate\r\n is that some operations of Transformer do not align closely that even compute-bound models contain a noticeable frac-\r\n with those of FCs. This results from a choice in this paper: tion of memory-bound operations. Transformer has three\r\n ParaDnn uses the RMSProp optimizer, keeping nodes per memory-boundoperations: (1) input fusion (9.0%), which\r\n layer uniform for FCs, while Transformer uses the adafactor includes multiply, subtract, and reduce; (2) loop fusion\r\n optimizer and has layers with 4k, 2k, and 512 nodes. (7.0%), which consists of control \ufb02ow operations (e.g., se-\r\n ParaDnn Analysis We \ufb01rst discuss the insights enabled lect and equal-to); and (3) CrossReplicaSum (3.9%), which\r\n by ParaDnn, of which the real-world models are a sub- sumsupthevaluesacross multiple weight replicas. These\r\n set. Figure 3(a) shows that large batch sizes make FCs 1Anactivation accumulation operation (CrossReplicaSum in\r\n more compute-bound, and more nodes make FCs more TensorFlow) uses \ufb02oat32 even with b\ufb02oat16 model weights. In\r\n memory-bound. That is because FCs with more nodes need this case, the arithmetic intensity is 1/(2\u00d74 bytes) = 0.125, i.e.,\r\n to transfer more weights/activations from the memory, and one \ufb02oating-point addition for every two data points loaded.\r\n ASystematicMethodologyforAnalysisofDeepLearningHardwareandSoftwarePlatforms\r\n 100 100\r\n bs-2048 bs-128\r\n 80 bs-4096 80 bs-256 40\r\n bs-8192 bs-512\r\n 60 bs-16384 60 filter-16 ResNet\r\n 40 40 FLOPS %20 filter-32 DenseNet\r\n filter-64 MobileNet\r\n 8-Core FLOPS%20 8-Core FLOPS%20 0 RetinaNet SqueezeNet\r\n 0 0 20 40 60 80\r\n 0 1-Core FLOPS%20406080100 201-Core FLOPS%406080100 Infeed %\r\n (a) FC (b) CNN Figure 5: The FLOPS utilizations and data infeed percentages of\r\n Figure 4: Communication overhead in a multi-chip system is ParaDnn (dots) and real-world (stars) CNNs.\r\n non-negligible, but is reduced with large batch sizes. lization. Speci\ufb01cally, FCs with \u2265 256 nodes and \u2264 512\r\n three operations contribute 19.9% of the total execution time. batch size run faster on a TPU with one core than on one\r\n (12.3% of the execution time is for data formatting, which with eight. Thus we consider FCs with larger than 1024\r\n has no arithmetic intensity or TPU FLOPS.) ResNet-50 has batch size in Figure 4. Based on Amdahl\u2019s law, the maxi-\r\n memory-bound loop fusion (9%), MaxPoolGrad (2.9%), mumnon-parallel fraction of the workloads is up to 60%\r\n andCrossReplicaSum(1.1%),whichsumsto13%,showing for FC and up to 40% for CNN. Using the largest batch\r\n the need for end-to-end optimization for DL accelerators. size shown in Figure 4, the 90th-percentile of non-parallel\r\n TakeawaysParaDnnexplores the design space and stresses fractions are 16% for FC and 8.8% for CNN.\r\n platform limits; real-world models represent the currently TakeawaysWithdiverse ParaDnn models, we observe that\r\n important design points. ParaDnn shows that the de- communication overhead in multi-chip systems is non-\r\n sign space is composed of very diverse models, from ex- negligible even for large FCs and CNNs. Using large batch\r\n tremely memory-bound to compute-bound; real-world mod- size can reduce overhead by increasing parallel computation\r\n els show that even compute-bound models contain non- without increasing weight transfers. Possible optimizations\r\n negligible fractions of memory-bound operations (19.9% include relaxed synchronization, model parallelism (Dean\r\n for Transformer and 13% for ResNet-50), which suggests et al., 2012), gradient compression (Lin et al., 2017b), and\r\n that memory bandwidth can affect other ML systems orig- weight pruning and compression (Han et al., 2015).\r\n inally designed to optimize computation. Researchers 4.4 Host-Device Balance\r\n can test system memory-boundness with ParaDnn. Ap- Previous subsections have focused on the performance of\r\n proaches for speeding up memory-bound operations in- the accelerator itself. We now turn to \u201cdata infeed,\u201d the\r\n clude caching (Hennessy & Patterson, 2011), operation process of preparing and moving input data to the TPU\r\n fusion (xla, 2018; Chen et al., 2018; Rotem et al., 2018), board. ParaDnn in other sections uses data synthesized\r\n aggressive data quantization (Banner et al., 2018), and com- from CPUhosts, which avoids most of the data infeed over-\r\n pression (Han et al., 2015; Lin et al., 2017b). head. HereweuseParaDnnCNNmodelswiththeImageNet\r\n 4.3 Multi-Chip Overhead dataset (Krizhevsky et al., 2012).\r\n ComputingspeedandmemorybandwidthofaTPUcoreare The TPU system includes a CPU host and a TPU de-\r\n not the only factors affecting training performance, because vice (Google, 2018). For image datasets, the host fetches\r\n typical large-scale systems use multiple chips (Dean et al., images from the network, decodes and preprocesses them,\r\n 2012). This section evaluates the scalability of a multi-chip and feeds them to the device. We refer this as data prepa-\r\n TPU system with ParaDnn. We quantify the multi-chip ration. The device then performs training computation on\r\n overhead by comparing the FLOPS utilization of 1-core the images. Data infeed includes network overhead, host\r\n (x-axis) and 8-core TPUs (y-axis) in Figure 4. If there were compute, and transfer between host and device.\r\n no multi-chip overhead, FLOPS utilization of 1-core and For each ParaDnn CNN and the ImageNet dataset, we use\r\n 8-core should be the same, i.e., all points should lie on the the TPU pro\ufb01ler to collect FLOPS utilization and infeed\r\n dashed line in Figure 4 showing x = y. time percentage, which is the fraction of time the acceler-\r\n ParaDnnallows us to explore models with a wide range of ator spends waiting for data. Figure 5 shows the results as\r\n communication overhead. Figure 4 shows that an 8-core dots, along with the real-world CNNs as stars. ParaDnn\r\n TPU exhibits noticeably lower FLOPS utilization than a models are very diverse, ranging from 0 to 50% FLOPS\r\n 1-core TPU, re\ufb02ecting signi\ufb01cant inter-core communication utilization and 0 to 90% infeed time. ParaDnn shows that\r\n overhead. For FC, the maximum FLOPS utilization in an 8- many CNNs have signi\ufb01cant infeed time and that larger\r\n coreTPUis62%,comparedto100%ina1-coreTPU.Multi- CNNstendtohavelowerinfeedtime. LargeCNNs,those\r\n chip overhead is less noticeable in CNNs, with FLOPS with more \ufb01lters and/or more layers, are the most suitable\r\n utilization decreasing from 55% in the 1-core TPU to 40% for the TPU system, because the accelerator spends more\r\n in the 8-core. It is worse for FCs because there are more time training each image and CPU infeed time per image\r\n weights to synchronize across cores than for CNNs. is \ufb01xed. The high-performance TPU system targets large\r\n Our analysis method indicates that large workloads can workloads. Consistent with the communication overhead\r\n amortize the parallelism overhead, and it highlights batch study in Section 4.3, small workloads do not have enough\r\n size as the key hyperparameter that affects communication parallelism to utilize the TPU ef\ufb01ciently.\r\n overhead. Increasing batch size reduces the FLOPS uti- Somerealmodelsshowopportunities to optimize for data-\r\n lization gap by increasing computation without increasing infeed bottlenecks. SqueezeNet has the highest infeed time\r\n weight synchronization. On the 8-core TPU, FCs need at because it is designed to accommodate mobile devices by\r\n least 16k batch size to achieve more than 50% FLOPS uti- using small numbers of \ufb01lters per layer. While MobileNet\r\n ASystematicMethodologyforAnalysisofDeepLearningHardwareandSoftwarePlatforms\r\n Mem-Bound | | Mem-Bound | | L 6og2(e xa m p les /s5.5) L6og2 (ex a m p les/s5.5) L6og2 (ex am p les/s5.5)\r\n 3.0 3.0 7 5.0 7 5.0 7 5.0\r\n 2.5 2.32.5 8 4.5 8 4.5 8 4.5\r\n 2.3 9 4.0 9 4.0 9 4.0\r\n 2.0 2.0 10 3.5 10 3.5 10 3.5\r\n 1.5 3.0 11 3.0 11 3.0 11 3.0\r\n 1.5 2.5 12 2.5 12 2.5 12 2.5\r\n 1.0 2.3 13 2.0 13 2.0 13 2.0\r\n FLOPS v3/v21.0 MatMul 0.5 MatMul 2.0 Log2(Batch Size)14 1.5 Log2(Batch Size)14 1.5 Log2(Batch Size)14 1.5\r\n Speedup1.5 7 8 910 111213 1.0 7 8 9 10 111213 1.0 7 8 9 1011 1213 1.0\r\n 0.510 1 100 101 102 0.0 10 1 100 101 102 Log2(# Nodes) Log2(# Nodes) Log2(# Nodes)\r\n Floating Ops/Byte Floating Ops/Byte 1.0 FC CNN RNN (a) CPU (b) GPU (c) TPU\r\n (a) FC Ops (b) CNNOps (c) TPU v3 over v2 Figure 7: Examples/second of ParaDnn FC models with \ufb01xed\r\n Figure 6: Speedup of TPU v3 over v2 for (a) FC operations, depth (64). Larger memory allows the CPU to run larger models.\r\n (b) CNN operations, and (c) ParaDnn models. The red line ment estimate is corroborated by the 1.5\u00d7 speedup of other\r\n (75ops/byte) is the in\ufb02ection point in the TPU v2 roo\ufb02ine (Fig 3). memory-boundoperations, represented by the non-MatMul\r\n FCoperations in the lower left corner of Figure 6(a). The\r\n also targets mobile devices, it has one convolution layer that performance of those operations does not increase with\r\n uses up to 1k \ufb01lters, eliminating time lost to data infeed. larger batch size, as shown by the vertical alignment of each\r\n RetinaNet, ResNet, and SqueezeNet show potential for im- operation type in Figure 3(b). Thus the 1.5\u00d7 speedup in\r\n provement through system optimizations. The performance Figure 6(a) is from bandwidth improvement.\r\n without data preparation shows that resolving the infeed Boundary Cases The compute-bound MatMuls in Fig-\r\n bottleneck can lead to 37%, 34%, and 180% performance ure 6(b) become memory-bound on TPU v3, so the speedup\r\n improvement, respectively. is < 2.3\u00d7. Such operations have arithmetic intensity be-\r\n Takeaways ParaDnn shows the design space of FLOPS tween 75 and 117, because the roo\ufb02ine in\ufb02ection point of\r\n and data-infeed time, and reveals that large workloads with v3 is at x = 420/(2.4\u00d71.5) = 117. CrossReplicaSum (yel-\r\n abundantparallelism are not host-bound on large accelerator lowdots)issloweddownonTPUv3,whichmaybebecause\r\n systems. Real-world models show that the performance of more replicas across more MXUs.\r\n of some workloads can be improved. When designing an End-to-EndModelsInFigure6(c)themaximumspeedups\r\n accelerator system, scaling performance of the CPU host are 2.83\u00d7 (FC), 2.31\u00d7(CNN), and 3.11\u00d7(RNN). Speedup\r\n to match the accelerator is crucial for utilization of the increases with model width (second column of Table 2), and\r\n accelerator\u2019s computation resource. the maximumspeedupisachievedbythelargestwidth. FCs\r\n 4.5 TPUv3 with close to 3\u00d7 speedup are dominated by memory-bound\r\n In this section, we systematically quantify the differences MatMuls. Exceptions are RNNs with more than 3\u00d7; these\r\n between TPU v2 and v3. Figure 6 compares the two us- have the largest embedding size (900), indicating that TPU\r\n ing ParaDnn (dots), plus ResNet and Transformer (stars). v3optimizes large embedding computations.\r\n Batch size for v3 is twice that for v2, thanks to its doubled Takeaways ParaDnn allows examining new platforms\r\n memorycapacity. Figures 6(a) and 6(b) use a variation of with a wider range of workloads, from memory-bound to\r\n the roo\ufb02ine model, showing arithmetic intensity on the x- compute-bound, than using real-world models alone. Com-\r\n axis and operation speedup on the y-axis. Data point colors paring TPU v3 to v2 as an example, ParaDnn can show the\r\n representing operation types are consistent with those in system upgrade bene\ufb01ting operations with different arith-\r\n Figures 3(b) and 3(d). As a reference, the red dashed line metic intensity. TPU v3 shows three main levels of speedup:\r\n is the in\ufb02ection point in the TPU v2 roo\ufb02ine from Figure 3, 2.3\u00d7forcompute-boundoperations, 3\u00d7 for memory-bound\r\n where arithmetic intensity is 75ops/byte (180TFLOPS / MatMuls, and 1.5\u00d7 for other memory-bound operations.\r\n 2.4TB/s). The operations on the left of the red line are This is the result of its 2.3\u00d7 FLOPS, 2\u00d7 memory capacity,\r\n memory-bound; those on the right are compute-bound. We and 1.5\u00d7 memory bandwidth.\r\n group the operations in four classes, as follows. 5 CROSS-PLATFORMCOMPARISON\r\n Compute-BoundOpsThepeakFLOPSofTPUv3is2.3\u00d7 In this section, we show ParaDnn\u2019s utility in cross-platform\r\n that of v2, so the performance of compute-bound operations comparison, with CPU, GPU, and TPU as exemplars along\r\n is improved by about 2.3\u00d7 on v3. Such operations are on the continuum between general purpose processors and\r\n the right of the red dashed line in Figure 6(b). specialized accelerators. ParaDnn shows the sensitivity\r\n Memory-Bound Ops (2\u00d7 batch size) The maximum of speedup to model hyperparameters, allowing users to\r\n speedup of the memory-bound operations (mainly the Mat- choose platforms based on model hyperparameters of inter-\r\n Muls in Figures 6(a) and 6(b)) is 3\u00d7. Tripled speedup est, rather than on model characterizations that happen to\r\n comes from doubled batch size (owing to doubled mem- have been reported. ParaDnn also reveals the fundamental\r\n ory capacity) and memory bandwidth improvement. The architectural differences between platforms and shows the\r\n memory bandwidth increase of v3 over v2 has not been trade-offs between \ufb02exibility and specialization.\r\n of\ufb01cially disclosed, but we can estimate it. Doubled batch \u2022 TheTPUishighly-optimizedforlargebatchesandCNNs,\r\n size means doubled arithmetic intensity. On the slanted and has the highest training throughput.\r\n line of a roo\ufb02ine model, that means doubled FLOPS, be- \u2022 TheGPUismore\ufb02exibleandprogrammableforirregular\r\n cause the ratio of FLOPS to arithmetic intensity is \ufb01xed. computations, such as small batches and non-MatMul\r\n Switching from v2\u2019s roo\ufb02ine to v3\u2019s thus increases FLOPS operations. Training of large FC models bene\ufb01ts from its\r\n by twice the bandwidth improvement. So the 3\u00d7 overall sophisticated memory system and higher bandwidth.\r\n speedup suggests that v3 bandwidth improvement over v2 \u2022 The CPU is the most programmable, so it achieves the\r\n is 3/2 = 1.5\u00d7, to 3.6TB/s. highest FLOPS utilization for RNNs, and it supports the\r\n OtherMemory-BoundOpsThe1.5\u00d7bandwidthimprove- largest model because of its high memory capacity.\r\n ASystematicMethodologyforAnalysisofDeepLearningHardwareandSoftwarePlatforms\r\n 101 FC: TPU/GPU 101 FC: TPU/GPU 103FC: GPU/CPU\r\n node-64\r\n node-256\r\n FC Speedup GPU/CPU 102 node-1k\r\n FC Speedup TPU/GPU 100 bs-1k 100 node-512 Node node-4k\r\n Batch bs-2k node-1k Layer 101\r\n Layer bs-4k node-2k Batch\r\n Output Speedupsbs-8k node-4k Input Speedups\r\n Input bs-16k 1 node-8k 0\r\n Node 10 1 7 8 10 7 8 Output 10\r\n 10 10 10 10 1.0 0.50.0 0.51.0 107 109\r\n 1.0LR Weights0.50.00.51.0Params Params LR Weights Params\r\n (a) LR Weights (b) Batch Size (c) Node (a) LR Weights (b) Node\r\n Figure 8: (a) Sensitivity analysis of TPU over GPU speedups. Figure 9: (a) The sensitivity analysis of (b) GPU over CPU\r\n Speedupscolor-coded by (b) batch size and (c) FC nodes per layer. speedups for FCs.\r\n 5.1 Fully-Connected DNNs\r\n Examples/second measures the number of examples trained it is optimized for large FC memory reuse and streaming\r\n per second, a proxy for end-to-end performance. Heat maps requirements. This is shown by the negative weights of node\r\n in Figure 7 compare ParaDnn FCs for three platforms, with count, layer count, and input size in Figure 8(a) and the\r\n varying node counts and batch sizes but \ufb01xed layer count trend in Figure 8(c), corroborated by the overall negatively-\r\n (64). WeuseLRweightsfromSection4.1toquantifythehy- correlated trend of speedup with parameter count in Figure 8.\r\n perparameter effects (not shown owing to space limitations). FCshaveminimalweightreuseandlargemodelshavemore\r\n Layer and node counts have negative weights because it weights, so they put a lot of pressure on the memory system.\r\n is more time-consuming to train larger models with many The GPU has a more mature memory system and higher\r\n layers and nodes. Batch size greatly improves throughput on memory bandwidth than the TPU, which makes it better\r\n the GPU and TPU, but not the CPU because the parallelism suited to the memory requirements of large FCs.\r\n available with small batch sizes can fully utilize a CPU. GPUoverCPUFigure9(a)showstheLRweightsofGPU-\r\n InFigure7,thewhitesquaresindicatemodelsthatencounter over-CPU speedup. Figure 9(b) shows the design space\r\n out-of-memory issues. Only the CPU supports the largest color-coded by node count. The GPU is a better platform\r\n models, and the GPU supports larger models than the TPU. for large FCs because its architecture can better exploit the\r\n This is because every hardware core keeps one copy of the parallelism available with large batches and models. Recall\r\n model, so memory per core determines the largest model from Figure 8 that large FCs prefer the GPU over the TPU.\r\n supported, as explained in Section 3. The CPU has the SotheGPUisthebestplatformforlargeFCs,butmodels\r\n highest memory per core (120GB), and the GPU (16GB) with large batches perform best on the TPU.\r\n is higher than the TPU (8GB). While TPUs and GPUs 5.2 CNNandRNN\r\n maydraw more attention, as of today the only choice for\r\n extremely large models is the CPU, which supports all WenowdescribethespeedupofParaDnnCNNsandRNNs.\r\n model sizes. For example, Facebook uses dual-socket CPU Since our conclusions for CPUs are similar to those in the\r\n servers with large memories to train ranking models (FC previous section, we omit them in the interest of brevity.\r\n networks) (Hazelwood et al., 2018). That fact highlights CNNFigures 10(a)\u201310(c) show the speedups of the TPU\r\n the need for model parallelism and pipelining on the GPU over the GPU. All CNNs perform better on the TPU. Batch\r\n and TPU(Deanetal., 2012; Jia et al., 2018; Blog, 2019) to size is still the key to better TPU-over-GPU speedup for\r\n allow those powerful platforms to support larger models. CNNs,shownbyitspositive LR weight in Figure 10(a) and\r\n TPU over GPU To further investigate the best hardware the increasing speedup with batch size in Figure 10(b). The\r\n platform for FC models, we analyze TPU over GPU TPUisthebestplatform for large CNNs, suggesting that its\r\n speedups. Figure 8(a) plots the linear regression weights architecture is highly optimized for the spatial reuse char-\r\n across FC hyperparameters for TPU over GPU speedup. acteristics of CNNs. This is shown by the positive weights\r\n Figures 8(b) and 8(c) show the design space for FCs as scat- in Figures 10(a) and 10(c), where models with more \ufb01lters\r\n ter plots, with numbers of model parameters on the x axis and blocks have higher speedups. It is different from Sec-\r\n and speedups on the y axis. To display the effects of hy- tion 5.1, showing that the TPU is not preferred for large FCs.\r\n perparameters, we color-code data points to re\ufb02ect batch This suggests that the TPU handles large CNNs better than\r\n size (Figure 8(b)) and node count per layer (Figure 8(c)). large FCs, because CNNs reuse weights, but FCs seldom\r\n Overall, 62% of the FCs perform better on the TPU. do, which results in greater memory traf\ufb01c. The GPU is a\r\n The TPU is well suited for large batch training because feasible choice for small CNNs. These conclusions only\r\n systolic arrays excel at increasing throughput (Kung, 1982). apply to a single GPU; multi-GPU cases may be different.\r\n The positive weight of batch size in Figure 8(a) and the RNNFigures10(d)\u201310(e) show the speedup of TPU over\r\n horizontal color bands in Figure 8(b) indicate that large GPU. We display the embedding size in Figure 10(e) be-\r\n batch size is the key to higher speedup. This suggests that cause the magnitude of its weight is the greatest in Fig-\r\n the TPU MXUs, implemented with systolic arrays, need ure 10(d). Embedding size has negative weight, and embed-\r\n large batches to reach full utilization. The GPU is a better ding computation is more sparse than matrix multiplication.\r\n choice for small batches, because it executes computation in This suggests that the TPU is less \ufb02exible for doing non-\r\n warps,packingsmallbatchesandschedulingthemonstream MatMul computations than the GPU. The TPU is better\r\n multiprocessors more easily (Nickolls & Dally, 2010). at dense computations like MatMuls. Even so, RNNs are\r\n TheGPUisabetterchoiceforlargemodels,suggestingthat still up to 20\u00d7 faster on the TPU. Optimizing non-MatMul\r\n computations is another opportunity for TPU enhancement.\r\n ASystematicMethodologyforAnalysisofDeepLearningHardwareandSoftwarePlatforms\r\n CNN: TPU/GPU CNN: TPU/GPU TPU/GPU Speedups\r\n 6\u00d710 0 6\u00d7 100\r\n 101 Google\r\n 4\u00d710 0 4\u00d7 100\r\n 3\u00d710 0 3\u00d7 100 Nvidia\r\n CNN Speedup TPU/GPU 100\r\n Batch 2\u00d710 0 bs-128 2\u00d7 100\r\n Filter bs-256 filter-16\r\n Block Speedups bs-512 Speedups filter-32 FC Res Bottle RNN LSTM GRU Trans Retina ResNet Dense Squeeze\r\n Output 10 0 bs-1k 1 00 filter-64\r\n Input 106 10 7 10 8 10 6 10 7 10 8 FLOPS Utilization %\r\n 1.0 LR Weights0.50.00.51.0 Params Params 60\r\n (a) LR Weights (b) Batch Size (c) Filter Size 40\r\n RNN: TPU/GPU 20\r\n 0 C G T G T G T G T G T G T G T G T C G T C G T C G T\r\n 10 1 FC Trans Res Bottle Retina ResNet Dense Squeeze RNN LSTM GRU \r\n RNN Speedup TPU/GPU Figure 11: (Top) TPU over GPU speedups of all workloads. (Bot-\r\n Maxlength tom) FLOPSutilization comparison for all platforms.\r\n Batch embed-100\r\n Vocab Speedups embed-500\r\n EmbeddingLayer 10 0 embed-900\r\n 105 106 10 7 RNNoptimizations may be able to increase utilization on\r\n 1.0 LR Weights0.50.00.51.0 Params the GPU and the TPU. Real models with more \ufb01lters have\r\n (d) LR Weights (e) Embedding Size higherFLOPS,whichiswhyResNet-50andRetinaNethave\r\n Figure 10: The sensitivity analysis and speedups of TPU over higher FLOPS than DenseNet and SqueezeNet.\r\n GPUfor(a)\u2013(c) CNNsand(d)\u2013(e) RNNs.\r\n 5.3 Overall Comparison 6 SOFTWARESTACKADVANCES\r\n This section summarizes the speedup of TPU over GPU and CustomDLhardwareopensopportunities for dramatic soft-\r\n the FLOPSutilization of all ParaDnn and real models. We ware optimizations. ParaDnn is also useful for comparing\r\n useboxplots(Figure11)tosummarizeeachParaDnnmodel software performance, by analyzing the performance of dif-\r\n type because performance has a wide range. The bar in the ferent TensorFlow (TF) and CUDA versions. We study data\r\n box shows the median. The upper and lower boundaries type quantization with software versions, because it depends\r\n of the box are 9th and 91st percentiles. The upper and onsoftware support. Software versions are summarized in\r\n lower bars outside of the box are 2nd and 98th percentiles. the legend of Figure 12. ParaDnn allows more compre-\r\n Outliers are shown as dots. We do not show the results hensive analysis of software updates than real models. It\r\n of using CPUs to train CNNs because it is extremely time can also reveal software optimization focus (e.g., TF 1.9\r\n consuming and unlikely to contribute additional insights. optimizes small batches); we omit these details for brevity.\r\n TPUover GPUSpeedup Figure 11(top) summarizes the 6.1 TensorFlowVersions and TPUPerformance\r\n TPUoverGPUspeedupsofallmodels. Notethat the real Thecompiler for the TPU is XLA (Leary & Wang, 2017),\r\n models use larger batch sizes on TPU than on GPU. The shipped with TF. Figure 12(top) shows TPU speedups ob-\r\n speedup of TPU over GPU depends heavily on the nature tained by running TF 1.7 to 1.12, treating 1.7 with \ufb02oat32\r\n of the workload measured. The speedup of parameterized as the baseline. Moving from TF 1.7 to 1.12 improves per-\r\n models varies widely, from less than 1\u00d7 to 10\u00d7, while the formance for all ParaDnn models. Although FC and CNN\r\n speedup of real workloads ranges from 3\u00d7 (DenseNet) to encounter performance regression with TF 1.8, TF 1.9 \ufb01xes\r\n 6.8\u00d7 (SqueezeNet). ParaDnn represents a more complete this anomaly and improves overall performance. RNN per-\r\n view of potential workloads, and each real workload repre- formance is not improved much until TF 1.11. TF 1.11\r\n sents the concerns of certain users. Benchmarking platforms shows 10\u00d7 speedup for RNNs. Transformer, ResNet-50,\r\n with two kinds of workloads offer a more systematic under- and RetinaNet are improved continuously over TF updates.\r\n standing of their behavior than those with only one kind. Interestingly, SqueezeNet is improved starting from TF 1.11,\r\n Tofurther compare the TPU with the GPU, while relaxing while the performance of DenseNet and MobileNet see little\r\n the constraint on the GPU\u2019s software stack, we also include bene\ufb01t. In the 7 months (222 days) between the release of\r\n speedup relative to the GPU performance of ResNet-50, re- TF1.7.0 (03/29/2018) and that of TF 1.12.0 (11/05/2018),\r\n portedinNVIDIA\u2019sDeveloperBlog(Case,2018)(annotated software stack performance was improved signi\ufb01cantly. The\r\n as NVIDIAinFigure 11(top)). Note that NVIDIA\u2019s version 90th-percentile speedup of TPU is 7\u00d7 for FC, 1.5\u00d7 for\r\n of ResNet-50 uses unreleased libraries, and we were un- Residual CNN, 2.5\u00d7 for Bottleneck CNN, 9.7\u00d7 for RNN,\r\n able to reproduce the results. The speedup using ResNet-50 and 6.3\u00d7 for LSTM and GRU.\r\n from Google is 6.2\u00d7, compared with 4.2\u00d7, which suggests B\ufb02oat16enables signi\ufb01cant performance improvement for\r\n software optimization can signi\ufb01cantly impact performance. ParaDnnFCsandCNNs. 90th-percentile speedups are up\r\n FLOPSUtilization Figure 11(bottom) shows the FLOPS to 1.8\u00d7 for FC and Bottleneck CNN, and 1.3\u00d7 for Residual\r\n utilization of all workloads and platforms. On average, the CNN. TPU can support doubled batch sizes with 16 bits.\r\n maximumFLOPSutilization of the TPU is 2.2\u00d7 that of the Transmitting fewer bits also relieves bandwidth pressure,\r\n GPUforallCNNmodels,andtheratiois3\u00d7forRNNs. The speedingupmemory-boundoperations. Largerperformance\r\n TPUFLOPSutilizationofTransformers is consistent with increases may be possible with further bitwidth reductions.\r\n FCswith4kbatchsize, as in Figure 2. For RNNs, the TPU 6.2 CUDAVersionsandGPUPerformance\r\n has less than 26% FLOPS utilization and the GPU has less\r\n than 9%, while the CPU has up to 46% because of its better Figure 12(bottom) shows GPU performance across versions\r\n programmability. RNNs have more irregular computations of CUDA and TF. The baseline is TF 1.7 and CUDA 9.0\r\n than FCs and CNNs, owing to temporal dependency in with \ufb02oat32. TF 1.8 does not improve GPU performance.\r\n the cells and variable-length input sequences. Advanced Bylowering memory traf\ufb01c and enabling larger batch sizes,\r\n ASystematicMethodologyforAnalysisofDeepLearningHardwareandSoftwarePlatforms\r\n TPU performance vs Tensorflow Versions\r\n 10.0 Baseline: TF1.7 32b7: TF1.7 16b Wefocusonlargebatches, as the platforms were designed\r\n 7.5 8: TF1.8 16b for large batch training, and smaller batches may lead to\r\n 9: TF1.9 16b\r\n 5.0 1: TF1.11 16b different conclusions.\r\n 2: TF1.12 16b\r\n 2.5 8 RELATEDWORK\r\n 0.0789127891278912789127891278912789127891278912789127891278912 This paper presents a performance analysis methodology\r\n FC Res Bottle Trans Retina Resnet Dense Mobile Squeeze RNN LSTM GRU that includes a tool, ParaDnn. The set of models generated\r\n GPU Performance vs CUDA and TF Versions\r\n 6 Baseline: TF1.7 CUDA9.0 32b1: TF1.7 CUDA9.0 16b1.5by ParaDnn is not designed to replace other benchmark\r\n 2: TF1.8 CUDA9.0 16b\r\n 4 3: TF1.8 CUDA9.2 16b suites, but to complement existing suites to study the design\r\n 4: TF1.12 CUDA10 16b space more comprehensively, as discussed in Section 2.\r\n 2 1.0 Domain-speci\ufb01c benchmark suites that are complemen-\r\n 123 4 123 4 1234 1234 1234 1234 1234 123412341234 tary with our methodology include MLPerf (Mattson et al.,\r\n FC Res Bottle Retina Resnet Dense Squeeze RNN LSTM GRU 2019), Fathom (Adolf et al., 2016), CortexSuite (Thomas\r\n Figure 12: (Top) TPU performance with TensorFlow updates. et al., 2014), and (Hauswald et al., 2015a;b; Coleman et al.,\r\n (Bottom) GPU performance with CUDA and TF updates. 2017; Wang et al., 2019c; Tao et al., 2018). Benchmarks\r\n bitwidthreductioncanspeedupCNNsbymorethan2\u00d7. We have been the driving force for compiler and architecture\r\n note that CUDA9.2speedsupResNet-50signi\ufb01cantlymore design for decades, and notable examples include the SPEC\r\n (8%) than other real workloads (< 1%), and it speeds up CPU(Henning,2006)andPARSECmultiprocessorbench-\r\n ParaDnn CNNsmorethanFCsorRNNs. CUDA10speeds marks (Bienia et al., 2008). In the same spirit as param-\r\n upother models, but not SqueezeNet. It signi\ufb01cantly speeds eterized benchmarks, synthetic benchmarks are common,\r\n up ParaDnn FCs and CNNs, but not RNNs. The overall such as BenchMaker (Joshi et al., 2008), SYMPO (Ganesan\r\n 90th-percentile improvement for FCs, CNNs, and RNNs is et al., 2010), AI Matrix (Wei et al., 2018), and (Kim et al.,\r\n up to 5.2\u00d7, 2.9\u00d7, and 8.6%, respectively. CUDA updates 2014; Turki et al., 2012; Stroobandt et al., 2000; Schaffter\r\n have less impact than do TF updates on the TPU, likely et al., 2011; Saleem et al., 2015). Because of self-similarity\r\n because CUDA and GPU platforms have greatly matured of benchmark suites (Wang et al., 2019a), users should care-\r\n since becoming popular before 2010, while TPU v2 for fully select proper benchmarks to work with. Our effort of\r\n training was only announced in May 2017. generating more diverse DL workloads is different from pre-\r\n vious work. Our use of DL models to compare up-to-date\r\n 7 LIMITATIONS OF THIS WORK platforms, Google\u2019s TPU v2/v3, and NVIDIA\u2019s V100 GPU,\r\n ScopeThisworkdoesnotstudyDLinference,cloudover- distinguishes this work from previous performance compar-\r\n head, multi-node systems, accuracy, convergence, or other isons (Shi et al., 2016; Bahrampour et al., 2016; Kothari,\r\n DLframeworks. Speci\ufb01cally, NVIDIA\u2019s eight-node DGX-1 2011; Che et al., 2009; He et al., 2010; Gupta et al., 2019;\r\n or Google\u2019s 256-TPU systems are not studied. We inten- Wangetal., 2019b).\r\n tionally leave these topics to future work, as each deserves 9 CONCLUSION\r\n in-depth study. They need different metrics such as latency, This paper presents a comprehensive performance analysis\r\n and different setups such as tuning numbers of hardware methodology and its utility in deep learning. We conduct\r\n nodes, inter-node bandwidth, and synchronization mecha- case studies to analyze and compare two generations of\r\n nisms. Cloud overhead may be more acute and brings up specialized platforms (TPU v2/v3), three heterogeneous\r\n moreresearch questions. TensorFlow is used because it sup- architectures (TPU, GPU, and CPU), and two specialized\r\n ports all three platforms. Previous work shows that frame- software stacks (TensorFlow and CUDA). The methodol-\r\n workimplementation can largely affect performance (Wang ogyis complementary to traditional performance analysis\r\n et al., 2019b), and using PyTorch, which recently starts to approaches. This paper motivates application of our method-\r\n support the TPU, may lead to different conclusions. ology to other hardware and software systems.\r\n Thevalidity of extrapolating training throughput to time-to- 10 ACKNOWLEDGEMENT\r\n accuracy remains an open question. Recent work studied\r\n the number of training steps to accuracy as a function of the This workwassupportedinpartbyGooglesTensorFlowRe-\r\n batch size (Shallue et al., 2018). It shows that very large search Cloud (TFRC) program, NSF Grant CCF1533737,\r\n batch sizes result in sub-linear scaling, and the best batch and the Center for Applications Driving Architectures\r\n size depends largely on the model and the optimizer. In a (ADA), one of six centers of JUMP, a Semiconductor Re-\r\n multi-node system, synchronization becomes more compli- search Corporation program co-sponsored by DARPA. The\r\n cated, which results in different convergence behaviors. authors would like to thank Frank Chen, Blake Hecht-\r\n Tractability Readers should take caution when making con- man, Jim Held, Glenn Holloway, Dan Janni, Peter Matt-\r\n clusions using a \ufb01nite set of workloads and platforms. While son, Lifeng Nai, David Patterson, Francesco Pontiggia,\r\n this methodologyisdesignedtoavoidover-emphasizingone Parthasarathy Ranganathan, Vijay Reddi, Bjarke Roune,\r\n subset of workloads, we have to constrain the diversity of Brennan Saeta, Zak Stone, Sophia Shao, Anitha Vijayaku-\r\n model architectures and hyperparameters to make this work mar, Shibo Wang, Qiumin Xu, Doe HyunYoon, Cliff Young\r\n tractable. For example, the CNN models have the ResNet for their support and feedback.\r\n architecture, and RNN models are cells stacked together.\r\n In reality, more diverse models with a combination of em-\r\n bedding, FC, CNN and RNN layers are emerging. We also\r\n constrain the range of ParaDnn hyperparameters (Table 2).\r\n ASystematicMethodologyforAnalysisofDeepLearningHardwareandSoftwarePlatforms\r\n REFERENCES Dean, J. Recent advances in arti\ufb01cial intelligence and the\r\n TensorFlow: Using JIT compilation. 2018. https:// implications for computer system design. Hot Chips,\r\n www.tensorflow.org/xla/jit. 2017.\r\n Adolf, R., Rama, S., Reagen, B., Wei, G.-Y., and Brooks, D. Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M.,\r\n Fathom: Reference workloads for modern deep learning Mao,M.,Senior, A., Tucker, P., Yang, K., Le, Q. V., et al.\r\n methods. In Workload Characterization (IISWC), 2016 Large scale distributed deep networks. In Advances in\r\n IEEEInternational Symposium on, pp. 1\u201310. IEEE, 2016. neural information processing systems, pp. 1223\u20131231,\r\n 2012.\r\n Amodei, D., Ananthanarayanan, S., Anubhai, R., Bai, J., Dean, J., Patterson, D., and Young, C. A new golden\r\n Battenberg, E., Case, C., Casper, J., Catanzaro, B., Cheng, age in computer architecture: Empowering the machine-\r\n Q., Chen, G., et al. Deep speech 2: End-to-end speech learning revolution. IEEE Micro, 38(2):21\u201329, 2018.\r\n recognition in English and Mandarin. In International Ganesan, K., Jo, J., Bircher, W. L., Kaseridis, D., Yu, Z.,\r\n Conference on Machine Learning, pp. 173\u2013182, 2016. and John, L. K. System-level max power (SYMPO)-a\r\n Bahrampour,S., Ramakrishnan, N., Schott, L., and Shah, M. systematic approach for escalating system-level power\r\n Comparative study of Caffe, Neon, Theano, and Torch consumption using synthetic benchmarks. In Parallel\r\n for deep learning. In ICLR, 2016. Architectures and Compilation Techniques (PACT), 2010\r\n 19th International Conference on, pp. 19\u201328. IEEE, 2010.\r\n Banner, R., Hubara, I., Hoffer, E., and Soudry, D. Scalable Google. CloudTPUsystemarchitecture. GoogleCloudDoc-\r\n methods for 8-bit training of neural networks. arXiv umentation, 2018. https://cloud.google.com/\r\n preprint arXiv:1805.11046, 2018. tpu/docs/system-architecture.\r\n Bienia, C., Kumar, S., Singh, J. P., and Li, K. The PARSEC Gupta, U., Wang, X., Naumov, M., Wu, C.-J., Reagen,\r\n benchmarksuite: Characterization and architectural im- B., Brooks, D., Cottel, B., Hazelwood, K., Jia, B., Lee,\r\n plications. In Proceedings of the 17th international con- H.-H. S., et al. The architectural implications of Face-\r\n ference on Parallel architectures and compilation tech- book\u2019s DNN-based personalized recommendation. arXiv\r\n niques, pp. 72\u201381. ACM, 2008. preprint arXiv:1906.03109, 2019.\r\n Blog, G. A. Introducing GPipe, an open source library Han, S., Mao, H., and Dally, W. J. Deep compres-\r\n for ef\ufb01ciently training large-scale neural network mod- sion: Compressing deep neural networks with pruning,\r\n els. https://ai.googleblog.com/2019/03/ trained quantization and Huffman coding. arXiv preprint\r\n introducing-gpipe-open-source-library. arXiv:1510.00149, 2015.\r\n html,2019.\r\n Case, L. Volta Tensor Core GPU achieves new AI perfor- Hauswald, J., Kang, Y., Laurenzano, M. A., Chen, Q., Li,\r\n mancemilestones. Nvidia Developer Blog, 2018. C., Mudge, T., Dreslinski, R. G., Mars, J., and Tang, L.\r\n DjiNN and Tonic: DNN as a service and its implica-\r\n Chao, C. and Saeta, B. Cloud TPU: Codesigning architec- tions for future warehouse scale computers. In Computer\r\n ture and infrastructure. Hot Chips, 2019. Architecture (ISCA), 2015 ACM/IEEE 42nd Annual Inter-\r\n national Symposium on, pp. 27\u201340. IEEE, 2015a.\r\n Che,S.,Boyer,M.,Meng,J.,Tarjan,D.,Sheaffer,J.W.,Lee, Hauswald, J., Laurenzano, M. A., Zhang, Y., Li, C., Rovin-\r\n S.-H., and Skadron, K. Rodinia: A benchmark suite for ski, A., Khurana, A., Dreslinski, R. G., Mudge, T.,\r\n heterogeneous computing. In Workload Characterization, Petrucci, V., Tang, L., et al. Sirius: An open end-to-end\r\n 2009. IISWC 2009. IEEE International Symposium on, voice and vision personal assistant and its implications\r\n pp. 44\u201354. Ieee, 2009. for future warehouse scale computers. In the Twentieth\r\n Chen, T., Chen, Y., Duranton, M., Guo, Q., Hashmi, A., International Conference on Architectural Support for\r\n Lipasti, M., Nere, A., Qiu, S., Sebag, M., and Temam, ProgrammingLanguagesandOperatingSystems(ASP-\r\n O. BenchNN:Onthebroadpotential application scope LOS), volume 50, pp. 223\u2013238. ACM, 2015b.\r\n of hardware neural network accelerators. In Workload Hazelwood, K., Bird, S., Brooks, D., Chintala, S., Diril, U.,\r\n Characterization (IISWC), 2012 IEEE International Sym- Dzhulgakov, D., Fawzy, M., Jia, B., Jia, Y., Kalro, A.,\r\n posium on, pp. 36\u201345. IEEE, 2012. Law, J., Lee, K., Lu, J., Noordhuis, P., Smelyanskiy, M.,\r\n Chen, T., Moreau, T., Jiang, Z., Shen, H., Yan, E., Wang, Xiong, L., and Wang, X. Applied machine learning at\r\n L., Hu, Y., Ceze, L., Guestrin, C., and Krishnamurthy, A. Facebook: A datacenter infrastructure perspective. In\r\n TVM:End-to-endoptimization stack for deep learning. High Performance Computer Architecture (HPCA), 2018\r\n arXiv preprint arXiv:1802.04799, 2018. IEEEInternational Symposium on, pp. 620\u2013629. IEEE,\r\n 2018.\r\n Coleman, C., Narayanan, D., Kang, D., Zhao, T., Zhang, J., He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn-\r\n \u00b4\r\n Nardi, L., Bailis, P., Olukotun, K., Re, C., and Zaharia, M. ing for image recognition. In Proceedings of the IEEE\r\n DAWNBench: Anend-to-enddeeplearningbenchmark conference on computer vision and pattern recognition\r\n and competition. Training, 100(101):102, 2017. (CVPR), pp. 770\u2013778, 2016.\r\n ASystematicMethodologyforAnalysisofDeepLearningHardwareandSoftwarePlatforms\r\n \u00b4\r\n He, Q., Zhou, S., Kobler, B., Duffy, D., and McGlynn, Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollar, P.\r\n T. Case study for running HPC applications in public Focal loss for dense object detection. arXiv preprint\r\n clouds. In Proceedings of the 19th ACM International arXiv:1708.02002, 2017a.\r\n SymposiumonHighPerformanceDistributedComputing, Lin, Y., Han, S., Mao, H., Wang, Y., and Dally, W. J.\r\n pp. 395\u2013401. ACM, 2010. Deep gradient compression: Reducing the communica-\r\n Hennessy, J. L. and Patterson, D. A. Computer architecture: tion bandwidth for distributed training. arXiv preprint\r\n a quantitative approach. Elsevier, 2011. arXiv:1712.01887, 2017b.\r\n Henning, J. L. SPEC CPU2006 benchmark descriptions. Mattson, P., Cheng, C., Coleman, C., Diamos, G., Micike-\r\n ACMSIGARCHComputerArchitectureNews,34(4):1\u2013 vicius, P., Patterson, D., Tang, H., Wei, G.-Y., Bailis, P.,\r\n 17, 2006. Bittorf, V., et al. MLPerf training benchmark. arXiv\r\n Hogg,R.V.,McKean,J.,andCraig,A.T. Introduction to preprint arXiv:1910.01500, 2019.\r\n mathematical statistics. Pearson Education, 2005. Naumov,M.,Mudigere,D.,Shi,H.-J. M., Huang, J., Sun-\r\n Howard,A.G.,Zhu,M.,Chen,B.,Kalenichenko,D.,Wang, daraman, N., Park, J., Wang, X., Gupta, U., Wu, C.-J.,\r\n W., Weyand, T., Andreetto, M., and Adam, H. Mo- Azzolini, A. G., et al. Deep learning recommendation\r\n bileNets: Ef\ufb01cient convolutional neural networks for mo- modelfor personalization and recommendation systems.\r\n bile vision applications. arXiv preprint arXiv:1704.04861, arXiv preprint arXiv:1906.00091, 2019.\r\n 2017. Nickolls, J. and Dally, W. J. The GPU computing era. IEEE\r\n Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, Micro, 30(2), 2010.\r\n K. Q. Densely connected convolutional networks. In Repository, T. M. https://github.com/\r\n CVPR,volume1,pp. 3,2017. tensorflow/tpu. Github,2018.\r\n Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Research, B. Deepbench https://github.com/\r\n Dally, W. J., and Keutzer, K. SqueezeNet: AlexNet-level baidu-research/DeepBench. 2017.\r\n accuracy with 50x fewer parameters and \u00a10.5 MB model\r\n size. arXiv preprint arXiv:1602.07360, 2016. Rotem, N., Fix, J., Abdulrasool, S., Deng, S., Dzhabarov,\r\n Jia, Z., Zaharia, M., and Aiken, A. Beyond data and model R., Hegeman, J., Levenstein, R., Maher, B., Nadathur,\r\n parallelism for deep neural networks. arXiv preprint S., Olesen, J., et al. Glow: Graph lowering com-\r\n arXiv:1807.05358, 2018. piler techniques for neural networks. arXiv preprint\r\n arXiv:1805.00907, 2018.\r\n Joshi, A., Eeckhout, L., and John, L. The return of synthetic Saleem, M., Mehmood, Q., and Ngomo, A.-C. N. Feasible:\r\n benchmarks. In 2008 SPEC Benchmark Workshop, pp. Afeature-based SPARQL benchmark generation frame-\r\n 1\u201311, 2008. work. In International Semantic Web Conference, pp.\r\n Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, 52\u201369. Springer, 2015.\r\n G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, Schaffter, T., Marbach, D., and Floreano, D.\r\n A., et al. In-datacenter performance analysis of a tensor GeneNetWeaver: in silico benchmark generation\r\n processing unit. In Computer Architecture (ISCA), 2017 and performance pro\ufb01ling of network inference methods.\r\n ACM/IEEE44thAnnualInternational Symposium on, pp. Bioinformatics, 27(16):2263\u20132270, 2011.\r\n 1\u201312. IEEE, 2017.\r\n Kim, K., Lee, C., Jung, J. H., and Ro, W. W. Workload syn- Shallue, C. J., Lee, J., Antognini, J., Sohl-Dickstein, J.,\r\n thesis: Generating benchmark workloads from statistical Frostig, R., and Dahl, G. E. Measuring the effects of data\r\n execution pro\ufb01le. In Workload Characterization (IISWC), parallelism on neural network training. arXiv preprint\r\n 2014 IEEE International Symposium on, pp. 120\u2013129. arXiv:1811.03600, 2018.\r\n IEEE, 2014. Shi, S., Wang, Q., Xu, P., and Chu, X. Benchmarking\r\n Kothari, K. Comparison of several cloud computing state-of-the-art deep learning software tools. In Cloud\r\n providers. Elixir Comp. Sci. & Engg, 2011. ComputingandBigData(CCBD),20167thInternational\r\n Conference on, pp. 99\u2013104. IEEE, 2016.\r\n Krizhevsky, A., Sutskever, I., and Hinton, G. E. ImageNet Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou,\r\n classi\ufb01cation with deep convolutional neural networks. I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M.,\r\n In Advances in neural information processing systems, Bolton, A., et al. Mastering the game of Go without\r\n pp. 1097\u20131105, 2012. humanknowledge. Nature, 550(7676):354, 2017.\r\n Kung, H.-T. Why systolic architectures? IEEE computer, Stroobandt, D., Verplaetse, P., and Van Campenhout, J. Gen-\r\n 15(1):37\u201346, 1982. erating synthetic benchmark circuits for evaluating CAD\r\n Leary, C. and Wang, T. XLA: TensorFlow, compiled. Ten- tools. IEEE Transactions on Computer-Aided Design of\r\n sorFlow Dev Summit, 2017. Integrated Circuits and Systems, 19(9):1011\u20131022, 2000.\r\n ASystematicMethodologyforAnalysisofDeepLearningHardwareandSoftwarePlatforms\r\n Tao, J.-H., Du, Z.-D., Guo, Q., Lan, H.-Y., Zhang, L., Zhou, In this section, we explain the CPU work\ufb02ow of our artifact,\r\n S.-Y., Xu, L.-J., Liu, C., Liu, H.-F., Tang, S., Chen, W., which can be run in any environment from servers to even\r\n Liu, S.-L., and Chen, Y.-J. BenchIP: Benchmarking in- laptops. The same work\ufb02ow with small modi\ufb01cations can\r\n telligence processors. Journal of Computer Science and be used to evaluate other platforms supporting Python and\r\n Technology, 33(1):1\u201323, 2018. TensorFlow.\r\n Thomas, S., Gohkale, C., Tanuwidjaja, E., Chong, T., Lau, A.2 Artifact check-list (meta-information)\r\n D., Garcia, S., and Taylor, M. B. CortexSuite: A synthetic \u2022 Program: ParaDnn\r\n brain benchmark suite. In IISWC, pp. 76\u201379, 2014.\r\n Turki, M., Mehrez,H.,Marrakchi,Z.,andAbid,M. Towards \u2022 Compilation: Refer to the dependencies for Python 3 and Ten-\r\n synthetic benchmarks generator for CAD tool evaluation. sorFlow 1.x.\r\n In Ph. D. Research in Microelectronics and Electronics \u2022 Hardware: CPU,GPUorTPUinstalledwithTensorFlow1.6-\r\n (PRIME), 2012 8th Conference on, pp. 1\u20134. VDE, 2012. 1.13.\r\n Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, \u2022 Execution: Bash and Python scripts.\r\n L., Gomez, A. N., Kaiser, \u0141., and Polosukhin, I. Atten-\r\n tion is all you need. In Advances in Neural Information \u2022 Metrics: Training throughput as examples per second.\r\n Processing Systems, pp. 5998\u20136008, 2017. \u2022 Output: Performance datasets and analysis \ufb01gures.\r\n Wang, Y., Lee, V., Wei, G.-Y., and Brooks, D. Predicting \u2022 Experiments: See below.\r\n newworkloadorCPUperformancebyanalyzingpublic\r\n datasets. ACM Transactions on Architecture and Code \u2022 Howmuchdiskspacerequired(approximately)?: 10GB.\r\n Optimization (TACO), 15(4):1\u201321, 2019a.\r\n Wang, Y. E., Wu, C.-J., Wang, X., Hazelwood, K., and \u2022 How much time is needed to prepare work\ufb02ow (approxi-\r\n Brooks,D.Exploitingparallelismopportunitieswithdeep mately)?: 2 hours.\r\n learning frameworks. arXiv preprint arXiv:1908.04705, \u2022 Howmuchtimeisneededtocompleteexperiments(approx-\r\n 2019b. imately)?: 2 hours.\r\n Wang,Y.E.,Zhu,Y.,Ko,G.G.,Reagen,B.,Wei,G.-Y.,and \u2022 Publicly available?: Yes.\r\n Brooks, D. Demystifying Bayesian inference workloads. \u2022 Codelicenses (if publicly available)?: Apache License 2.0\r\n In 2019 IEEE International Symposium on Performance\r\n Analysis of Systems and Software (ISPASS), pp. 177\u2013189. \u2022 Work\ufb02owframeworkused?: ParaDnn\r\n IEEE, 2019c.\r\n Wei, W., Xu, L., Jin, L., Zhang, W., and Zhang, T. AI \u2022 Archived (provide DOI)?: 10.5281/zenodo.3687363\r\n matrix-synthetic benchmarks for DNN. arXiv preprint A.3 Description\r\n arXiv:1812.00886, 2018.\r\n Williams, S., Waterman, A., and Patterson, D. Roo\ufb02ine: A.3.1 Howtoaccess\r\n an insightful visual performance model for multicore Our source code is available on GitHub: https://\r\n architectures. Communications of the ACM, 52(4):65\u201376, github.com/Emma926/paradnn.\r\n 2009.\r\n Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., A.3.2 Hardwaredependencies\r\n Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, AnyCPU,GPU,orTPUplatformsthatsupportPython3.x\r\n K., et al. Google\u2019s neural machine translation system: and TensorFlow 1.6-1.13.\r\n Bridgingthegapbetweenhumanandmachinetranslation.\r\n arXiv preprint arXiv:1609.08144, 2016. A.3.3 Software dependencies\r\n A ARTIFACTAPPENDIX Python \u2265 3.7.6\r\n A.1 Abstract TensorFlow 1.6-1.13\r\n Our artifact provides the source code of ParaDnn, scripts A.3.4 Datasets\r\n to run ParaDnn and collect data, performance data sam- This work\ufb02ow will generate data sets. No extra data set is\r\n ples, and a Jupyter Notebook that demonstrate our analysis needed.\r\n methods. Reproducing the results in this paper requires the\r\n exact versions of software and hardware. Work\ufb02ows for A.3.5 Models\r\n GPUandTPUrequireaccesstospeci\ufb01chardware,NVIDIA\r\n V100GPUandCloudTPUv2/v3,whichmaynotbewidely This work\ufb02ow runs the FC, CNN and RNN models in\r\n available to all users. ParaDnn.\r\n ASystematicMethodologyforAnalysisofDeepLearningHardwareandSoftwarePlatforms\r\n A.4 Installation A.8 Notes\r\n Python For more questions, please \ufb01le issues on GitHub.\r\n $ sudo apt update A.9 Methodology\r\n $ sudo apt install python3-dev python3-pip\r\n TensorFlowPleaserefertotheinstallationguideforTensor- Submission, reviewing and badging methodology:\r\n Flow. If GPU is used, tensor\ufb02ow-gpu and its dependencies\r\n are needed. If Cloud TPU is used, no installation is needed \u2022 http://cTuning.org/ae/\r\n because software is included in the cloud instance. submission-20200102.html\r\n After setting up virtual environment based on the Tensor- \u2022 http://cTuning.org/ae/\r\n Flowinstallation guide, please do reviewing-20200102.html\r\n $ pip3 install tensorflow==1.13.1\r\n $ pip3 install google-api-python-client \u2022 https://www.acm.org/publications/\r\n $ pip3 install oauth2client policies/artifact-review-badging\r\n $ pip3 install notebook\r\n $ pip3 install seaborn\r\n $ pip3 install matplotlib\r\n $ pip3 install sklearn\r\n ParaDnn\r\n $ git clone https://github.com/Emma926/paradnn\r\n A.5 Experimentwork\ufb02ow\r\n Test Environment\r\n $ cd paradnn/\r\n $ python test.py --use_tpu=False\r\n RunParaDnn\r\n $ # cd to paradnn/paradnn\r\n $ cd paradnn/\r\n $ bash run/fc_cpu.sh\r\n Collect Performance Data\r\n $ cd ../scripts\r\n $ python get_perf.py\r\n RunAnalysisTools\r\n $ cd scripts/plotting\r\n $ jupyter notebook\r\n Open \u201cDemo.ipynb\u201d and run through the code blocks se-\r\n quentially.\r\n A.6 Evaluation and expected result\r\n Our analysis methodology is presented in the plots from\r\n Jupyter Notebook. The methodology is evaluated if the\r\n plots show similar trends as the ones in the paper.\r\n A.7 Experimentcustomization\r\n All the run scripts are in paradnn/paradnn/run/*.sh. To run\r\n RNNmodelsofParaDnn,simplydo\u201cbashrun/rnn cpu.sh\u201d.\r\n Torunonotherplatforms, such as GPUs and Cloud TPUs,\r\n use \ufb01les \u201crun/* gpu.sh\u201d and \u201crun/* tpu.sh\u201d. Users can\r\n also change the range of ParaDnn parameters in paradnn/-\r\n paradnn/run/*.sh. The sweeping range of parameters largely\r\n determines the reproducibility of this artifact. Users can\r\n also modify Jupyter Notebooks, for example, to analyze the\r\n speedup of TPU over CPU.\r\n", "award": [], "sourceid": 12, "authors": [{"given_name": "Yu", "family_name": "Wang", "institution": "Harvard University"}, {"given_name": "Gu-Yeon", "family_name": "Wei", "institution": "Harvard University"}, {"given_name": "David", "family_name": "Brooks", "institution": "Harvard University"}]}