{"title": "BlueConnect: Decomposing All-Reduce for Deep Learning on Heterogeneous Network Hierarchy", "book": "Proceedings of Machine Learning and Systems", "page_first": 241, "page_last": 251, "abstract": "As deep neural networks get more complex and input datasets get larger, it can take days or\neven weeks to train a deep neural network to the desired accuracy. Therefore, enabling distributed deep learning at a\nmassive scale is a critical, since it offers the potential to reduce the training\ntime from weeks to hours. In this paper, we present BlueConnect, an\nefficient communication library for distributed deep learning that is highly optimized for popular GPU-based platforms.\nBlueConnect decomposes a single all-reduce operation into a large number of parallelizable reduce-scatter and all-gather operations\nto exploit the trade-off between\nlatency and bandwidth, and adapt to a variety of network configurations. Therefore, each individual operation can be mapped\nto a different network fabric and take advantage of the best performing library for that fabric.\nWe integrated BlueConnect into Caffe2, and demonstrated that BlueConnect significantly\npushes the state-of-the-art in large-scale deep learning\nby reducing communication overhead by 87\\% on 192 GPUs for Resnet-50 training over prior arts.", "full_text": " BLUECONNECT: DECOMPOSINGALL-REDUCEFORDEEPLEARNINGON\r\n HETEROGENEOUSNETWORKHIERARCHY\r\n MinsikCho1 UlrichFinkler2 DavidKung2 HilleryHunter2\r\n ABSTRACT et al., 2012). Hardware accelerators such as GPU/TPU and\r\n Asdeepneural networks get more complex and their accompanying software stacks have provided a sig-\r\n input datasets get larger, it can take days or even ni\ufb01cant amount of speed up (Jouppi et al., 2017; NVidia,\r\n weekstotrainadeepneuralnetworktothedesired 2017b). However, deep neural network training for speech\r\n accuracy. Therefore, enabling distributed deep and vision can still take days and even weeks. Therefore,\r\n learning at a massive scale is critical, since it of- parallelization by distributing the deep learning training to\r\n fers the potential to reduce the training time from many (upwards of hundreds) GPUs over a cluster or on a\r\n weeks to hours. In this paper, we present Blue- cloud environment is critical to cut the training time from\r\n Connect, an ef\ufb01cient communication library for weeks to hours and minutes (Goyal et al., 2017; Iandola\r\n distributed deep learning that is highly optimized et al., 2015; Jia et al., 2018; You et al., 2017a;b).\r\n for popular GPU-based platforms. BlueConnect Distributed deep learning is challenging because as the num-\r\n decomposes a single all-reduce operation into a ber of learners (or GPUs) increases, the computation time\r\n large number of parallelizable reduce-scatter and decreases while the amount of communication stays con-\r\n all-gather operations to exploit the trade-off be- stant (Goyal et al., 2017; Uber, 2017; You et al., 2017a),\r\n tween latency and bandwidth, and adapt to a va- resulting in unfavorable computation to communication ra-\r\n riety of network con\ufb01gurations. Therefore, each tios, and thus diminished returns on more learners. One\r\n individual operation can be mapped to a differ- can either increase the computational workload with a large\r\n ent network fabric and take advantage of the best mini-batch size in stochastic gradient decent (SGD) (i.e.,\r\n performing implementation for the correspond- weakscaling) and/or decrease the communication overhead.\r\n ing fabric. According to our experimental results However, it is known that a large mini-batch beyond a cer-\r\n on two system con\ufb01gurations, BlueConnect can tain point can degrade training quality (Balles et al., 2016;\r\n outperform the leading industrial communication Keskar et al., 2016; Krizhevsky, 2014), not to mention that\r\n library by wide margin, and the BlueConnect mini-batch size is limited by the GPU memory capacity in\r\n integrated Caffe2 can signi\ufb01cantly reduce syn- practice. Therefore, in addition to enabling deep learning\r\n chronization overhead by 87% on 192 GPUs for with large mini-batch sizes (Goyal et al., 2017; Jia et al.,\r\n Resnet-50 training over prior schemes. 2018; You et al., 2017a;b), it is crucial to develop a fully op-\r\n timized communication mechanism tuned for deep learning\r\n for massive scale-out that can a) maximize the bandwidth\r\n 1 INTRODUCTION utilization in popular deep learning environments like GPU-\r\n Deeplearning has become the de-facto technique for an in- based cluster/cloud, and b) minimize the linearly growing\r\n creasing number of cognitive applications, including vision, communication latency with the number of learners (Srid-\r\n speech, and language translation (Amodei et al., 2015; Ioffe haran et al., 2018).\r\n &Szegedy,2015;Jia et al., 2014). The success is driven by In this paper, we report the performance of an ef\ufb01cient\r\n the availability of an enormous volume of data and advances communication library for deep learning, BlueConnect, that\r\n in deep neural networks, which in turn make deep learning provides a highly ef\ufb01cient all-reduce algorithm for SGD, an\r\n one of the most computationally demanding AI applica- integral part in modern deep learning frameworks (Abadi\r\n tions (Amodei et al., 2015; Chen et al., 2016; Krizhevsky et al., 2016; Chen et al., 2015; Facebook, a;b; Goyal et al.,\r\n 2017; Jia et al., 2014; Niitani et al., 2017; NVidia, 2017a;\r\n 1IBM Systems, Austin, Texas, USA 2IBM T. J. Watson Re- Seide & Agarwal, 2016). The key idea in BlueConnect is to\r\n search Center, Yorktown Heights, New York, USA. Correspon- decompose one all-reduce operation into series of reduce-\r\n dence to: Minsik Cho . scatter and all-gather patterns in a topology-aware fashion,\r\n Proceedings of the 2nd SysML Conference, Palo Alto, CA, USA, whichenablesalarge-scaledeeplearningwithreducedcom-\r\n 2019. Copyright 2019 by the author(s).\r\n BlueConnect: Decomposing All-Reduce for Deep Learning on Heterogeneous Network Hierarchy\r\n munication overhead. Our technical contribution includes: Table 1. Notations.\r\n \u2022 BlueConnectadaptstothehierarchyofcommunication \u03b1 non-zero latency time per transfer at each network switch\r\n bandwidths by leveraging topology-awareness, so that wi bandwidth (unit/sec) of network switch type i\r\n s a network switch instance j with w\r\n it fully utilizes the heterogeneous network architec- i.j i\r\n W a set of network bandwidths, {wi|\u2203i \u2208 Z }\r\n ture in popular deep learning platforms (IBM, 2017a; \u22650\r\n S a set of switch instances, {s |w \u2208W,\u2203j \u2208Z }\r\n NVidia, a). i.j i \u22650\r\n P a set of learners\r\n N the gradient\u2019s size in unit\r\n \u2022 Through topology-aware decomposition, BlueConnect c(r,w) a set of learners that perform reduce scatter\r\n also minimizes the communication latency overhead, and all gather over bandwidth w with a learner r\r\n the critical bottleneck in large-scale deep learning.\r\n \u2022 For each decomposed piece, BlueConnect can mix- like cloud, b) the traf\ufb01c generated by deep learning is highly\r\n and-match various reduce-scatter and all-gather imple- bursty and extremely large (i.e., 100MB -1GB), while ex-\r\n mentations/algorithms over different network fabrics isting techniques have been optimized for relatively small\r\n to maximize network utilization. and frequent exchanges, and c) most existing algorithms are\r\n not tuned for new network fabrics (i.e, NVLink (NVLink,\r\n The rest of the paper is organized as follows. We present 2017)). As future GPUs/accelerators double their perfor-\r\n preliminaries in Section 2. Section 3 discusses our pro- manceeachgeneration,thegradientsynchronizationinSGD\r\n posed algorithm, BlueConnect. Experimental results are in will become a considerable bottleneck in large-scale deep\r\n Section 4, followed by the conclusion in Section 5. learning (Keuper, 2016). Hence, it is in great demand to\r\n study an ef\ufb01cient communication technique for deep learn-\r\n 2 PRELIMINARIES ing that addresses the 3 issues mentioned above.\r\n Other approaches to reduce communication overhead in\r\n 2.1 Prior Arts deep learning are largely based on approximation of fully\r\n To enable large scale distributed deep learning with hun- synchronous SGD (Wang & Joshi, 2018) including asyn-\r\n dreds of GPUs under popular data-parallelism (Amodei chronous SGD (ASGD)whereeachlearner can subscribe\r\n et al., 2015; Goyal et al., 2017; You et al., 2017a), the batch theupdatedweightsfromaparameterserverasynchronously\r\n size must be in the thousands since GPU utilization and com- (i.e., removing the synchronization barrier and suppressing\r\n pute to communication ratio are low for single digit batch bursty traf\ufb01c) (Niu et al., 2011; Zhang et al., 2015a;b) and\r\n size per GPU for typical neural networks. Since a large decentralized SGD where each learner communicates only\r\n batch size in deep learning may cause poor convergence, with a subset of all learners (Lian et al., 2017). It is shown\r\n there have been recent efforts to mitigate such convergence that such approximated or stochastic methods can improve\r\n and generalization issues (Goyal et al., 2017; Keskar et al., the scalability of distributed deep learning but at a cost of po-\r\n 2016; You et al., 2017a). (Goyal et al., 2017) proposed a tentially reduced accuracy and instable convergence (Chen\r\n linear learning rate scaling rule and performed learning rate et al., 2016).\r\n warm-upfromasmall/safe value to the larger target value 2.2 Notations and Basic Performance Models\r\n in the early training phase, and then resorting to the usual\r\n step-wise descent. For gradient synchronization, (Goyal Notations used in this paper are listed in Table 1. We ignore\r\n et al., 2017) leveraged a deep learning communication li- arithmetic operation time, as it is trivially cheap in deep\r\n brary (Facebook, b) to demonstrate that Resnet-50 (He et al., learning on GPUs. Then, for a given data size n, a learner\r\n 2015) can be trained in one hour over 32 DGX-1\u2019s (256 countp,andabandwidthw,theperformanceofaring-based\r\n GPUs). Recently, (Jia et al., 2018) demonstrated that the communication pattern can be expressed as follows (Thakur\r\n same Resnet50 can be trained in four minutes over 1024 et al., 2005):\r\n GPUswithlow-precision (i.e., FP16). p\u22121n\r\n While this is an impressive result and there exist numerous Tr(p,n,\u03b1,w) = (p\u22121)\u03b1+ p w (1)\r\n communication algorithms for distributed computing plat- Whenthe latency between any two nodes is uniform and\r\n \u00b4\r\n forms (Almasi et al., 2005; Baidu, 2017; Jia et al., 2018; p is a power-of-two number, one can use recursive halv-\r\n Thakuretal.,2005),theyarenotnecessarilycustomizedand ing/doubling to obtain the same result with smaller latency,\r\n optimized for large scale distributed deep learning (Amodei which can expressed as follows (Thakur et al., 2005):\r\n et al., 2015): a) most existing techniques were developed\r\n for a homogeneous environment, while deep learning will Tc(p,n,\u03b1,w) = lg(p)\u03b1+ p\u22121 n (2)\r\n be increasingly deployed on a heterogeneous environment p w\r\n BlueConnect: Decomposing All-Reduce for Deep Learning on Heterogeneous Network Hierarchy\r\n S2.0\r\n w2 w2\r\n S1.0 S1.1\r\n w1 w1 w1 w1\r\n A0 B0 C0 D0\r\n w0 w0 w0 w0\r\n 1 3\r\n 0 . 2 . Figure 2. Two-level all-reduce (Jia et al., 2018) where only\r\n . 0 . 0\r\n 0 A1 S B1 0 C1 S D1\r\n S w0 w0 Sw0 w0 master learners are active in the 2nd step.\r\n A2 B2 C2 D2\r\n w0 w0 w0 w0\r\n Thakur et al., 2005) can be expressed with the following\r\n Figure 1. 4 nodes with 12 learners on heterogeneous network ar- performance model:\r\n chitecture connected hierarchically.\r\n N\r\n T =2(|P|\u22121){\u03b1+ |P| } (5)\r\n one lvl min {w }\r\n Bycombining Eq. (1) and (2), we de\ufb01ne the following to 0\u2264i<|W| i\r\n get the best of both: =2Tr(|P|,N,\u03b1,minW) (6)\r\n \u001a q\r\n Tr/c(p,n,\u03b1,w) = Tc(p,n,\u03b1,w) p = 2 ,q \u2208 Z (3) where there are 2(|P| \u2212 1) iterations in Eq. (5), and each\r\n Tr(p,n,\u03b1,w) otherwise iteration needs to transfer N data over w ,w ,...,w\r\n |P| 0 1 |W|\u22121\r\n Based on the ring and recursive communication pat- in the worst case (e.g., marked with dotted arrows from\r\n A to D in Fig. 1). Although one-level ring-based\r\n terns, we can compute the communication performance of 0 2\r\n broadcastorreduceasfollows(Thakuretal.,2005): all-reduce has been widely used for traditional high-\r\n performance computing, it is not quite suitable for large-\r\n Tbcast(p,n,\u03b1,w) = Treduce(p,n,\u03b1,w) scale deep learning for two reasons:\r\n =Tc(p,n,\u03b1,w)+Tr(p,n,\u03b1,w) (4)\r\n Since our focus is on heterogeneous network architec- \u2022 A node with multiple GPUs (up to 16 GPUs per\r\n node (Amazon)) may have multiple learners inside\r\n ture (Dichev & Lastovetsky, 2014), we extend the homoge- and increases |P| fast, which would rapidly increase\r\n neousmodel(Thakuretal.,2005)byusingdifferentwi. For the latency of deep learning communication (i.e., a\r\n example, we assume a typical hierarchically built cluster large multiplier to \u03b1).\r\n over tree-like heterogeneous network architecture (NVidia,\r\n b) as in Fig. 1 where 12 learners (P = {A ,B ,C ,D |\u2200i \u2208\r\n i i i i\r\n {0,1,2}}) are connected through heterogeneous network \u2022 Since deep learning typically runs on a heterogeneous\r\n switches in S = {s , s , s }. Regarding the network topology (e.g., Fig. 1), the performance of\r\n 0.{0,1,2,3} 1.{0,1} 2.0\r\n example in Fig 1, s0.\u2217 can represent an intra-node network one-level approach is gated by the slowest bandwidth\r\n like NVLink around 32GB/s per lane, while s1.\u2217 and s2.0 along the path (i.e., minW), not fully utilizing other\r\n mayrepresent inter-node switches for 100Gbps In\ufb01niBand. fast networks fabrics.\r\n In such cases, w1 and w2 would be 100Gbps and 200Gbps\r\n respectively to ideally match the total uplink bandwidth Toaddress this problem, a two-level approach is used in the\r\n from all the hanging nodes (e.g., fat-tree (Al-Fares et al., state-of-the-art deep learning softwares (Facebook, b; Jia\r\n 2008; NVidia, b)). et al., 2018; NVidia, 2017a) shown in Fig. 2. In the \ufb01rst step,\r\n 2.3 All-Reduce for Distributed SGD the gradients are reduced to the master learner on each node.\r\n Then, a small-scale one-level ring-based all-reduce is\r\n Thekeycommunicationpattern used in SGD synchroniza- appliedamongthemasterlearnersonly. Finally,thegradient\r\n tion in deep learning is all-reduce (Amodeietal., 2015; in the master learners is locally broadcast back to the other\r\n Baidu, 2017) which is popularly implemented with ring- learners within the same node, synchronizing all the learners\r\n based reduce scatterorall gather(Thakuretal., in the training task. When |P| is decomposed into two\r\n 2005). Based on Eq.(1,4), the synchronization costs of learner counts such as p (the number of learners within\r\n 0\r\n prior arts in deep learning can be computed. For exam- each node) and p (the number of master learners) like\r\n 1\r\n ple of one-level ring-based all-reduce in (Baidu, 2017; |P| = p p , the performance of such a two-level scheme\r\n 0 1\r\n BlueConnect: Decomposing All-Reduce for Deep Learning on Heterogeneous Network Hierarchy\r\n can be formally expressed as follows (Jia et al., 2018): All-Reduce\r\n T =T (p ,N,\u03b1,w )*reducetomaster*\r\n two lvl reduce 0 0 Reduce-Scatter All-Gather\r\n +T (p ,N,\u03b1,w ) *bcastfrommaster*\r\n bcast 0 0\r\n +2T (p ,N,\u03b1, min {w })\r\n r/c 1 0\u2264i<|W| i Reduce-Scatter Reduce-Scatter All-Gather All-Gather\r\n Reduce-Scatter Reduce-Scatter All-Gather All-Gather\r\n =2T (p ,N,\u03b1,w )+2T (p ,N,\u03b1,w )\r\n c 0 0 r 0 0 Reduce-Scatter Reduce-Scatter All-Gather All-Gather\r\n P Reduce-Scatter Reduce-Scatter All-Gather All-Gather\r\n +2T ( ,N,\u03b1,minW) (7) . . . .\r\n r/c p . P/p0 . P/pi . P/pi . P/p0\r\n 0\r\n Although we can trivially show that Eq. (7) has smaller Figure 3. All-reduce can be decomposed into multiple stages of\r\n latency overhead than Eq. (6), it would still suffer from the parallelizable reduce-scatter and all-gather operations.\r\n following three limitations:\r\n \u2022 Latency overhead can be large when p \u226a |P|. granularity nor \ufb02exibility suf\ufb01cient enough to utilize the\r\n 0\r\n \u2022 Performance is still gated by minW. underlying hardware and the highly optimized implemen-\r\n tations (i.e., ones offered by the hardware vendors) ef\ufb01-\r\n \u2022 Manylearners stay idle during the 2nd step, leading to ciently. We, however, found that the reduce-scatter\r\n bandwidth under-utilization. and all-gather can be further decomposed into mul-\r\n tiple stages of parallelizable reduce-scatter and\r\n \u2022 reduce/broadcastatthe\ufb01rststepandthelaststep all-gather operations in some symmetric cases. In\r\n is expensive. detail, Fig. 3 shows that all-reduce can be \ufb01rst broken\r\n into one reduce-scatter followed by all-gather\r\n OurproposedBlueConnectinSection3addressestheselimi- (the arrows indicate dependency). However, additional de-\r\n tations with a novel topology-awareschemeasinSection3.2 composition is possible if the following integer factorization\r\n based on the all-reduce decomposition in Section 3.1. exists:\r\n |P| = p p p ...p = Yp (p \u2208 N,p >1) (8)\r\n 3 BLUECONNECT 0 1 2 k i i i\r\n i