{"title": "TicTac: Accelerating Distributed Deep Learning with Communication Scheduling", "book": "Proceedings of Machine Learning and Systems", "page_first": 418, "page_last": 430, "abstract": "State-of-the-art deep learning systems rely on iterative distributed training to tackle the increasing complexity of models and input data. In this work, we identify an opportunity for accelerating distributed DNN training in systems that rely on graph representation for computation, such as TensorFlow and PyTorch, through communication scheduling. We develop a system, TicTac, that reduces the iteration time by identifying and enforcing parameter transfers in the order in which the parameters are consumed by the underlying computational model, thereby guaranteeing near-optimal overlap of communication and computation. Our system is implemented over TensorFlow and enforces the optimal ordering by prioritization of parameter transfers at the Parameter Server in data parallel training. TicTac requires no changes to the model or developer inputs and improves the throughput by up to $37.7\\%$ in inference and $19.2\\%$ in training, while also reducing straggler effect by up to $2.3\\times$. Our code is publicly available.", "full_text": " TICTAC: ACCELERATING DISTRIBUTED DEEP LEARNING WITH\r\n COMMUNICATIONSCHEDULING\r\n SayedHadiHashemi*1 SangeethaAbduJyothi*1 RoyHCampbell1\r\n ABSTRACT\r\n State-of-the-art deep learning systems rely on iterative distributed training to tackle the increasing complexity\r\n of models and input data. In this work, we identify an opportunity for accelerating distributed DNN training in\r\n systems that rely on graph representation for computation, such as TensorFlow and PyTorch, through commu-\r\n nication scheduling. We develop a system, TicTac, that reduces the iteration time by identifying and enforcing\r\n parameter transfers in the order in which the parameters are consumed by the underlying computational model,\r\n thereby guaranteeing near-optimal overlap of communication and computation. Our system is implemented over\r\n TensorFlow and enforces the optimal ordering by prioritization of parameter transfers at the Parameter Server in\r\n data parallel training. TicTac requires no changes to the model or developer inputs and improves the throughput\r\n by up to 37.7% in inference and 19.2% in training, while also reducing straggler effect by up to 2.3\u00d7. Our code\r\n is publicly available.\r\n 1 INTRODUCTION improve the learning time by hours in these long-running\r\n Deep learning has grown signi\ufb01cantly in the past decade, learning jobs.\r\n fuelled by the \ufb02exibility of development offered by ma- Theiteration time in deep learning systems depends on the\r\n chine learning frameworks, availability of rich data, and time taken by (i) computation, (ii) communication and (iii)\r\n readily accessible distributed high-performance comput- the overlap between the two. When workers receive the pa-\r\n ing. The computational cost of training sophisticated deep rametersfromtheparameterserveratthebeginningofeach\r\n learning modelshaslongoutgrownthecapabilitiesofasin- iteration, all parameters are not used simultaneously; they\r\n gle high-end machine, leading to distributed training being are consumed based on the dependencies in the underlying\r\n the norm in a typical AI pipeline. Training a deep learning DAG.Whileoneparticularscheduleofparametertransfers\r\n model is an iterative job which may take days to weeks in (over the complete set of parameters in a given model in a\r\n high-end clusters today. single iteration) may facilitate faster computation, another\r\n Computational graphs are used to represent the training may cause blockage. Hence, identifying the best schedule\r\n jobs in state-of-the-art systems (Abadi et al., 2016; Chen of parameter transfers is critical for reducing the blocking\r\n et al., 2015; Paszke et al., 2017). In the commonly-used on computation (determined by DAG dependencies), and\r\n Model Replica or data parallel mode of training, the input in turn improving the overlap and the iteration time.\r\n data is partitioned and processed at participating workers We observe that the schedule of data transfers in current\r\n using identical computational graphs. Each iteration typi- systems(Abadietal.,2016;Chenetal.,2015;Paszkeetal.,\r\n cally lasts milliseconds to seconds. At the end of each it- 2017) is determined arbitrarily during execution without\r\n eration, servers exchange a relatively large amount of data considering the impact on overlap. We quantify the ob-\r\n associated with parameter updates to aggregate the results served combinations in TensorFlow and \ufb01nd that in a trial\r\n of the iteration. This communication overhead has a sub- with 1000 iterations on ResNet-V2-50, every iteration had\r\n stantial impact on throughput of the system and also limits a unique order of received parameters which has not been\r\n its scalability (Sridharan et al., 2018; Alistarh et al., 2017). observedpreviously. Thisrandomorderofparametertrans-\r\n Evenasmallimprovementincommunicationoverheadcan fers at workers has two performance implications. First,\r\n * 1 the iteration time, and in turn throughput (number of sam-\r\n Equal contribution Department of Computer Science, Uni- ples processed per second), suffers signi\ufb01cantly due to sub-\r\n versity of Illinois at Urbana-Champaign, Urbana, IL. Correspon- optimal overlap. Second, even in the same iteration, multi-\r\n dence to: Sayed Hadi Hashemi . ple workers might follow different schedules of data trans-\r\n Proceedings of the 2nd SysML Conference, Palo Alto, CA, USA, fers, leading to stragglers during synchronized training.\r\n 2019. Copyright 2019 by the author(s).\r\n TicTac: Accelerating Distributed Deep Learning with Communication Scheduling\r\n Past work has attempted to address this issue by enforc- are reading parameters from the PS or decentralized work-\r\n ing the same order of parameter transfers at all workers. ers as shown in Figure 3. While decentralized aggrega-\r\n However, these solutions are restricted to earlier systems tion techniques (such as all-reduce and Horovod (Sergeev\r\n with layer-by-layer model representation (Arnold, 2016; &Balso, 2018)) are gaining traction in high performance\r\n Cuietal., 2016; Zhang et al., 2017) where \ufb01nding the opti- networking, TicTac does not address such systems and is\r\n mal order of execution is trivial (Cui et al., 2014). In mod- focused on PS.\r\n ern systems with DAG representation (Abadi et al., 2016; In this section, we give a brief overview of deep learning\r\n Paszke et al., 2017), this is a non-trivial challenge. systems, prior techniques proposed in these systems to mit-\r\n Inthiswork,wedeviseasystematicmethodologyforderiv- igate network overhead, and opportunities for further opti-\r\n ing near-optimal schedules of parameter transfers through mization.\r\n critical path analysis on the underlying computational\r\n graph. This allows maximal overlap of computation and 2.1 NetworkOptimizationinDNNtraining\r\n communication and prevents stragglers arising from ran- In deep learning systems, high GPU utilization can be\r\n dom order of parameter transfers at workers. We also achievedintwoways: (i)whentotalcommunicationtimeis\r\n develop a lightweight resource-level enforcement mecha- less than or equal to the computation time and (ii) with ef\ufb01-\r\n nism over TensorFlow (Abadi et al., 2016). These tech- cient overlap of communication and computation. Several\r\n niques form the core of our system, TicTac, which achieves techniqueshavebeenproposedtoimproveGPUutilization.\r\n substantial performance improvement while requiring no\r\n changes in the model or developer inputs. Increasing computation time: The fraction of com-\r\n In summary, we make the following contributions: putation time relative to communication time can be in-\r\n \u2022 We identify an opportunity for improving performance creased by increasing the batch size (Iandola et al., 2016).\r\n in state-of-the-art deep learning systems with Parameter However, this approach suffers from decreased accu-\r\n Server-based aggregation through prioritized parameter racy (Keskar et al., 2016) and may not be generally appli-\r\n transfers (\u00a72). cable under resource constraints. (Goyal et al., 2017; Cho\r\n \u2022 We de\ufb01ne a metric to quantify the ef\ufb01ciency of a given et al., 2017; You et al., 2017; Akiba et al., 2017).\r\n execution: the overlap coef\ufb01cient (\u00a73). Decreasing communicationtime: Solutions for reducing\r\n \u2022 We propose two heuristics, TIC and TAC, for near- networkcommunicationhavetakenmultipleapproaches\u2014\r\n optimal scheduling of computation and communication modifying the machine learning algorithm to reduce com-\r\n in Model Replica with Parameter Server. munication cost (Alistarh et al., 2017; Wen et al., 2017;\r\n \u2022 We implement our system over TensorFlow (\u00a7 5). The Zhang et al., 2017), reducing the precision of parameter\r\n code is publicly available. 1 representation (Vanhoucke et al., 2011; Courbariaux et al.,\r\n 2015; Gupta et al., 2015), changing the network primitives\r\n \u2022 We extensively evaluate the performance of our system to collective (e.g. all reduce) (Goyal et al., 2017; Cho et al.,\r\n in GPU and high-end CPU environments under training 2017; Amodei et al., 2015; You et al., 2017; Akiba et al.,\r\n and inference of DNN models and show that throughput 2017) or broadcast (Zhang et al., 2017).\r\n can be improved by up to 37% (\u00a76). Smarter interleaving of computation and communica-\r\n 2 BACKGROUNDANDMOTIVATION tion: Several layer-by-layer systems (Arnold, 2016; Cui\r\n et al., 2016; Zhang et al., 2017), where the models are se-\r\n Oursystemfocusesonnetworkoptimizationindeeplearn- quential and obtaining the order is trivial (Cui et al., 2014),\r\n ing frameworks with DAG representation of computational adopt this approach. These solutions are not applicable\r\n graphs (Abadi et al., 2016; Paszke et al., 2017), Model to current DAG-based systems such as TensorFlow (Abadi\r\n Replica (MR) mode of distribution and Parameter Servers. et al., 2016) and PyTorch (Paszke et al., 2017). The inter-\r\n The performance improvement provided by TicTac is ben- resource dependency considered in (Cui et al., 2016) (with\r\n e\ufb01cial in two key environments. First, it improves through- GPUmemory) and in (Zhang et al., 2017) (with network)\r\n put and iteration time in clud environment with commodity is constrained to layer-by-layer models.\r\n hardware or on-demand clusters where high resiliency is In this work, we focus on improving the iteration time\r\n critical (workers may be preempted). Second, in online re- through better and predictable overlap of communication\r\n inforcementlearningwithworkersfortrainingandseparate and computation. Techniques for optimizing communica-\r\n active agents for inference, enforced ordering can improve tion and communication time are orthogonal to our system\r\n the inference time. In this environment, the active agents and may be used in parallel with TicTac.\r\n 1https://github.com/xldrx/tictac\r\n TicTac: Accelerating Distributed Deep Learning with Communication Scheduling\r\n Partition Partition PS: 0 Partition Worker: 0\r\n op2 Send 0 Recv 0\r\n Time Read Send 1 Send 0\r\n recv2 NIC recv1 recv2\r\n Recv 1 Aggregate Update\r\n Recv 0\r\n op1 Processor op1 op2\r\n recv1 (b) Good Execution Order Figure 2: Distributed execution of Model-Replica with Parame-\r\n ter Server\r\n NIC recv2 recv1\r\n (a) Toy Computational Parameter Servers\r\n Graph Processor op1 op2 Parameters\r\n (c) Bad Execution Order Read Update\r\n Figure 1: Impact of multi-resource operation ordering on perfor- Agents Workers\r\n mance\r\n Observations Input Data\r\n 2.2 Opportunity for Optimization Inference Training\r\n We demonstrate the opportunity for accelerating DNN Figure 3: A general reinforcement learning setup\r\n training through a better understanding of the internal\r\n computational model in TensorFlow which is a Directed ResNet-v2-50,Inception-v3andVGG-16networksandob-\r\n AcyclicGraph(DAG).Theparametertransfersaredenoted serve the order of network transfers at a single worker. The\r\n by send and recv operations in the DAG. In MR, each observed order of parameter transfer is unique in ResNet-\r\n worker has an identical copy of the computational DAG. v2-50 and Inception-v3 networks across the 1000 runs. In\r\n In the worker DAG, all recv ops are roots and send ops VGG-16,weobserve493uniquecombinationsacross1000\r\n are leaves. Thus recv ops can block the initialization of runs.\r\n a computation branch in the DAG. Since the activation of\r\n various branches of computation in the DAG is dependent 2.3 ComparisonwithOtherDistributedSystems\r\n on the recv at the root of the branch, the ordering in MR It is worth noting that deep learning systems with computa-\r\n can be reduced to the ordering of recv ops in workers. tional graphs are fundamentally different from graph pro-\r\n DAGatthe PS is different from that at workers. PS DAG cessing systems (Malewicz et al., 2010; Hoque & Gupta,\r\n has \ufb01ve ops per parameter: aggregation, send, recv, read, 2013;Xinetal.,2013). Indeeplearning, thegraphisarep-\r\n and update. Since send and recv at the PS are not blocked resentation of the computation to be done on the input data.\r\n bycomputation, our focus is on the worker DAG. In graph processing systems, the graph itself is the input to\r\n In the simple DAG shown in Figure 1a, a sample worker the computation. As a result, graphs in DNN frameworks\r\n DAG,therearetwopossibleschedulesforparametertrans- are a few orders of magnitude smaller than a typical large-\r\n fers. If recv (parameter 1 transfer from PS to the worker) scale graph processing system. Iterations in DNN frame-\r\n 1\r\n happens before recv (parameter 2 transfer), it reduces the works are identical, and network communication pattern is\r\n 2\r\n blocking on computation time and improves the overlap. \ufb01xed. This may not be true for graph processing systems.\r\n The reverse order results in increased iteration time due to Instreamprocessingsystems,therelationshipbetweenpro-\r\n blocking on computation. Thus, in a distributed environ- cessing elements are represented using graphs. These sys-\r\n ment, network can block computation based on dependen- temsallowpipelining,withdifferentpartitionsofinputdata\r\n cies in the DAG. This can lead to under-utilization of com- being processed in different elements along the pipeline at\r\n putational capacity, in turn resulting in sub-optimal perfor- the same time. In contrast, DNN frameworks process the\r\n mance. In addition, variation in iteration time caused by entire batch of input at a processing element at a worker.\r\n random order of parameter transfers across multiple work- Pipelining is not employed in this environment. Hence, op-\r\n ers can lead to straggling effect. timizations proposed for stream processing cannot be bor-\r\n Theimpactofpooroverlapcanbesigni\ufb01cantinDNNtrain- rowed here.\r\n ing due to complexity of state-of-the-art models. For in-\r\n stance, ResNet-v2-152 (He et al., 2016) has 363 param- 3 QUANTIFYING PERFORMANCE\r\n eters with an aggregate size of 229.5MB. The computa-\r\n tional graph associated with this neural network has 4655 In this section, we explore methods for quantitatively com-\r\n operations in the TensorFlow framework. Finding the op- paring the ef\ufb01ciency of multiple schedules. Towards this\r\n timal schedule in this complex DAG involves evaluating goal, we formally de\ufb01ne the scheduling problem and inves-\r\n 363! combinations. Werun1000iterationsoflearningover tigate the feasibility of \ufb01nding an optimal solution. Finally,\r\n TicTac: Accelerating Distributed Deep Learning with Communication Scheduling\r\n wede\ufb01neametricthat is used to quantify the ef\ufb01ciency of in the DAG. The Cmax represents the goal of scheduling is\r\n a schedule. to minimize the last node completion time.\r\n 3.1 Scheduling Problem This problem is still open (Brucker & Knust, 2007) and\r\n simpler cases are proven to be NP-Hard. While there exist\r\n The objective is to \ufb01nd the optimal schedule of network approximations for relaxed versions of this problem, to the\r\n transfers that minimizes the iteration time by improving the best of our knowledge, there is no solution or approxima-\r\n communication/computation overlap. The network trans- tion with guaranteed bounds for our original problem.\r\n fers of parameters (recv ops) are roots in the computational\r\n graphattheworker. Thebranchofcomputationopsdepen- 3.2 De\ufb01ningOverlapCoef\ufb01cient\r\n dent on a recv op can be executed only after the network ThetwomajorcontributorstototalDNNiterationtime(T)\r\n transfer is completed. Thus, the order of network transfers are network transfer time or the communication time (N)\r\n can determine the order of computation as well as the ex- and the computation time (C). Since the computation and\r\n tent of overlap. We focus on improving the overlap, and in communication may overlap, the total time T \u2264 N + C.\r\n turn the iteration time, by choosing a near-optimal schedule GivenaGPU/CPU/TPUenvironment,weassumethecom-\r\n of parameter transfers. putation time, C, to be constant. We ignore the case of\r\n Theinputstothisoptimizationproblemare: (a)theworker computation stragglers and focus on communication.\r\n DAG, and (b) a time oracle. The time oracle (Time(op)) Wede\ufb01netwometrics that de\ufb01ne the DNN iteration time:\r\n predicts the execution time of a given op. For computa- (a) the communication/computation ratio, \u03c1 and (b) the\r\n tion ops, this indicates the elapsed time on a computation overlap coef\ufb01cient, \u03b1. The ratio of communication to com-\r\n resource. For communication ops, this represents the trans- putation, denoted by \u03c1, determines the extent of bene\ufb01ts\r\n fer time on the communication medium. We compute the achievable. When \u03c1 < 1, communication time is smaller\r\n timeassumingthattheresourceisdedicatedtotheopunder than the total computation time, providing ample opportu-\r\n consideration. nity for running GPUs at high utilization.\r\n Theoutputoftheschedulingalgorithmisafeasiblesched- Thesecondfactoraffecting the GPU utilization is the over-\r\n ule of ops in the DAG tagger by priorities. Ops in a com- lap coef\ufb01cient, \u03b1 = N+C\u2212T . N + C is the iteration\r\n putational DAG may have multiple feasible topological or- min(N,C)\r\n ders. However, some of them may result in a bad iteration time when there is no overlap, and T is the actual itera-\r\n time (as explained in Figure 1). We want to limit the exe- tion time. The difference between these quantities is the\r\n cution path to take the one that improves the training per- extent of overlap. The maximum overlap possible is given\r\n formance. We achieve this with priority numbers. Priority by min(N,C), which is achieved when the smaller quan-\r\n number is a positive integer assigned to an op in the DAG. tity completely overlaps with the large quantity. The dif-\r\n Ahigher priority op is given a lower priority number. An ference is normalized by this factor to obtain the overlap\r\n op may not be assigned a priority if it need not be ordered. coef\ufb01cient, \u03b1 \u2208 [0,1].\r\n Multiple ops may be assigned the same priority if their rel- The GPU utilization (U = C) can be represented in terms\r\n ative order is insigni\ufb01cant. of these coef\ufb01cients: T\r\n The order is enforced in the following manner. When we C 1\r\n need to select a new op from the ready-to-execute queue, U = N +C\u2212\u03b1\u2217min(N,C) = 1+\u03c1\u2212\u03b1\u2217min(\u03c1,1)\r\n we randomly choose from among the set of ops that con-\r\n tain the lowest priority number and those without any pri- The goal of our scheduling algorithms is to achieve high\r\n ority number. It is worth noting that priority only speci\ufb01es GPUef\ufb01ciencybymaximizing\u03b1,i.e., increasing the over-\r\n relative order among candidate ops in the ready-to-execute lap of communication and computation. The impact of our\r\n queue at a given resource, and the resulting order will still scheduling algorithm on \u03b1, and in turn the GPU utilization\r\n respect the topological order speci\ufb01ed by the DAG. is plotted in Figure 4 using Inception v3 with 2 workers and\r\n The problem of \ufb01nding the optimal schedule is NP-hard. 1 PS as an example.\r\n Asimpler version of the optimal execution problem with\r\n homogeneous hardware can be formally de\ufb01ned as follow 4 SCHEDULING ALGORITHMS\r\n (Using notion in (Pinedo, 2008)): P |M ,prec|C\r\n m i max In this section, we present two heuristics to derive the opti-\r\n In this formulation, Pm represents multiple parallel re- mal schedule of recv ops using a given worker DAG (\u00a73).\r\n sources with identical performance. M assigns the opera-\r\n i The intuition behind our heuristics is to prioritize trans-\r\n tions to speci\ufb01c resources, i.e., computation ops vs. com- fers that speed up the critical path in the DAG by reducing\r\n munication. prec describes the dependency relation of ops blocking on computation caused by parameter transfers.\r\n TicTac: Accelerating Distributed Deep Learning with Communication Scheduling\r\n ) 1.0 TensorFlow + TicTac 99% Algorithm 1: Property Update Algorithm\r\n 90% 65% 55% 50%\r\n ( 70%\r\n 0.8 85% 75% // Update properties for the given the set of\r\n t\r\n n outstanding read ops R\r\n e\r\n i TensorFlow\r\n c\r\n i 0.6 1 Function UpdateProperties(G, Time, R):\r\n f\r\n f\r\n e 2 foreach op \u2208 G do\r\n o 3 op.M \u2190P Time(r);\r\n C 0.4\r\n 60% 45% \u2200r\u2208op.dep\u2229R\r\n p\r\n a 4 end\r\n l\r\n r\r\n e 0.2 5 foreach op \u2208 R do\r\n v\r\n O 95% 80% 40% 6 op.P \u2190 0;\r\n 0.0 7 op.M+ \u2190+\u221e;\r\n 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 8 end\r\n C om mu nication to Computation Ratio () 9 foreach op \u2208 G \u2212 R do\r\n 10 D\u2190op.dep\u2229R;\r\n Figure 4: Improvement in GPU utilization with TicTac 11 if |D| = 1 then\r\n 12 \u2200r \u2208 D : r.P \u2190 r.P + Time(op);\r\n 13 end\r\n 14 if |D| > 1 then + +\r\n Timing-Independent Communication scheduling 15 \u2200r \u2208 D : r.M \u2190min{r.M ,op.M};\r\n 16 end\r\n (TIC): In TIC, we assign priorities based only on vertex 17 end\r\n dependencies in the DAG (ignoring the execution time of 18 end\r\n each op). Higher priorities are given to transfers which\r\n are least blocking on computation. In this algorithm, we\r\n ignore the time oracle, Time, and assume all ops have\r\n equal cost. Directly-Dependent Compute Load (recvOp.P): This\r\n Timing-Aware Communication scheduling (TAC): In property represents the computational bene\ufb01t of complet-\r\n this algorithm, we prioritize transfers that maximize \u03b1 by ing a recv op. More speci\ufb01cally, it is the total Time(op)\r\n using information on (a) vertex dependencies among ops for all ops that can be activated only by completing this\r\n speci\ufb01ed by the computational DAG, and (b) execution recvOp, but not without it. These ops are those whose\r\n time of each op estimated with time oracle. communicationdependenciescontainonlythisoutstanding\r\n recvOp(itisadmissibletohavecommunicationdependen-\r\n 4.1 Opproperties cies on other completed recv operations). For example, in\r\n Figure 1a, recv .P = Time(op ) and recv .P = 0 since\r\n 1 1 2\r\n Before delving into the algorithms, we de\ufb01ne properties noopcanexecutewithcompletion of only recv .\r\n 2\r\n associated with ops that are used in the scheduling algo-\r\n +\r\n rithms. The inputs are the worker data\ufb02ow DAG (G), ImpendingCommunicationLoad(recvOp.M ): This\r\n a time oracle (Time), available communication channels property helps us to identify candidate recv ops to be acti-\r\n on a device (C) and a set of outstanding (to-be-activated) vated, given the current recv is completed. In more detail,\r\n recvs ops (R). We assume that recv ops not in R have it is the minimum communication cost to activate a compu-\r\n their corresponding transfers completed. These properties tation op which has multiple recv dependencies including\r\n are updated using the algorithm 1. the one under consideration. For example, in Figure 1a,\r\n read .M+ = read .M+ = Cost(read )+Cost(read ).\r\n 1 2 1 2\r\n Communication Dependency (op.dep): This is the set Please note that recvOp.M+ includes the communication\r\n of recv ops that an op is directly or transitively dependent time of that recvOp.\r\n on. For example, in \ufb01gure 1a, op .dep = {recv ,recv }.\r\n 2 1 2\r\n Weextractthecommunicationdependenciesusingadepth- 4.2 Timing-Independent Communication Scheduling\r\n \ufb01rst post-\ufb01x graph traversal on the DAG. (TIC)\r\n CommunicationTime(Op.M): Communicationtimeof The goal of this algorithm is to prioritize those transfers\r\n anopisthetotalnetworktransfertimerequiredtocomplete whichreducesblockingonnetworktransfers. Ourintuition\r\n that op. For a recv op, this is the time required to complete is that information on DAG structure alone can provide sig-\r\n its corresponding transfer, given by Time(recvOp). For ni\ufb01cant improvement.\r\n other ops, this is the total time to complete all outstanding To achieve this goal, we de\ufb01ne a generic time function\r\n dependent transfers, given by which only uses the number of communication ops instead\r\n P Time(r). For example, in Figure 1a, of time taken by an op. We use this simple cost function to\r\n r\u2208op.dep\u2229R\r\n op .M = Time(recv ) and op .M = Time(recv ) +\r\n 1 1 2 1 generate the schedule in Timing-Independent Communica-\r\n Time(recv ).\r\n 2 tion scheduling (TIC).\r\n For recv ops, we de\ufb01ne two additional properties. General Time Oracle: We de\ufb01ne a simple universal time\r\n TicTac: Accelerating Distributed Deep Learning with Communication Scheduling\r\n recv D op3 Comparator: We combine results from the two cases to\r\n make a comparator that extends to multiple read ops. This\r\n recv A op1 recv C op2 is an approximate induction, which may not be correct in\r\n op3 general. The result is the Comparator function in algo-\r\n recv B op2 recv A op1 recv B rithm 3. It is easy to prove that this function is transitive\r\n and can be used for partial ordering.\r\n (a) Case 1 (b) Case 2 Theorderingalgorithmtakesapartition graph on a worker,\r\n Figure 5: Sample DAG calculates the communication dependencies, then while\r\n oracle as follows: there is an outstanding recv op, it updates properties, \ufb01nds\r\n ( the smallest recv op with respect to the comparator and\r\n Time (op) = 0 if op is not recv (1) then removes the recv from the outstanding set and assign\r\n General 1 if op is recv it a higher priority relative to others.\r\n Thecomplete solution is given in Algorithm 2. Algorithm 3: Timing-AwareCommunicationScheduling(TAC)\r\n // Compare two given recv ops\r\n 1 Function Comparator(Op ,Op ): Bool\r\n A B\r\n 2 A\u2190min(P ,M );\r\n Algorithm 2: Timing-IndependentCommunicationScheduling(TIC) A B\r\n 3 B\u2190min(PB,MA);\r\n 1 Function TIC(G) 4 if A 6= B then\r\n 2 FindDependencies(G) ; 5 return A < B\r\n 3 UpdateProperties(G,R,Time={Computation: 0, Communication: 1}); 6 else\r\n + 7 return M+ < M+\r\n 4 \u2200opinG,ifopisrecv : op.priority \u2190 op.M ; 8 end A B\r\n 5 end 9 end\r\n 10 Function TAC(G,Time)\r\n 11 FindDependencies(G) ;\r\n 4.3 Timing-AwareCommunicationScheduling(TAC) 12 R\u2190{op|\u2200opinG,opisrecv};\r\n 13 count \u2190 0;\r\n 14 while R is not empty do\r\n The goal of this algorithm is to prioritize those transfers 15 UpdateProperties(G,R,Time);\r\n which reduces the blocking of computation, i.e., speeding 16 Find the minimum op from R wrt Comparator;\r\n 17 RemoveopfromR;\r\n up transfers on the critical path. To achieve this goal, the 18 op.priority \u2190 count;\r\n algorithm focuses on two cases. First, it considers the op- 19 count \u2190 count+1;\r\n 20 end\r\n portunityforoverlappingcommunicationandcomputation. 21 end\r\n Second, in the case of equal overlap or absence of it, it\r\n looks at the impending transfers to choose one which elim-\r\n inates the computation block sooner. 5 SYSTEMDESIGN\r\n To better describe the logic, we begin with an example for In this section, we provide a brief overview of the system\r\n each case. design and implementation.\r\n Case 1: In Figure 5a, when deciding between two read The system has four main components: the tracing mod-\r\n ops, A and B, A should precede B iff: ule, the time oracle estimator, the ordering wizard, and the\r\n A\u227aB \u21d0\u21d2 T(A\u2192B)