{"title": "Resource Elasticity in Distributed Deep Learning", "book": "Proceedings of Machine Learning and Systems", "page_first": 400, "page_last": 411, "abstract": "Elasticity\u2014scaling out or in depending upon resource demand or availability\u2014allows a system to improve its efficiency or performance. This leads to potentially significant cost savings and shorter job completion times. However, elasticity is not possible in today\u2019s distributed deep learning deployments, in large part because the most widely used frameworks such as TensorFlow are built assuming that resource allocation must be fixed throughout the lifetime of the job.\n\nIn this work, we demonstrate that these assumptions are not fundamental to distributed deep learning and present the first autoscaling engine for these workloads. Our system takes into account cost as well as scaling efficiency when making scaling decisions. As a side benefit, we reuse the same autoscaling mechanisms to remove persistent stragglers. We evaluate our system by training ResNet on CIFAR-10 and ImageNet, and we find a reduction in job completion time of up to 45% and a reduction in GPU time of up to 85.1%.", "full_text": " RESOURCEELASTICITY IN DISTRIBUTED DEEP LEARNING\r\n AndrewOr1 HaoyuZhang12 MichaelJ.Freedman1\r\n ABSTRACT\r\n Elasticity\u2014scaling out or in depending upon resource demand or availability\u2014allows a system to improve its\r\n ef\ufb01ciency or performance. This leads to potentially signi\ufb01cant cost savings and shorter job completion times.\r\n However, elasticity is not possible in today\u2019s distributed deep learning deployments, in large part because the most\r\n widely used frameworks such as TensorFlow are built assuming that resource allocation must be \ufb01xed throughout\r\n the lifetime of the job.\r\n In this work, we demonstrate that these assumptions are not fundamental to distributed deep learning and present\r\n the \ufb01rst autoscaling engine for these workloads. Our system takes into account cost as well as scaling ef\ufb01ciency\r\n whenmakingscalingdecisions. As a side bene\ufb01t, we reuse the same autoscaling mechanisms to remove persistent\r\n stragglers. We evaluate our system by training ResNet on CIFAR-10 and ImageNet, and we \ufb01nd a reduction in job\r\n completion time of up to 45% and a reduction in GPU time of up to 85.1%.\r\n 1 INTRODUCTION\r\n 1400\r\n Distributed learning today often over- or under-provisions 1200 No stragglers\r\n resources. The state-of-the-art approach of assigning re- 1 straggler\r\n sources to a job is largely manual: an expert estimates what 1000\r\n is an appropriate set of resources for a given job based on 800\r\n past experiences with similar jobs. For example, for training 600\r\n ResNet (He et al., 2016) on ImageNet (Deng et al., 2009), 400\r\n it is common to choose a number of GPUs such that the 200\r\n batch size per GPU is 64 or 128 images (Jia et al., 2018; Average throughput (img/s)\r\n Ying et al., 2018; Sun et al., 2019). For workloads that 0 2 4 6 8 10 12 14 16\r\n are not well known, these guidelines do not exist and users Number of workers\r\n have to resort to trial-and-error to \ufb01nd an appropriate set of Figure 1. Even a single straggler can persistently hurt throughput\r\n resources for their jobs. scaling in synchronous SGD training. After the 4th worker, which\r\n However, a trial-and-error approach is expensive. Every is 33% slower than the other workers on average, the difference\r\n iteration destroys all processes along with their in-memory between the two curves widens as more workers are added.\r\n program state, such as preprocessed input samples and the\r\n computation graph, and this can take minutes to rebuild. ing resources that do not actually contribute to observable\r\n Further, deciding how much resources to assign to a given progress. Further, even if the amount of resources assigned\r\n job requires knowing the scaling characteristics of the job. to a job is optimal, the set of resources may not be, for\r\n This is dif\ufb01cult to estimate ahead of time without suf\ufb01cient example, when there are persistent stragglers in the system\r\n experience with the job itself or similar jobs. (see Figure 1). Existing approaches handle stragglers by\r\n For these reasons, users often rely on suboptimal but \ufb01xed overprovisioning resources, but never actually remove the\r\n sets of resources for their jobs. This means their jobs are stragglers (Abadi et al., 2016; Karakus et al., 2017).\r\n potentially either running more slowly than what the users 1.1 Resource elasticity\r\n can afford (in terms of paying for extra resources), or wast-\r\n 1Department of Compute Science, Princeton University, Prince- Contrary to existing practice, this paper argues that resource\r\n ton, NJ, USA 2Google AI, Mountain View, CA, USA. Correspon- allocation in distributed learning should be elastic. A job\r\n dence to: Andrew Or . should request more resources if throughput feedback is\r\n positive, and relinquish resources if training progress would\r\n Proceedings of the 3rd MLSys Conference, Austin, TX, USA, not be affected by the loss of these resources.\r\n 2020. Copyright 2020 by the author(s).\r\n Resource Elasticity in Distributed Deep Learning\r\n Incloudenvironments,thesystemcanincrementally\ufb01ndthe model(Keskar et al., 2016). Our goal is to provide \ufb02exible\r\n most cost ef\ufb01cient set of resources for a job, while ensuring resource allocations without affecting the quality of training.\r\n that the user is not paying for extra resources that do not\r\n contribute to training progress due to scaling limits. In 1.3 Autoscaling engine for distributed learning\r\n multi-tenant environments, the system can share resources To realize these goals, we built an autoscaling engine on\r\n moreef\ufb01ciently with other jobs running in the cluster and top of TensorFlow, using Horovod (Sergeev & Del Balso,\r\n take advantage of diurnal cycles. In both cases, straggler 2018) as the underlying communication mechanism. This\r\n mitigation comes for free; with the appropriate statistics, we architecture decouples computation from synchronization\r\n can simply replace workers that are consistently slow. across workers in the system. During changes in resource\r\n Theidea of resource elasticity has already been widely ex- allocation, our system reuses existing processes and saves\r\n plored in many other areas such as cloud services (AWS), all relevant program state in memory to minimize idle time.\r\n distributed batch processing (Dean & Ghemawat, 2008; Za- In addition to building such a system, this paper makes the\r\n haria et al., 2012), stream processing (Gedik et al., 2013), following contributions:\r\n and cluster management (Rensin, 2015). Applying it in the\r\n context of distributed learning can reap similar bene\ufb01ts. \u2022 Outline the main architectural limitations in state-of-\r\n the-art distributed learning systems that prevent re-\r\n 1.2 Mainchallenges source elasticity.\r\n Oneof the main reasons why many users still rely on the \u2022 Design new scaling heuristics for distributed learning\r\n manual trial-and-error process today is because the notion that take both throughput and cost into account.\r\n of static resource allocation is baked into state-of-the-art dis- \u2022 Present the \ufb01rst distributed learning system that han-\r\n tributed learning systems. TensorFlow (Abadi et al., 2016), dles persistent stragglers without overprovisioning re-\r\n a widely popular deep learning framework, embeds a job\u2019s sources.\r\n cluster membership information into a computation graph\r\n that cannot be easily modi\ufb01ed once training begins. Syn- This work is primarily focused on synchronous stochastic\r\n chronization between workers relies on mechanisms that gradient descent (SGD) training using the allreduce parame-\r\n wait for a \ufb01xed number of tokens or messages that corre- ter update architecture (Patarasuk & Yuan, 2009). However,\r\n sponds to the number of workers in the system. the techniques presented also apply to other forms of dis-\r\n Other popular frameworks such as PyTorch (Paszke et al., tributed learning. We do not focus on the multi-tenant use\r\n 2017) and MXNet (Chen et al., 2015) have similar con- case in depth, as the problems there are already partially\r\n straints. Although PyTorch supports dynamic graphs, they solved by SLAQ (Zhang et al., 2017b), a cluster sched-\r\n are only dynamic with respect to inputs and operations, not uler that gives more resources to jobs that converge faster.\r\n with respect to the number of processes in the system. The Rather, the goal of this paper is to take the \ufb01rst step in bring-\r\n lack of native support for dynamic resource allocation in ing attention to the need for resource elasticity in distributed\r\n these frameworks poses a signi\ufb01cant barrier for users who learning.\r\n wish to utilize their resources more ef\ufb01ciently.\r\n Realizing resource elasticity for distributed learning is more 2 BACKGROUND\r\n than just an engineering effort, however. There are funda- Inthissection, weelaborateonthemainhurdlesforresource\r\n mental differences between distributed learning and other elasticity in distributed deep learning systems today. We\r\n areas, such as distributed batch processing, that already begin by discussing the design limitations in TensorFlow\r\n support it. that makes autoscaling dif\ufb01cult (\u00a72.1), then turn to the scal-\r\n First, because these existing areas are largely driven by dy- ing complications posed by the batch size and discuss how\r\n namic workloads, resource utilization is often used as a these complications affect autoscaling (\u00a72.2).\r\n metric for scaling. In contrast, resource demands in dis-\r\n tributed learning are largely static throughout the lifetime 2.1 Architectural limitations in TensorFlow\r\n of the job; workers are often highly utilized and never idle. Thecommonpracticefortrainingdeeplearningmodelswith\r\n This means we need to design our own scaling heuristics TensorFlow involves replicating the model graph across\r\n instead of just reuse existing ones. workers in the system. Model parameters are mirrored\r\n Second, in distributed learning, scaling out generally in- across the workers and synchronized at the end of each step.\r\n volves increasing the overall batch size across all the work- Operations in the model graph that cross device boundaries\r\n ers in the system, but this affects the convergence of the are connected via special Send and Receive operations\r\n that TensorFlow adds to the graph.\r\n Resource Elasticity in Distributed Deep Learning\r\n Today, there is no easy way to change cluster membership convergence of the model (Keskar et al., 2016; Smith &\r\n at runtime. This would require replacing existing Send and Le, 2017; Hoffer et al., 2017), and the effects of this have\r\n Receiveoperationswithnewonesthatproperlyreconnect been observed at scale (Goyal et al., 2017; Jia et al., 2018;\r\n the new set of workers. However, this is non-trivial because Mikamietal., 2018; Sun et al., 2019). For common work-\r\n oncetrainingbegins,theframeworkhasalreadyperformeda loads, there are often well known thresholds for the global\r\n series of optimizations such as rewriting the graphs, building batch size before training begins to diverge. For example,\r\n the input data pipeline, and potentially JIT compiling for this threshold is 8192 for training ResNet on ImageNet, as-\r\n accelerators (XLA). suming no additional optimization techniques are applied.\r\n Further, synchronization primitives in TensorFlow rest However, \ufb01nding this threshold for arbitrary workloads is\r\n on the assumption that the number of processes is \ufb01xed an open problem that we will not address in this paper.\r\n throughout the lifetime of the program. For example, Thus, \ufb01nding an ef\ufb01cient amount of resources for a given\r\n SyncReplicasOptimizeruses\ufb01xed-sizedqueuesto job involves searching within these constraints. The search\r\n wait on all workers at the end of each step. The newer space can be large, however. Even after \ufb01xing the global\r\n MultiWorkerMirroredStrategyAPI,whichisthe batch size for a given workload, there are still many con\ufb01g-\r\n recommended method for distribution in TensorFlow 2.0, urations to explore. For example, how much less ef\ufb01cient is\r\n makessimilar assumptions. Model parameters created un- 256x32(numberofGPUstimeslocalbatchsize)vs64x128?\r\n der this strategy\u2019s scope are automatically mirrored across Is the tradeoff worth the extra cost? These questions are\r\n the speci\ufb01c replicas that were con\ufb01gured in the beginning. dif\ufb01cult to answer without suf\ufb01cient experience with the\r\n Expanding the cluster requires rebuilding the model graph job, and they are common for users deploying their models\r\n under a new distribution strategy for the new set of workers. even without elasticity.\r\n Similar limitations also exist in other widely used frame- Implications for scaling. How should the batch sizes\r\n works. Thus, to build an autoscaling system for distributed change when new workers are added to the system? There\r\n learning workloads, we must either resolve these architec- are two ways of handling this. The \ufb01rst \ufb01xes the local batch\r\n tural constraints or \ufb01nd ways to work around them. We will size, which preserves the per device ef\ufb01ciency but allows\r\n return to this in more detail in \u00a76. the global batch size to climb, and this may affect conver-\r\n gence(weakscaling). Thesecond\ufb01xestheglobalbatchsize,\r\n 2.2 Howtosetthebatchsize? which decouples convergence from scaling, but sacri\ufb01ces\r\n Depending on the context, the term batch size can refer to per device throughput since the local batch size must fall as\r\n either the number of samples assigned to each device, or the the cluster expands (strong scaling).\r\n total number of samples across all devices. In this paper, we Oursystemtakesthemiddleground: weaccept a maximum\r\n refer to the former as the local batch size and the latter as global batch size from the user and \ufb01x the local batch size\r\n the global batch size, and use the term minibatch to refer to until this threshold is crossed. If this limit is not provided,\r\n the global batch size\u2019s worth of data processed in each step. the system will assume that either the model convergence\r\n In homogeneous systems, the global batch size is simply will not be signi\ufb01cantly affected under large global batch\r\n the local batch size multiplied by the number of devices in sizes, or the user considers the potential degradation in\r\n the system. Therefore, \ufb01nding the right number of devices model quality acceptable.\r\n for a given job is tantamount to \ufb01nding the right values for 3 SYSTEMOVERVIEW\r\n these batch sizes. Today, this process is often driven by the\r\n constraints on both batch sizes. In a typical distributed learning setup, minibatches are set\r\n Constraints on the local batch size. Performance of accel- to be small enough such that the system as a whole can\r\n erators such as GPUs and TPUs rely on processing large exploit good parallelism. Thus, the time to process each\r\n batches of data for better parallelism. However, device minibatch is generally short (subsecond to seconds) and\r\n memory is limited, and all input data and the intermedi- runtime statistics available at the end of each minibatch can\r\n ate computation results in a minibatch must \ufb01t in memory. be accessed frequently.\r\n This sets a hard upper limit for the local batch size. For Theautoscaling engine has two major components: scaling\r\n example, for training ResNet-50 on ImageNet, an NVIDIA heuristics (\u00a74) and straggler detection (\u00a75) (Figure 2). It\r\n V100 GPU can \ufb01t up to 170 samples at \ufb02oat32 precision. collects these runtime statistics to build a distribution of\r\n Although there is no lower limit, resource utilization dete- throughputs for each individual worker and for each cluster\r\n rioriates signi\ufb01cantly as the local batch size decreases. size. With these distributions, the engine can make \ufb01ne-\r\n Constraints on the global batch size. Recent work has grained scaling decisions at the minibatch granularity. In\r\n shown that using large global batch sizes can affect the practice, it is more reasonable to make a decision every\r\n Resource Elasticity in Distributed Deep Learning\r\n Figure 3. Scaling schedule for scaling out (left) and scaling in\r\n (right). Autoscaling decisions follow the direction of the arrows.\r\n ThecheckmarkorredXbetweentwopointsrepresentswhether\r\n scaling conditions were met when scaling from the \ufb01rst point to the\r\n second. The red dot represents the \ufb01nal cluster size the schedule\r\n settled on.\r\n Figure 2. Autoscaling engine collects runtime statistics to make\r\n \ufb01ne-grained autoscaling decisions. cluster size is already at the upper limit). If the scaling\r\n conditions passed, then we continue adding until this is no\r\n n(e.g., 10) minibatches to ensure that we have suf\ufb01cient longer true (left \ufb01gure).\r\n numbers of data points. If the scaling conditions failed, then it means it is not ef\ufb01-\r\n Onceascaling decision has been made, the system needs to cient to expand the cluster. We must keep searching, how-\r\n only update the parameter synchronization portion of the ex- ever, because the most ef\ufb01cient point may be smaller than\r\n ecution engine. Other in-memory state used for computing our current size. Thus, when this happens, the system will\r\n gradients, such as preprocessed samples, model parameters, jump to next lowest point that it does not have runtime\r\n andcomputationgraphs,canbereusedasis. Thisallowsthe statistics for, as shown in the right \ufb01gure of Figure 3. This\r\n system to minimize idle time when responding to changes process continues until the scaling conditions begin to pass.\r\n in resource allocations.\r\n 4.2 Scaling conditions\r\n 4 SCALING HEURISTICS Scalingconditionsrefertotheconditionsforwhichaworker\r\n Anautoscaling system relies on scaling heuristics to deter- should be added to the system. In general, the system will\r\n mine when to add or remove how many workers. Scaling try to keep the number of workers at the point before the\r\n heuristics consists of two components: a scaling schedule conditions \ufb01rst begin to fail.\r\n that decides when to add or remove how many workers, and 4.2.1 Throughput scaling ef\ufb01ciency\r\n a set of scaling conditions that decides whether it is still\r\n ef\ufb01cient to keep scaling. We \ufb01rst discuss the former while The\ufb01rstmetricoursystemconsidersisincrementalthrough-\r\n assuming the existence of the latter. put scaling ef\ufb01ciency, de\ufb01ned as follows:\r\n 4.1 Scaling schedule \u2206R/\u2206k\r\n Our system uses a simple schedule that adds K workers sk,d = r\r\n k\r\n every N batches until the scaling conditions are violated.\r\n Theunit of scaling K depends on the workload and system k = current number of workers\r\n constraints; for example, a reasonable value for K is the d = number of workers to add = \u2206k\r\n number of GPUs on a machine, where each worker is as- R =aggregatethroughputwithkworkers\r\n signed a single GPU. The scaling interval N controls how k\r\n r =perworkerthroughputwithkworkers = R /k\r\n much information the system is given before making an k k\r\n \u2206R=R \u2212R\r\n autoscaling decision. A large N means decisions are more k+d k\r\n reliable because more runtime statistics are collected, but Then, the scaling condition is simply whether this term is\r\n delays \ufb01nding a more ef\ufb01cient cluster size. In practice, even above a certain threshold:\r\n a small value such as N = 10 is often suf\ufb01cient.\r\n Figure 3 shows how the schedule adds and removes workers s >S,S\u2208[0,1]\r\n k,d\r\n using the scaling conditions. The system always starts with\r\n only one data point on the throughput curve, which is not Intuitively, this term measures how much each extra worker\r\n enough to make any decisions. Therefore, the \ufb01rst step contributes to the overall performance of the system relative\r\n is always to add K workers (or remove K workers if the\r\n Resource Elasticity in Distributed Deep Learning\r\n 5000 15\r\n Total cost\r\n 4000 Throughput 14\r\n 13\r\n 3000\r\n 12\r\n 2000\r\n 11 Total cost ($) (a) (b) (c)\r\n 1000 10\r\n Average throughput (img/s)0 9 Figure 5. Example utility functions provided by the user. Y-axis is\r\n 5 10 15 20 25 30 35 40 utility in terms of absolute dollars and x-axis is total training time.\r\n Number of workers Utility functions are monotonically decreasing. In cases when\r\n Figure 4. An example of how total cost and throughput scale with jumps are present, such as in step functions, the system modi\ufb01es\r\n cluster size. Dotted line represents perfect scaling for throughput. the functions to replace the gaps with linear functions, as shown\r\n bythe red dotted line in 5(c).\r\n to how mucheachexisting worker contributes. For example, Figure 4 plots the total cost and throughput for a job training\r\n in the simple case when d = 1, this expression becomes ResNet on CIFAR-10 on a CPU cluster. The machines used\r\n Rk+1\u2212Rk are most similar to m5.4xlarge instances on EC2, which\r\n s =\r\n k,1 r have 16 cores and 64GB memory each and cost $0.768\r\n k per hour when this paper was written. Thus, we set P =\r\n Suppose R = 400, then R = 500 means the new worker 0.768/3600 since we express step time in terms of seconds.\r\n 4 5\r\n contributes the same amount as each existing worker and Findingthenumberofworkersthatgeneratesthemostvalue\r\n the scaling ef\ufb01ciency is 1 (perfect scaling), and R5 = 450 depends on (1) the total cost C of running the job with that\r\n means the scaling ef\ufb01ciency is only 0.5. A negative s\r\n k,d manyworkers,asshownabove,(2)thetotalcompletiontime\r\n meanstheaggregate throughput actually dropped as a result T of the job, and (3) how much completing a job within T\r\n of adding d workers. This can happen if the increase in is worth in dollar amount, denoted as utility U(T).\r\n communication costs outweigh the bene\ufb01ts of increased\r\n parallelism, for instance. Utility function. U(T) is a monotonically decreasing func-\r\n Note that unlike normal de\ufb01nitions of scaling ef\ufb01ciency, tion; a rational user should always prefer shorter completion\r\n s is relative to the previous number of workers instead times. Figure 5 lists common forms of U(T). A linearly\r\n k,d decreasing function indicates a constant preference for low-\r\n of relative to a single worker. This is because we often do ering T regardless of the value of T. An concave downward\r\n not have the data point for running with a single worker, as function indicates a higher preference to lowering T when\r\n these jobs are distributed. T is high, but this preference diminishes as T decreases.\r\n 4.2.2 Utility vs cost Astepfunction can be used to express a target completion\r\n time for the job.\r\n Tomanyusers,metrics like dollar cost and job completion Given a utility function and a complete total cost curve, the\r\n time are more intuitive to reason about than scaling ef\ufb01- usercandirectlycomparethetwoto\ufb01ndtheoptimalnumber\r\n ciency. If the user can provide an estimate of the utility of workers. More speci\ufb01cally, since total completion time\r\n gained from a given job in terms of dollar amount, then the T =T +t \u2217b,whereT isthetotaltimespentonthejob\r\n system can use this information to \ufb01nd a cluster size that p k p\r\n maximizes utility while minimizing cost. so far, T is a function of the number of workers k, and the\r\n user can search across all possible values of k to \ufb01nd the\r\n Total cost. We de\ufb01ne the expected total cost at any given point that minimizes U(T(k)) \u2212 C(k).\r\n point of the job as the following: Unfortunately, the total cost curve is not available in the\r\n C=C +t \u2217b\u2217k\u2217P beginning of the job. Instead, the autoscaling engine must\r\n p k discover this curve incrementally. In this case, directly\r\n Cp =pastcost, the amount spent on this job so far comparing the values of U(T(k)) and C(k) is potentially\r\n k = current number of workers problematic; a simple scaling condition such as U(T(k)) >\r\n C(k) will not work in cases when the utility function is\r\n tk = step time, a function of number of workers \ufb02at. For example, if the utility in the region T < T of\r\n 1\r\n b = number of remaining batches Figure 5(c) is always greater than the total cost, then we will\r\n P =priceofaninstance per time unit, a constant keep adding workers to the system even though total cost is\r\n increasing but utility is not.\r\n Resource Elasticity in Distributed Deep Learning\r\n Scaling condition. Instead, when deciding whether to add 1 alpha = 0.1\r\n a worker, what the system should compare is the marginal 2 fraction_threshold = 0.8\r\n utility and the marginal cost: 3 fraction_averages = {}\r\n 4\r\n U(T(k+1))\u2212U(T(k))>C(k+1)\u2212C(k) 5 # Called at the end of every step\r\n 6 def record_stats(per_worker_throughputs):\r\n If this condition is true, then the (k + 1)th worker is worth 7 m = median(per_worker_throughputs)\r\n 8 for t in per_worker_throughputs:\r\n more than it costs, and so the overall system is better off 9 fraction = t / m\r\n compared to having only k workers. This is the scaling 10 fraction_averages[worker] =\r\n condition our autoscaling engine uses. 11 alpha * fraction + (1 - alpha) *\r\n fraction_averages[worker]\r\n Tomakethisworkforstepfunctions, we must additionally 12\r\n bridge the gaps between the steps, as shown in the red dotted 13 # Called every N steps for each worker\r\n line in 5(c). We add a line that begins at the rightmost point 14 def is_straggler(worker):\r\n of the higher step (T ) and ends at either the rightmost point 15 return fraction_averages[worker] <\r\n 1 fraction_threshold\r\n of the lower step (in the case of 3 or more steps) or the initial\r\n T (T ). This enables the system to transition between steps. Algorithm 1. Straggler detection.\r\n 0\r\n Since a utility function explicitly speci\ufb01es the user\u2019s pref- Toaddress this problem, our system maintains an exponen-\r\n erences, the utility scaling condition should override the tial weighted moving average (EWMA) of the throughput\r\n decision made by the throughput scaling condition. If a fraction for each worker. A worker will be treated as a per-\r\n utility function is not provided, however, the system will sistent straggler only if its average throughput fraction falls\r\n fall back to using the throughput scaling condition. belowthethreshold. Small interruptions will only adjust the\r\n 5 STRAGGLERDETECTION average instead of marking the worker as not a straggler.\r\n Nowwehaveamechanismtodetectpersistent stragglers.\r\n Stragglers are a common source of slowdown in distributed Toensurethenumberofworkersstaysthesame,weaddare-\r\n systems. In synchronous distributed training, even a single placement worker to the system when removing a straggler.\r\n straggler can prevent the entire cluster from advancing to Asanoptimization, our system delays the actual removal of\r\n the next step due to synchronization barriers at the end of the straggler to when the replacement joins. This ensures\r\n each minibatch. Using per-worker runtime statistics, the that the system\u2019s performance at any given time is no worse\r\n autoscaling engine can easily keep track of which workers than if the straggler is not removed.\r\n are consistently slower than others on average.\r\n Our system runs a simple straggler detection algorithm at 6 IMPLEMENTATION\r\n the end of each step (Algorithm 1). A worker is considered Asdescribed in section 2, the architectural choices made in\r\n a potential straggler if its throughput as a fraction of the TensorFlowposesigni\ufb01cantchallengestorealizing resource\r\n median throughput falls below a threshold. Intuitively, this elasticity out of the box. This section describes how we\r\n threshold represents the minimum throughput (relative to worked around these challenges when building an autoscal-\r\n the median) that the system will tolerate. A threshold of ing engine on top of this framework.\r\n 1 means all workers below the median will be treated as\r\n stragglers, while a threshold of 0 means no workers will be Checkpoint restart. Before implementing our system, we\r\n treated as stragglers. One reasonable threshold is 0.8, which considered a coarse-grained alternative of achieving re-\r\n meansthesystemwill tolerate at most a 1.25x increase in source elasticity that is available out-of-the-box. This ap-\r\n total training time caused by stragglers. proach saves the model to checkpoints, kills all existing\r\n Removingaworkerassoonasitsthroughputfraction falls processes, spawns new ones on the new set of resources,\r\n below this threshold is too aggressive. Our goal is to detect and restores from the saved checkpoints. However, this ap-\r\n persistent stragglers, not workers that happen to run slowly proach is too costly (\u00a77.2) so we quickly dismissed it as a\r\n for one or two batches. One solution is to remove the worker viable mechanism for our autoscaling engine.\r\n only if it has been consistently slow for N batches in a row. Bypassing TensorFlow distribution logic. Our system\r\n However, this does not detect the case when a persistent treats TensorFlow processes as individual programs, linking\r\n straggler happens to run normally once in a while, e.g. on them only through gradient synchronization using Horovod,\r\n every N \u22121\u2019th batch. a third-party allreduce library. Each worker is unaware of\r\n the existence of the other workers, \ufb01nishing each step by ap-\r\n Resource Elasticity in Distributed Deep Learning\r\n plying a black-box (allreduce) function to its own gradients 7.1 End-to-end\r\n before applying them to its own model. Figures 6 and 7 compare training with and without using\r\n This architecture decouples the computation graph from the the autoscaling engine we built. To evaluate how our system\r\n numberofworkersinthesystem. Whenanewworkerjoins, reacts to different workloads, we train on two datasets of\r\n each existing worker only needs to detach the old allreduce vastly different sizes, CIFAR-10 and ImageNet.\r\n operation from its graph and reattach the new one. Note that In this experiment, static refers to using a con-\r\n this is signi\ufb01cantly easier to do than replacing the internal stant number of GPUs throughout the entire job, while\r\n SendandReceiveoperationsbecausetheallreduceoper- autoscaling refers to starting with a certain number\r\n ation, as an external function applied to a worker\u2019s gradients, of GPUsbutpotentially settling on a different number that\r\n is a much narrower interface. The remainder of the graph is more resource ef\ufb01cient. We use the throughput scaling\r\n does not need to change, and so we do not need to rebuild condition with a scaling ef\ufb01ciency threshold of 0.1 and\r\n the model graph. con\ufb01gure our system to scale on 4 GPU increments.\r\n Minimizing transition time. Every second of time spent\r\n on transitioning to a new resource allocation is wasted time 7.1.1 CIFAR-10\r\n that could have been used for training the model. Thus, we Wetrain ResNet-56 on CIFAR-10 for 200 epochs using a\r\n wish to minimize the transition time. \ufb01xed global batch size of 1024. The number of GPUs used\r\n Whenaworker process \ufb01rst starts, it must perform a few in each trial varies from 4 to 24, and each worker is assigned\r\n time consuming tasks before training the \ufb01rst batch, such as one GPU.\r\n loading libraries, building the model graph, preprocessing This workload is most ef\ufb01ciently run on 4 or fewer GPUs.\r\n the \ufb01rst batch of data, and warming up the GPUs assigned As shown in the static line in Figure 6(a), completion\r\n to the process, if any. Depending on the workload, these time actually increases as more GPUs are added. This is due\r\n tasks can take minutes altogether. In an autoscaling system, in large part to the large drop in performance that results\r\n it is unacceptable to have all resources go idle for minutes from having to use smaller local batch sizes.\r\n every time the system adjusts a job\u2019s resource allocation.\r\n Tominimizethetransition time, our system delays when a Ourautoscaling engine recognizes this and converges to 4\r\n new worker joins to until after it has performed all of the GPUsinall trials. By doing so, we reduce the total com-\r\n above by running a few batches on its own. This brings the pletion time by 8.23% on average and up to 16.0%. The\r\n transition time down to a few seconds for most workloads. real gain, however, comes from the potential cost savings of\r\n from discarding GPUs that do not contribute to additional\r\n Bootstrappingnewworkers. Whenanewworkerjoinsthe performance. In particular, our system reduces the total\r\n system, it must update its model parameters to the existing GPUtime,whichscaleswithcost,by58.6%onaverageand\r\n values before it can begin training, otherwise the gradients it upto85.1%(Figure6(b)).\r\n computes will not make sense to the other workers. Our sys- An autoscaling system must respond to misallocation of\r\n tem bootstraps new workers by having them synchronously resources quickly in order to reduce cost. Figure 6(c) shows\r\n fetch different slices of the model parameters from different that all trials converge to 4 workers within 61.0 seconds of\r\n existing workers in parallel. As we will see in the next sec- training and up to 78.4 seconds. This duration is dominated\r\n tion, this adds a few more seconds to the transition time for bythetimeit takes for new workers to bootstrap, which is\r\n training ResNet-50 on ImageNet. unavoidable. Note that in the mean time existing workers\r\n 7 EVALUATION continue to train and keep the resources busy.\r\n Weranourexperimentsintwoenvironments. The \ufb01rst is a 7.1.2 ImageNet\r\n cluster of 60 machines, each with 16 Intel Xeon CPUs oper- Wetrain ResNet-50 on ImageNet using a maximum global\r\n ating at 2.6GHz, 64GB of memory, and a 1 Gbps network batch size of 4096. Due to the size of the dataset, we only\r\n connection. The second is a cluster of 8 machines, each train this model for 5 epochs, which is by far long enough\r\n with 8 NVIDIA V100 GPUs, 64 Intel Xeon CPUs operating for the number of workers to converge in autoscaling\r\n at 2.2Ghz, 250GB of memory, and a 16 Gbps network con- mode. Unlike in the CIFAR-10 experiment, we \ufb01x the local\r\n nection. End-to-end evaluation is run on the GPU cluster batch size while letting global batch size rise within a limit\r\n while other benchmarks are run on the CPU cluster. of 4096. This allows the system to scale more ef\ufb01ciently,\r\n since the per GPU ef\ufb01ciency is not affected.\r\n Figure 7(a) shows that within the range of 32 and 64 GPUs,\r\n completion time is minimized with 64 GPUs. This is also\r\n Resource Elasticity in Distributed Deep Learning\r\n 1500 35000\r\n static 24 4\r\n 1400 30000 8\r\n autoscaling 20\r\n 1300 25000 12\r\n 1200 20000 16 16\r\n 20\r\n 1100 15000 12 24\r\n 1000 GPU time (s)10000 8\r\n static Number of workers\r\n 900 autoscaling 5000 4\r\n 800 4 8 12 16 20 24 0 5 10 15 20 25 0 20 40 60 80 100120140 160180\r\n Time to complete 200 epochs (s)Starting number of workersStarting number of workers Time elapsed (s)\r\n (a) Completion time (b) GPUtime (c) Number of workers\r\n Figure 6. Training ResNet-56 on CIFAR-10. In \ufb01gure 6(c), each line refers to a different starting number of workers.\r\n 5000 180000\r\n static 170000 64\r\n 4500 autoscaling\r\n 160000 56\r\n 4000\r\n 150000\r\n 3500 140000 48 32\r\n 3000 130000 40\r\n GPU time (s) 40 48\r\n 2500 120000 static Number of workers 56\r\n 110000 autoscaling 32 64\r\n 2000\r\n Time to complete 5 epochs (s)3240485664 10000032 40 48 56 64 0 100200 300 400 500600 700 800\r\n Starting number of workers Starting number of workers Time elapsed (s)\r\n (a) Completion time (b) GPUtime (c) Number of workers\r\n Figure 7. Training ResNet-50 on ImageNet. In \ufb01gure 7(c) each line refers to a different starting number of workers.\r\n the number of GPUs our autoscaling engine converges to. CIFAR-10 ImageNet\r\n Oursystemreduces the total completion time by 19.4% on autoscaling (+) 3.179 6.813\r\n average and up to 45.0%. For longer jobs such as running autoscaling (-) 2.612 4.376\r\n the same experiment for 100 epochs, we expect to see a checkpoint restart 72.756\u2217 81.186\u2217\r\n muchmoredramaticimprovement,since the time spent on\r\n running on sets of resources that are less ef\ufb01cient (e.g. with Table 1. Average idle time during transition in seconds. For au-\r\n 32workers) will be smaller as a percentage. toscaling, (+) refers to adding a worker while (-) refers to removing\r\n However, unlike in the CIFAR-10 experiment, the average one. For checkpoint restart, the transition time is the same in both\r\n GPUtimeincreases by 7.39% on average and up to 14.7% cases. (\u2217) does not include CUDA library loading time, which can\r\n (at 48 workers). This happens when the increase in perfor- add another minute.\r\n manceisnotenoughtooffset the cost of the extra workers.\r\n FromFigure 7(b), we can see that the GPU time is higher epochs starting with 32 workers, the time to reach 64 GPUs\r\n even relative to running 64 workers in static mode. The (583 seconds) will only be 1.16% of the total training time\r\n average increase of 5.66% is caused by a combination of (\u2248 100/5 * 2502 seconds).\r\n running at a suboptimal number of workers for a period of\r\n time, and the overhead from the idle time during changes in 7.2 Transition time\r\n resource allocation. Asdescribed in section 6, transition time refers to how long\r\n Scaling from the initial set of GPUs to 64 GPUs took 264 resources in the cluster go idle for while the job transitions\r\n seconds on average and up to 583 seconds (Figure 7(c)). to a new resource allocation. Minimizing this duration is\r\n The time to converging to the target number of workers crucial for ensuring high resource utilization.\r\n is much longer than in the CIFAR-10 experiment. This Table 1 compares the transition time using the checkpoint\r\n is primarily because many trials in this experiment scale restart strawman with those using our autoscaling engine.\r\n out multiple times, and each time the system needs to wait For training ResNet on both CIFAR-10 and ImageNet, tran-\r\n for the new workers to load the libraries, build the graph, sition time in our system is no more than a few seconds.\r\n fetch existing model parameters etc. Note that scaling only Adding workers takes longer because the new worker must\r\n needs to happen in the beginning so the overheads are less fetch model parameters from existing workers. For Ima-\r\n observable for long jobs. For example, if we train 100 geNet, this adds a few extra seconds to the transition time.\r\n Resource Elasticity in Distributed Deep Learning\r\n 3500\r\n $0.01/hr $0.1/hr $1.0/hr $10.0/hr 3000\r\n No stragglers\r\n 10 2500 1.33x straggler\r\n 8 10 1.33x replaced\r\n 8 2000\r\n 6 6 2x straggler\r\n 4 1500 2x replaced\r\n 4\r\n 2 2 1000 4x straggler\r\n 0 4x replaced\r\n 00 40 80 120 160 200 0 30 60 90 120 150 500\r\n Average throughput (img/s)0\r\n 100 10 20 30 40 50\r\n 80 100 Number of workers\r\n 80\r\n 60 60 Figure 9. Every 4th worker is a straggler, up to 10 stragglers. Strag-\r\n 40 40\r\n Utility ($)20 20 glers are removed shortly after they are introduced.\r\n 0\r\n 00 40 80 120 160 200 0 30 60 90 120 150\r\n 10000 all price points complete within 50 minutes, the fastest for\r\n 8000 10000 this experiment. If the user provides a step utility function\r\n 8000\r\n 6000 6000 with a different target (60 minutes in \ufb01gure 8), however,\r\n 4000 4000 our system converges to that target instead of the minimum.\r\n 2000 2000\r\n 0 Intuitively, this is because further reducing the completion\r\n 00 40 80 120 160 200 0 30 60 90 120 150\r\n time is not worth the extra cost of the additional workers,\r\n Expected completion time (min) since utility is \ufb02at below the target completion time. In\r\n Figure 8. Higher utility lowers completion times for a wider range general, once the system has reached the highest tier of the\r\n of worker prices. Each point represents a trial with a different price function, it will not make any further attempts to scale out.\r\n per worker per time unit. The left column plots the linear function 7.4 Straggler removal\r\n while the right column plots the step function. Lower plots use\r\n higher utility. As a byproduct of autoscaling, straggler removal ensures\r\n that a job is not only using the right amount of resources,\r\n Compared to checkpoint restart, however, our system has but also using the resources allocated to it ef\ufb01ciently. In\r\n signi\ufb01cantly shorter transition times, up to 95.6% reduction this experiment, we train ResNet-56 on CIFAR-10 for 100\r\n for CIFAR-10 and 91.6% reduction for ImageNet. epochsusingaglobalbatchsizeof1024. Weignorethescal-\r\n ing condition and spawn 1 worker every time after running\r\n 7.3 Utility functions 10 batches on a new cluster size. Additionally, we intro-\r\n In this section, we evaluate two utility functions used by duce 1 straggler into the system for every 3 normal workers\r\n our autoscaling engine. As described in 4.2.2, the linear added, up to 10 stragglers. Slowdown refers to computation\r\n function represents a constant utility gain for lowering the slowdown; a 2x straggler is simulated by assigning twice as\r\n total completion time regardless of what the completion manysamplestothat worker per step.\r\n time is, while the step function can be used to express a Figure 9 shows that our system responds stragglers quickly.\r\n target completion time. This experiment trains ResNet-56 Every time a straggler is introduced there will be a down-\r\n on CIFAR-10 for 50 epochs, using a global batch size of wardspike in throughput. However, these spikes are short;\r\n 1024 in all trials. a straggler is already marked for removal by the time the\r\n Figure 8 plots these utility functions with the total comple- newworkerinthenextroundjoins. In this experiment, the\r\n tion times of running with different prices per worker per average lifetime of a straggler is about 1-2 minutes, since\r\n time unit. In both the linear and step functions, lower utility that is the amount of time it takes for a new worker to join\r\n values cause a wider spread of completion times across dif- after being spawned.\r\n ferent price points. In particular, an overly expensive price Figure 10 shows that the replacing the stragglers in this man-\r\n of $10/hour for each worker causes our system to converge ner has a large effect on the completion time of the job. In\r\n to a lower number of workers since the utility gain is not particular, our straggler removal strategy reduces the com-\r\n worth it. This in\ufb02ates completion time by 2.5x for the linear pletion time by 51.4%, 34.1%, and 19.9% for a 4x, 2x, and\r\n function and 2x for the step function, compared to running 1.33x slowdown respectively. Compared to running without\r\n with $0.01/hour. stragglers, our strategy has similar completion times; all\r\n In both functions, when the value of the utility provided trials were within 12.5% of the baseline.\r\n by the user is high, the difference in price matters less. In\r\n the linear function, when the maximum utility is $10000,\r\n Resource Elasticity in Distributed Deep Learning\r\n 8000 schedules such as the exponential increase in Apache Spark\r\n No stragglers may help reduce scaling overheads. We leave exploring\r\n 6000 Stragglers\r\n Replaced alternative schedules as future work.\r\n 4000\r\n Straggler mitigation. Distributed synchronous SGD is vul-\r\n 2000 nerabletostragglers. Asasystemscales,theireffectswillbe\r\n Completion time (s)\r\n 0 1x 1.33x 2x 4x moreandmorepronounced. State-of-the-art techniques of\r\n Computation slowdown handling them include using backup workers (Abadi et al.,\r\n Figure 10. Replacing stragglers with new workers yields comple- 2016; Chen et al., 2016), redundant data encoding (Karakus\r\n tion times similar to running with no stragglers in the \ufb01rst place. et al., 2017), bounded staleness (Ho et al., 2013), and work\r\n reassignment (Harlap et al., 2016).\r\n 8 RELATEDWORK However, these are all variants of overprovisioning. For\r\n example, assigning backup workers means a small fraction\r\n Elasticity in machine learning. The idea of elasticity has of the computed results will be wasted. Further, in the event\r\n beenexploredinthecontextofdistributed machine learning. of persistent stragglers, these techniques do not address\r\n (Huang et al., 2015) considers elasticity in the context of the root cause by removing these stragglers. We argue\r\n declarative machine learning on MapReduce. However, this is largely because of the widespread assumptions that\r\n manyofthetechniques presented are designed speci\ufb01cally resources must be \ufb01xed in distributed learning workloads.\r\n for the master-worker architecture in MapReduce\u2014which Oursystemcanbeusedalongsidewiththese techniques to\r\n itself already provides the mechanisms for elasticity\u2014and handle persistent stragglers.\r\n so do not apply to modern deep learning workloads. Utility functions. Utility functions are widely used in eco-\r\n (Qiao et al., 2018) explores elasticity in the context of CPU- nomics(Ratchford, 1982) and arti\ufb01cial intelligence (Russell\r\n based machine learning using the parameter server architec- &Norvig,2016)tospecify preferences for certain actions.\r\n ture. Their discussion of multi-tenant priority as a potential In computing systems, they have been used for VM re-\r\n scaling heuristic is complementary to our work. However, source management (Minarolli & Freisleben, 2011), live\r\n the low level event-driven API they propose is incompatible video analytics (Zhang et al., 2017a), and stream manage-\r\n with state-of-the-art deep learning frameworks. ment(Carney et al., 2003).\r\n (Harlap et al., 2017) exploits dynamic pricing on public\r\n clouds in order to lower costs for machine learning work- 9 CONCLUSION\r\n loads through elasticity. Although they speci\ufb01cally target In this paper, we presented the \ufb01rst autoscaling system for\r\n the parameter server architecture and does not consider distributed deep learning. As new types of models and\r\n deep learning workloads, many of their ideas can be applied workloads arise, such a system will help bridge the gap\r\n alongside those presented in this work. between unfamiliarity with a given job to quickly being\r\n Autoscaling systems. Resource elasticity and autoscaling able to run it with high resource ef\ufb01ciency. Although we\r\n are ubiquitous outside machine learning. Production sys- have addressed the main challenges that prevent resource\r\n tems in many areas such as cloud computing (AWS; Azure; elasticity today, there are still many remaining questions\r\n GCE), batch data processing (Dean & Ghemawat, 2008; that need to be answered by the broader community.\r\n Zaharia et al., 2012) and cluster management (Vavilapalli Resourceelasticity is the \ufb01rst step of a broader effort to tune\r\n et al., 2013; Hindman et al., 2011; Rensin, 2015), already systems parameters using live performance feedback from\r\n deploy autoscaling to cut cost and increase ef\ufb01ciency. Here training. In the future, it is worth exploring whether other\r\n wewill discuss two examples to highlight the differences systems parameters, such as synchronization mechanism,\r\n between these workloads and those in distributed learning. deviceplacement,andtypesofparallelism,canbeautotuned\r\n Dynamic resource allocation in Apache Spark (Or, 2014) dynamically to improve the performance of the system.\r\n scales out exponentially while there are tasks pending to\r\n be executed, and scales in when an executor (worker) is REFERENCES\r\n idle. Autoscaling in Kubernetes (Szczepkowski & Wiel-\r\n gus, 2016) scales in and out to maintain an average CPU Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean,\r\n utilization of 50% across all pods. In distributed learning, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al.\r\n there are always pending tasks, workers are never idle, and Tensor\ufb02ow: Asystemforlarge-scalemachinelearning. In\r\n resource utilization is stable across minibatches. Therefore, 12th USENIXSymposiumonOperatingSystemsDesign\r\n these scaling heuristics do not apply and our system must andImplementation (OSDI 16), pp. 265\u2013283, 2016.\r\n devise ones that are more speci\ufb01c to our workloads. Scaling\r\n Resource Elasticity in Distributed Deep Learning\r\n AWS. Auto Scaling: https://aws.amazon.com/ conference on computer vision and pattern recognition,\r\n autoscaling/. pp. 770\u2013778, 2016.\r\n Azure. Autoscale: https://azure.microsoft. Hindman, B., Konwinski, A., Zaharia, M., Ghodsi, A.,\r\n com/en-us/features/autoscale/. Joseph, A. D., Katz, R. H., Shenker, S., and Stoica, I.\r\n Mesos: A platform for \ufb01ne-grained resource sharing in\r\n Carney, D., C\u00b8etintemel, U., Rasin, A., Zdonik, S., Cherni- the data center. In NSDI, volume 11, pp. 22\u201322, 2011.\r\n ack, M., and Stonebraker, M. Operator scheduling in a\r\n data stream manager. In Proceedings of the 29th inter- Ho,Q.,Cipar,J., Cui, H., Lee, S., Kim, J. K., Gibbons, P. B.,\r\n national conference on Very large data bases-Volume 29, Gibson, G. A., Ganger, G., and Xing, E. P. More effective\r\n pp. 838\u2013849. VLDB Endowment, 2003. distributed ml via a stale synchronous parallel parameter\r\n server. In Advances in neural information processing\r\n Chen, J., Pan, X., Monga, R., Bengio, S., and Jozefowicz, R. systems, pp. 1223\u20131231, 2013.\r\n Revisiting distributed synchronous sgd. arXiv preprint Hoffer, E., Hubara, I., and Soudry, D. Train longer, general-\r\n arXiv:1604.00981, 2016. ize better: closing the generalization gap in large batch\r\n Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, training of neural networks. In Advances in Neural Infor-\r\n T., Xu, B., Zhang, C., and Zhang, Z. Mxnet: A \ufb02exible mation Processing Systems, pp. 1731\u20131741, 2017.\r\n and ef\ufb01cient machine learning library for heterogeneous Huang, B., Boehm, M., Tian, Y., Reinwald, B., Tatikonda,\r\n distributed systems. arXiv preprint arXiv:1512.01274, S., and Reiss, F. R. Resource elasticity for large-scale\r\n 2015. machine learning. In Proceedings of the 2015 ACM SIG-\r\n Dean, J. and Ghemawat, S. Mapreduce: simpli\ufb01ed data MODInternational Conference on Management of Data,\r\n processing on large clusters. Communications of the pp. 137\u2013152, 2015.\r\n ACM,51(1):107\u2013113, 2008. Jia, X., Song, S., He, W., Wang, Y., Rong, H., Zhou, F.,\r\n Deng,J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, Xie, L., Guo, Z., Yang, Y., Yu, L., et al. Highly scal-\r\n L. Imagenet: A large-scale hierarchical image database. able deep learning training system with mixed-precision:\r\n In 2009 IEEE conference on computer vision and pattern Training imagenet in four minutes. arXiv preprint\r\n recognition, pp. 248\u2013255. Ieee, 2009. arXiv:1807.11205, 2018.\r\n GCE. Autoscalinggroupsofinstances: https://cloud. Karakus, C., Sun, Y., Diggavi, S., and Yin, W. Straggler mit-\r\n google.com/compute/docs/autoscaler/. igation in distributed optimization through data encoding.\r\n In Advances in Neural Information Processing Systems,\r\n Gedik, B., Schneider, S., Hirzel, M., and Wu, K.-L. Elastic pp. 5434\u20135442, 2017.\r\n scaling for data stream processing. IEEE Transactions Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy,\r\n on Parallel and Distributed Systems, 25(6):1447\u20131463, M., and Tang, P. T. P. On large-batch training for deep\r\n 2013. learning: Generalization gap and sharp minima. arXiv\r\n \u00b4 preprint arXiv:1609.04836, 2016.\r\n Goyal, P., Dollar, P., Girshick, R., Noordhuis, P.,\r\n Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and Mikami, H., Suganuma, H., U-chupala, P., Tanaka,\r\n He, K. Accurate, large minibatch sgd: Training imagenet Y., and Kageyama, Y. Massively distributed SGD:\r\n in 1 hour. arXiv preprint arXiv:1706.02677, 2017. ImageNet/ResNet-50 training in a \ufb02ash. arXiv preprint\r\n Harlap,A.,Cui,H.,Dai,W.,Wei,J.,Ganger,G.R.,Gibbons, arXiv:1811.05233, 2018.\r\n P. B., Gibson, G. A., and Xing, E. P. Addressing the Minarolli, D. and Freisleben, B. Utility-based resource\r\n straggler problem for iterative convergent parallel ml. In allocation for virtual machines in cloud computing. In\r\n Proceedings of the Seventh ACM Symposium on Cloud 2011 IEEE symposium on computers and communica-\r\n Computing, pp. 98\u2013111. ACM, 2016. tions (ISCC), pp. 410\u2013417. IEEE, 2011.\r\n Harlap, A., Tumanov, A., Chung, A., Ganger, G. R., and Or, A. Apache Spark Dynamic Resource Al-\r\n Gibbons, P. B. Proteus: agile ml elasticity through tiered location: https://spark.apache.org/\r\n reliability in dynamic resource markets. In Proceedings of docs/latest/job-scheduling.html#\r\n the Twelfth European Conference on Computer Systems, dynamic-resource-allocation,2014.\r\n pp. 589\u2013604, 2017. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E.,\r\n He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer,\r\n ing for image recognition. In Proceedings of the IEEE A. Automatic differentiation in PyTorch. 2017.\r\n Resource Elasticity in Distributed Deep Learning\r\n Patarasuk, P. and Yuan, X. Bandwidth optimal all-reduce al- Zhang, H., Ananthanarayanan, G., Bodik, P., Philipose, M.,\r\n gorithms for clusters of workstations. Journal of Parallel Bahl, P., and Freedman, M. J. Live video analytics at\r\n andDistributed Computing, 69(2):117\u2013124, 2009. scale with approximation and delay-tolerance. In 14th\r\n Qiao, A., Aghayev, A., Yu, W., Chen, H., Ho, Q., Gib- USENIXSymposiumonNetworkedSystemsDesignand\r\n son, G. A., and Xing, E. P. Litz: Elastic frame- Implementation (NSDI 17), pp. 377\u2013392, 2017a.\r\n work for high-performance distributed machine learn- Zhang, H., Stafman, L., Or, A., and Freedman, M. J. Slaq:\r\n ing. In 2018 {USENIX} Annual Technical Conference quality-driven scheduling for distributed machine learn-\r\n ({USENIX}{ATC}18),pp.631\u2013644,2018. ing. In Proceedings of the 2017 Symposium on Cloud\r\n Ratchford, B. T. Cost-bene\ufb01t models for explaining con- Computing, pp. 390\u2013404. ACM, 2017b.\r\n sumer choice and information seeking behavior. Manage-\r\n ment Science, 28(2):197\u2013212, 1982.\r\n Rensin, D. K. Kubernetes - Scheduling the Future at Cloud\r\n Scale. 1005 Gravenstein Highway North Sebastopol, CA\r\n 95472, 2015. URL http://www.oreilly.com/\r\n webops-perf/free/kubernetes.csp.\r\n Russell, S. J. and Norvig, P. Arti\ufb01cial intelligence: a modern\r\n approach. Malaysia; Pearson Education Limited,, 2016.\r\n Sergeev, A. and Del Balso, M. Horovod: fast and easy\r\n distributed deep learning in TensorFlow. arXiv preprint\r\n arXiv:1802.05799, 2018.\r\n Smith, S. L. and Le, Q. V. A bayesian perspective on\r\n generalization and stochastic gradient descent. arXiv\r\n preprint arXiv:1710.06451, 2017.\r\n Sun, P., Feng, W., Han, R., Yan, S., and Wen, Y. Optimizing\r\n network performance for distributed DNN training on\r\n GPUclusters: Imagenet/alexnet training in 1.5 minutes.\r\n arXiv preprint arXiv:1902.06855, 2019.\r\n Szczepkowski, J. and Wielgus, M. Autoscaling in Kuber-\r\n netes: https://kubernetes.io/blog/2016/\r\n 07/autoscaling-in-kubernetes/,2016.\r\n Vavilapalli, V. K., Murthy, A. C., Douglas, C., Agarwal,\r\n S., Konar, M., Evans, R., Graves, T., Lowe, J., Shah, H.,\r\n Seth, S., et al. Apache hadoop yarn: Yet another resource\r\n negotiator. In Proceedings of the 4th annual Symposium\r\n onCloudComputing,pp. 5. ACM,2013.\r\n XLA, T. Optimizing Compiler for TensorFlow: https:\r\n //www.tensorflow.org/xla.\r\n Ying, C., Kumar, S., Chen, D., Wang, T., and Cheng, Y. Im-\r\n age classi\ufb01cation at supercomputer scale. arXiv preprint\r\n arXiv:1811.06992, 2018.\r\n Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J.,\r\n McCauley, M., Franklin, M. J., Shenker, S., and Stoica,\r\n I. Resilient distributed datasets: A fault-tolerant abstrac-\r\n tion for in-memory cluster computing. In Proceedings of\r\n the 9th USENIX conference on Networked Systems De-\r\n sign and Implementation, pp. 2\u20132. USENIX Association,\r\n 2012.\r\n", "award": [], "sourceid": 168, "authors": [{"given_name": "Andrew", "family_name": "Or", "institution": "Princeton University"}, {"given_name": "Haoyu", "family_name": "Zhang", "institution": "Google AI"}, {"given_name": "Michael", "family_name": "Freedman", "institution": "Princeton University"}]}