{"title": "FLEET: Flexible Efficient Ensemble Training for Heterogeneous Deep Neural Networks", "book": "Proceedings of Machine Learning and Systems", "page_first": 247, "page_last": 261, "abstract": "Parallel training of an ensemble of Deep Neural Networks (DNN) on a cluster of nodes is an effective approach to shorten the process of neural network architecture search and hyper-parameter tuning for a given learning task. Prior efforts have shown that data sharing, where the common preprocessing operation is shared across the DNN training pipelines, saves computational resources and improves pipeline efficiency. Data sharing strategy, however, performs poorly for a heterogeneous set of DNNs where each DNN has varying computational needs and thus different training rate and convergence speed. This paper proposes FLEET, a flexible ensemble DNN training framework for efficiently training a heterogeneous set of DNNs. We build FLEET via several technical innovations. We theoretically prove that an optimal resource allocation is NP-hard and propose a greedy algorithm to efficiently allocate resources for training each DNN with data sharing. We integrate data-parallel DNN training into ensemble training to mitigate the differences in training rates. We introduce checkpointing into this context to address the issue of different convergence speeds. Experiments show that FLEET significantly improves the training efficiency of DNN ensembles without compromising the quality of the result. ", "full_text": "                               FLEET:FLEXIBLEEFFICIENT ENSEMBLE TRAINING FOR\r\n                                         HETEROGENEOUSDEEPNEURALNETWORKS\r\n                         HuiGuan1 LaxmikantKishorMokadam2 XipengShen1 Seung-HwanLim3 RobertPatton34\r\n                                                                        ABSTRACT\r\n                     Parallel training of an ensemble of Deep Neural Networks (DNN) on a cluster of nodes is an effective approach\r\n                     to shorten the process of neural network architecture search and hyper-parameter tuning for a given learning\r\n                     task. Prior efforts have shown that data sharing, where the common preprocessing operation is shared across the\r\n                     DNNtrainingpipelines, saves computational resources and improves pipeline ef\ufb01ciency. Data sharing strategy,\r\n                     however, performs poorly for a heterogeneous set of DNNs where each DNN has varying computational needs\r\n                     and thus different training rate and convergence speed. This paper proposes FLEET, a \ufb02exible ensemble DNN\r\n                     training framework for ef\ufb01ciently training a heterogeneous set of DNNs. We build FLEET via several technical\r\n                     innovations. We theoretically prove that an optimal resource allocation is NP-hard and propose a greedy algorithm\r\n                     to ef\ufb01ciently allocate resources for training each DNN with data sharing. We integrate data-parallel DNN training\r\n                     into ensemble training to mitigate the differences in training rates and introduce checkpointing into this context\r\n                     to address the issue of different convergence speeds. Experiments show that FLEET signi\ufb01cantly improves the\r\n                     training ef\ufb01ciency of DNN ensembles without compromising the quality of the result.\r\n               1    INTRODUCTION                                                  Aneffective strategy for shortening the process of hyper-\r\n               Recent years have witnessed rapid progress in the develop-         parameter tuning and NAS is to concurrently train a set\r\n                                                                                  of DNNs on a cluster of nodes1, which is referred to as\r\n               ment of Deep Neural Networks (DNN) and their successful            ensemble training of DNNs. We refer to an ensemble of\r\n               applications to the understanding of images, texts, and other      DNNs with the same architecture as a homogeneous en-\r\n               data from sciences to industry (Patton et al., 2018; Math-         semble. Otherwise, the ensemble is called heterogeneous\r\n               uriya et al., 2018; Ratnaparkhi & Pilli, 2012).                    ensemble.\r\n               Anessential step to apply DNNs to a new data set is hyper-         Acommonensembletrainingstrategyistoduplicate a train-\r\n               parameter tuning\u2014that is, the selection of an appropriate          ing pipeline on multiple nodes to train DNNs in parallel. A\r\n               networkarchitectureandhyper-parameters(e.g., thenumber             typical DNN training pipeline is an iterative process includ-\r\n               of layers, the number of \ufb01lters at each layer, and the learn-      ing data fetching, preprocessing, and training. For the ease\r\n               ing rate scheduling). It is called Neural Architecture Search      of description, we refer to data fetching and preprocessing\r\n               (NAS)whenthetunedparametersdetermineaDNN\u2019sarchi-                   together as preprocessing. In ensemble training, training\r\n               tecture. Manydifferentsearchstrategieshavebeenproposed             stepsarenotidenticalbecausewetrainmodelswithdifferent\r\n               such as random search (Bergstra & Bengio, 2012; Li & Tal-          architectures and con\ufb01gurations. However, preprocessing\r\n               walkar, 2019), reinforcement learning (Zoph & Le, 2016;            is redundant across the pipelines, resulting in unnecessary\r\n               Zoph et al., 2018), evolutionary methods (Salimans et al.,         CPUusageandevenpoorpipelineperformance.\r\n               2017), and Bayesian Optimization (Kandasamy et al., 2018).\r\n               Mostexisting methods used today need to train a large set          Toeliminate the redundancies, Pittman et al. (Pittman et al.,\r\n               of DNN candidates with different architectures (e.g. 450           2018) proposed data sharing where the common preprocess-\r\n               networks being trained concurrently in (Zoph et al., 2018))        ing operations are shared across training pipelines of all\r\n               to identify the best model for a particular task.                  DNNsinanensemble. Theydemonstrated that data sharing\r\n                  1                               2                               is an effective strategy to reduce computational resource\r\n                   North Carolina State University Google Inc. (The work was      utilization and improve pipeline ef\ufb01ciency. Their solution,\r\n               done when he was at NCSU.) 3 Oak Ridge National Laboratory\r\n               4Rensselaer Polytechnic Institute. Correspondence to: Hui Guan     however, assumes relatively homogeneous computational\r\n               <hguan2@ncsu.edu>,XipengShen<xshen5@ncsu.edu>.                     needs for DNNs in an ensemble. It may perform poorly\r\n               Proceedings of the 3rd MLSys Conference, Austin, TX, USA,              1A \u201cnode\u201d in this paper refers to a machine in a cluster; one\r\n               2020. Copyright 2020 by the author(s).                             node may contain one or more CPUs and GPUs\r\n                                                                             FLEET\r\n                                                                                   the computational complexity of the problem and present\r\n                                 Data Access    Raw data          State of Data\r\n                    Storage                                                        an ef\ufb01cient greedy algorithm that groups a subset of DNNs\r\n                                                                  Operation        into a unit (named \ufb02otilla) and effectively maps DNNs to\r\n                                 Preprocessed                                      GPUsina\ufb02otilla on the \ufb02y. The algorithm incurs marginal\r\n                     Training                 Preprocessing\r\n                                    Data                                           runtime overhead while balancing the progressing pace of\r\n                Figure 1: A DNN training pipeline (Pittman et al., 2018).          DNNs. Second, we develop a set of techniques to seam-\r\n                                                                                   lessly integrate distributed data-parallel training of DNN,\r\n                                                                                   preprocessing sharing, and runtime DNN-to-GPU assign-\r\n                                                                                   mentstogetherintoFLEET,the\ufb01rstensembleDNNtraining\r\n                                                              Averaged\r\n                             Preprocessor\r\n                                        Model    Gradients\r\n                Every process\r\n                                                              Gradients            framework for heterogeneous DNNs. We introduce check-\r\n                reads different\r\n                   data                                                            pointing into this context to address the issue of different\r\n                                                                                   convergence speeds. FLEET features \ufb02exible and ef\ufb01cient\r\n                Data Storage\r\n                                                              Averaged\r\n                             Preprocessor\r\n                                        Model    Gradients\r\n                                                              Gradients            communications and effective runtime resource allocations.\r\n                                                          Reduce Operation         Experiments on 100 heterogeneous DNNs on SummitDev,\r\n                                                              Averaged             the Oak Ridge Leadership Computing Facility (Sec 6.1),\r\n                             Preprocessor\r\n                                        Model    Gradients\r\n                                                              Gradients            demonstrate that FLEET can speed up the ensemble training\r\n               Figure 2:     An illustration of data-parallel DNN train-           by 1.12-1.92X over the default training method, and 1.23-\r\n               ing (Sergeev & Del Balso, 2018).                                    1.97Xoverthestate-of-the-art framework that was designed\r\n                                                                                   for homogeneous ensemble training.\r\n               for an heterogeneous ensemble due to the variance of DNN            2    BACKGROUND\r\n               model training from two algorithmic characteristics.                This section provides the necessary background of DNN\r\n               The\ufb01rst algorithmic characteristic is varying training rate.        training pipeline and data-parallel DNN training.\r\n               Training rate of a DNN is the compute throughput of pro-            DNNTraining Pipeline. As shown in Figure 1, a typical\r\n               cessing units (e.g., CPUs and GPUs) used for training the           DNNtraining pipeline is an iterative process containing\r\n               DNN.EachDNNinanheterogeneousensemblecouldhave                       three main stages: data fetching, preprocessing, and training.\r\n               varyingcomputationalneedsandthusdifferenttrainingrates              In each iteration, data is fetched to the main memory and\r\n               with the same computing resources (Canziani et al., 2016;           then run through a sequence of preprocessing operations\r\n               Sze et al., 2017). If a DNN consumes preprocessed data              such as decoding, rotation, cropping, and scaling. The\r\n               slower than other DNNs, others will have to wait for the            preprocessed data is arranged into batches and consumed\r\n               slower one before evicting current set of cached batches            bythe training stage. The batch size is the number of data\r\n               whenweemploysynchronizeddatafetchingfor data shar-                  samples used simultaneously per step.\r\n               ing to ensure that each DNN is trained using the entire\r\n               dataset. This waiting lowers the utilization of computing           The modern computing clusters and data centers have\r\n               resources in the cluster and delays the overall training time       evolved into a hybrid structure that contains both CPUs and\r\n               of the ensemble.                                                    GPUsoneachnode. Theseheterogeneous CPU-GPUclus-\r\n               Thesecond one is varying convergence speed. Due to the              ters are particularly useful for DNN training as CPUs and\r\n               differences in network architecture or hyper-parameter set-         GPUscanworktogethertoaccelerate the training pipeline.\r\n               tings, some DNNs may require a larger number of epochs              Compared to the training stage, preprocessing is usually\r\n               (one epoch goes through all data samples once) to converge          less computation intensive. To pipeline the preprocessing\r\n               than others (Krizhevsky et al., 2012; He et al., 2016; Huang        and DNN training, typically preprocessing is performed\r\n               et al., 2017; Zagoruyko & Komodakis, 2016). There can be            on CPUs while training on another batch of data happens\r\n               scenarios where a subset of DNNs in the ensemble have al-           simultaneously on GPUs.\r\n               ready converged while the shared preprocessing operations           Data-Parallel DNN Training. Data-parallel DNN training\r\n               have to keep prepossessed data for the remaining DNNs.              trains a single DNN using multiple training pipelines where\r\n               Resources allocated to these converged DNNs will be under-          each pipeline handles a different subset of data. As illus-\r\n               utilized until the training of all the DNNs is completed.           trated in Figure 2, each pipeline fetches a different subset of\r\n               Toaddress the issues, we propose FLEET, a \ufb02exible ensem-            data from storage and prepossesses data independently. In\r\n               ble training framework for ef\ufb01ciently training a heteroge-          the training stage, gradients are calculated by each pipeline\r\n               neous set of DNNs. We build FLEET via several technical             and are reduced so that every pipeline has the same aver-\r\n               innovations. First, we formalize the essence of the problem         aged gradients. The averaged gradients are used to update\r\n               into an optimal resource allocation problem. We analyze             the model to make sure each pipeline has the same copy of\r\n                                                                                                       FLEET\r\n                                   Table 1: The job of different processes.                                     group contains more than one trainer process, they perform\r\n                      Process Type            Job Description                                                   data-parallel DNNtraining for one DNN.Eachtrainergroup\r\n                                              fetch data from storage, preprocess the data, and send            has a trainer as the training group master and zero or more\r\n                      Preprocesser            the preprocessed data to its paired training group master.        trainers as the training workers. The preprocessors commu-\r\n                                              receive the preprocessed data from its paired preprocesser,       nicate directly with only some master trainers, and those\r\n                      Training Group Master scatter it within its training group, broadcast the data to\r\n                                              other training group masters, and train the DNN using             master trainers forward the preprocessed data to other train-\r\n                                              the assigned batch of data.                                       ers. Figure 3 illustrates the ensemble training pipeline in\r\n                      Training Worker         receive the assigned batch of data from its training\r\n                                              group master and use it to train the DNN.                         FLEET.ThejobofeachprocessissummarizedinTable1.\r\n                                                                                                                Ef\ufb01ciency and Flexibility.                  Two important features of\r\n                                                                                                                FLEETareitsef\ufb01ciency and \ufb02exibility.\r\n                                                               T5       T6       T7      T8                     Theef\ufb01ciencyofFLEETcomesfromitsnovelresourceallo-\r\n                                                               (D4)    (D4)     (D4)    (D4) \r\n                                                                                                                cation strategy developed for DNN ensemble training. The\r\n                                                      T1                T4                                      strategy is powered by some fundamental understanding of\r\n                                   P1\r\n                                                     (D1)              (D3)                                     this resource allocation problem, and a greedy scheduling\r\n                                                                                                                algorithm designed speci\ufb01cally to heterogeneous ensemble\r\n                                                               T2                 T3 \r\n                                   P2\r\n                                                              (D2)               (D2)                           training. The algorithm seamlessly integrates data-parallel\r\n                     Figure 3: An illustration of the ensemble training pipeline in                             distributed training with ensemble training. As illustrated\r\n                     FLEET.P1andP2arepreprocessorsandT1-T8aretrainers.                                          in Figure 3, different number of GPUs can be allocated to\r\n                    There are four training groups, (T1), (T2, T3), (T4), (T5,                                  each DNN so that the DNNs can reach a similar training\r\n                    T6, T7, T8), which train the four DNNs D1-D4 respectively.                                  rate, avoiding the pipeline inef\ufb01ciency caused by the slowest\r\n                     Edges indicate transfers of preprocessed images.                                           DNNs. It overcomes the NP-hardness of the resource allo-\r\n                                                                                                                cation problem through a greedy design, grouping DNNs\r\n                     model parameters.                                                                          into multiple \ufb02otillas and periodically (re)allocate GPUs\r\n                                                                                                                to remaining DNNs in a global ef\ufb01cient manner. It further\r\n                     Pipelines in data-parallel DNN training can run either on                                  leverages check-pointing to mitigate the issue of varying\r\n                     the same computing node using intra-node communication                                     convergence speeds among DNNs. Together, FLEET is able\r\n                    (single node multiple GPU training) or different nodes us-                                  to achieve ef\ufb01cient ensemble training while enabling data\r\n                     ing inter-node communication (multiple node multiple GPU                                   sharing to save CPU usage.\r\n                     training). For the existing communication interfaces (e.g.,                                The\ufb02exibility of FLEET is in two aspects. First, decoupling\r\n                     MPI), the intra-node communication is usually more ef\ufb01-                                    preprocessing and training using different processes2 pro-\r\n                     cient than inter-node communication. Thus, it is preferred to                              vides the \ufb02exibility in con\ufb01guring the number of preproces-\r\n                     allocate pipelines on the same computing node rather than                                  sors such that the preprocessing throughput can match the\r\n                     ondifferent nodes. As it is common to run only one pipeline                                trainers\u2019 throughput without creating too many preproces-\r\n                     on a single GPU, the number of GPUs available to train                                     sors that may waste computing resource and power. Second,\r\n                     a DNNmodelpractically limits the maximum number of                                         as each trainer is associated with one GPU, resources for\r\n                     pipelines that can be created in data-parallel DNN training.                               training can be allocated in the granularity of GPUs (rather\r\n                     3     OVERVIEWOFFLEET                                                                      than nodes as in prior work (Pittman et al., 2018)). Each\r\n                                                                                                                GPU in a node can be assigned independently to DNNs.\r\n                    This section gives an overview of FLEET. FLEET is a \ufb02ex-                                    Each DNN in an ensemble can be trained using different\r\n                     ible pipeline software architecture for ef\ufb01cient ensemble                                  numbers of GPUs concurrently, giving \ufb02exibility for han-\r\n                     training of heterogeneous DNNs. It provides \ufb02exibility for                                 dling the heterogeneity in DNNs.\r\n                     con\ufb01guring the scheduling of DNNs on nodes and GPUs                                        Two-fold Enabling Techniques. The key technical contri-\r\n                     via separation of preprocessing and training into different                                butions that make FLEET possible are two-fold. The \ufb01rst\r\n                     processes and a collection of communication schemes. It                                    is theoretical, consisting of a deep understanding of the re-\r\n                     creates ef\ufb01ciency via heterogeneity-conscious runtime re-                                  source allocation problem and some novel algorithms for\r\n                     source allocation and scheduling, plus sharing of prepro-                                  assigning DNNs to GPUs. The second is empirical, con-\r\n                     cessing results among DNNs.                                                                sisting of a set of solutions to the various challenges for\r\n                     FLEETusestwotypesofprocesses,calledpreprocessorand                                         implementing FLEET above the array of complex software\r\n                     trainer, to perform preprocessing and training separately.                                      2The reason we used processes instead of threads is due to\r\n                    Atrainer group contains at least one trainer processes and                                  the Global Interpreter Lock in Python. As FLEET is built on\r\n                     is responsible for training one DNN in the ensemble. A                                     TensorFlow which is in Python, multi-processing brings maximum\r\n                     trainer process uses one GPU for training. When a trainer                                  parallelism into the training pipeline.\r\n                                                                                                  FLEET\r\n                                               Table 2: Notations.                                        called a \ufb02otilla. So there are K \ufb02otillas being trained in the\r\n                     Notation Description                                                                 execution, one \ufb02otilla a round.\r\n                     N         the number of DNNs in an ensemble.                                         Theoretically, a round can be a time period of an arbitrary\r\n                     M         the number of GPUs available in a cluster.\r\n                     K         the number of DNN \ufb02otillas.                                                length. We \ufb01rst focus on a simple case where a round\r\n                     D         the list of DNNs in an ensemble, D = [D ,\u00b7\u00b7\u00b7 ,D       ].\r\n                                                                        1         N                       \ufb01nishes when and only when the training of all the DNNs in\r\n                     F         the list of \ufb02otillas of DNNs, F = [F ,\u00b7\u00b7\u00b7 ,F    ].\r\n                                                                   1         K\r\n                     F         the k-th \ufb02otilla of DNNs, F  =[D(k),\u00b7\u00b7\u00b7 ,D(k)].                            a \ufb02otilla \ufb01nishes (e.g., converges or the maximum training\r\n                       k                                  k       1            N\r\n                                                                                 k\r\n                     D(k)      the i-th DNN in the k-th \ufb02otilla.                                          epochs reached). In this setting, the GPUs that are done\r\n                     Ni        the number of DNNs in the k-th \ufb02otilla.\r\n                       k                                                                                  with its work in the current \ufb02otilla earlier than other GPUs\r\n                     A         the list of GPU allocations, A = [A1,\u00b7\u00b7\u00b7 ,A     ]\r\n                                                                             K                            wouldhavesomeidlewaitingtime. The simplicity of this\r\n                     Ak        a Nk-by-M matrix, the GPU allocations for the k-th \ufb02otilla of DNNs.\r\n                     a(k)      whether the j-th GPU is assigned to D(k).                                  setting, however, makes the analysis easy to understand. We\r\n                      i,j      P                                     i\r\n                       (k)        M (k)                                        (k)\r\n                     m            j=1a     , the number of GPUs assigned to D     .                       will brie\ufb02y discuss the complexities of the more general\r\n                       i                i,j                                    i\r\n                     r(k)(m) the training rate of D(k) trained with m GPUs.\r\n                      i                             i                                                     settings at the end of Section 4.2.\r\n                                                                                                          Wenowgiveaformalde\ufb01nitionoftheresourceallocation\r\n                    components (TensorFlow, Horovod, Python, MPI, etc.) on                                problem in the focused setting. Each DNN in the ensemble\r\n                    a heterogeneous Multi-GPU supercomputer like Summit-                                  are placed into at least one of the \ufb02otillas Fk,k = 1,\u00b7\u00b7\u00b7 ,K\r\n                    Dev(Sum,2019). Wepresent the two-fold contributions in                                such that a list of K \ufb02otillas F = [F1,\u00b7\u00b7\u00b7 ,FK] cover all\r\n                    the next two sections respectively.                                                   the DNNs. Each \ufb02otilla, F = [D(k),\u00b7\u00b7\u00b7 ,D(k)], contains\r\n                                                                                                                                                k         1              Nk\r\n                                                                                                          no more than M DNNs (i.e., N                      \u2264 M) such that each\r\n                                                                                                                                                         k\r\n                    4     RESOURCEALLOCATIONALGORITHMS                                                    DNNinthe \ufb02otilla can have at least one GPU. Let A =\r\n                                                                                                          [A1,\u00b7\u00b7\u00b7 ,AK] be the GPU assignment for the K \ufb02otillas of\r\n                    Ef\ufb01cientensembletrainingisessentiallyanoptimalresource                                                                                                            (k)\r\n                                                                                                          DNNs. Each assignment A is a N -by-M matrix (a                                  )\r\n                    allocation problem. The resources involve CPUs and GPUs                                                                      k           k                        i,j\r\n                                                                                                          with:\r\n                    in the modern heterogeneous computing clusters. Under the                                        (\r\n                    context of data sharing, an optimal CPU allocation sets the                                        1,     if the j-th GPU is assigned to the model D(k),\r\n                                                                                                            (k)                                                                          i\r\n                    number of preprocessors to be the one that just meets the                             a      =\r\n                                                                                                            i,j        0,     otherwise,\r\n                    computing requirement of training DNNs. GPU allocation,\r\n                                                                                                                     Nk\r\n                    however,ismuchmorecomplexanddeterminesthepipeline                                               X(k)\r\n                                                                                                           s.t.           a     \u22641 (j=1,2,\u00b7\u00b7\u00b7 ,M),\r\n                    ef\ufb01ciency. We formalize it as an optimal resource alloca-                                               i,j\r\n                    tion problem and analyze its computational complexity; the                                      i=1\r\n                                                                                                                     M\r\n                    understanding motivates our later designs of the practical                                      X(k)\r\n                                                                                                                          a     \u22651 (i=1,2,\u00b7\u00b7\u00b7 ,N ).\r\n                    algorithms and the FLEET architecture. We next start with                                               i,j                                 k\r\n                    the problem de\ufb01nition.                                                                          j=1\r\n                    4.1    ProblemDe\ufb01nition                                                               Anoptimalresource allocation is an allocation strategy of\r\n                    There are two possible paradigms for scheduling DNNs                                  available GPUs in a cluster to DNNs in an ensemble such\r\n                    on GPUs. A local paradigm assigns a DNN to a GPU                                      that the end-to-end training time of the DNNs is minimized.\r\n                    immediately when the GPU becomes vacant. A global                                     Thede\ufb01nition is as follows:\r\n                    paradigm periodically examines the remaining DNNs and                                 De\ufb01nition 1 Optimal Resource Allocation. Given a DNN\r\n                    does a global (re)assignment of the DNNs to all GPUs.\r\n                    The local paradigm is relatively easy to understand; the                              ensemble D and a cluster of nodes with totally M GPUs,\r\n                    global paradigm has the potential to avoid the local optimal                          let T(D|F,A) be the end-to-end time to \ufb01nish the training\r\n                    but is more dif\ufb01cult to design. Particularly, to effectively                          of all the DNNs according to the list F and the correspond-\r\n                    realize the global paradigm, several open questions must be                           ing GPU assignment A. The optimal resource allocation\r\n                    answered: Is an optimal scheduling algorithm feasible? If                             problem is to \ufb01nd a schedule (F\u2217,A\u2217) such that\r\n                    so, what is it? If not, how to ef\ufb01ciently approximate it? This\r\n                    section focuses on the global paradigm and explores these                                              F\u2217,A\u2217 =argminT(D|F,A)                                       (1)\r\n                    open questions. For easy reference, we put into Table 2 the                                                                F,A\r\n                                                                                                                                                       K\r\n                    important notations used in the rest of this paper.                                                               =argminXT(F |A ),                                (2)\r\n                                                                                                                                               F,A               k    k\r\n                    In this scheduling problem, the entire execution trains N                                                                        k=1\r\n                    DNNsonM GPUsinKrounds. Thebeginningofaround\r\n                    is the time for globally (re)scheduling remaining DNNs on                             where, T(Fk|Ak) is the time spent on training the DNNs in\r\n                    GPUs. The set of DNNs being trained in each round is                                  the Fk with the assignment Ak for some epochs.\r\n                                                                                  FLEET\r\n                Algorithm 1 Greedy Algorithm                                             a task (DNN training) is hard if ever possible to predict:\r\n                Input: D,M // DNNensembleandthenumberofGPUs                              There is no known method that can accurately predict the\r\n                Output: F,A//Alistof\ufb02otillas and GPU assignments                         numberofepochs(andhencethetime)neededforaDNN\r\n                  1: R = profile(D)//Pro\ufb01letrainingrates of each DNN trained             to converge. Second, the relations among tasks (DNNs)\r\n                     using m = 1,\u00b7\u00b7\u00b7 ,M number of GPUs.                                  are \u201c\ufb02uid\u201d. The training of two DNNs are theoretically\r\n                  2: F,A,cands,k = [],[],D,1                                             independent: One does not depend on another\u2019s data or\r\n                  3: while |cands| > 0 do                                                control. But when they are put into the same \ufb02otilla, they\r\n                  4:   F ,m = createFlotilla(cands,R,M) // Step 1: Create\r\n                         k    k                                                          becomerelated: They would share the same preprocessed\r\n                       a new \ufb02otilla from candidate DNNs; return the \ufb02otilla of\r\n                       DNNs(F )andGPUcountvector(m )\r\n                                 k                             k                         data and hence need to keep a similar progressing pace.\r\n                  5:   A =getGPUAssignment(F ,m )//Step2: Figureouta\r\n                         k                           k    k                              These special aspects make the problem different from prior\r\n                       GPUassignmentforthe\ufb02otilla                                        problems and call for new algorithms to be designed.\r\n                  6:   dels = train(F ,A ) // Step 3: Load the latest checkpoint if\r\n                                      k    k\r\n                       available; train DNNs in the \ufb02otilla for some epochs; return      This section describes our algorithm. It \ufb01rst introduces\r\n                       converged models (dels).                                          four principles we followed in developing the solution and\r\n                  7:   cands\u2212 = dels// Removeconverged models from candi-                then elaborates our greedy algorithm. We will explain the\r\n                       dates (cands)\r\n                  8:   F.append(F ); A.append(A ); k+ = 1                                solution in the context of the global paradigm and discuss\r\n                                    k                k\r\n                  9: end while                                                           howit is also applicable to the local paradigm at the end of\r\n                                                                                         this section.\r\n                4.2    Complexity Analysis                                               4.3.1    Principles\r\n                In this part, we argue that the Optimal Resource Allocation              Aresource allocation strategy involves grouping the DNNs\r\n                problem is NP-hard in general. The argument comes from                   into \ufb02otillas and assigning the DNNs in each \ufb02otilla to the\r\n                the classic results in Parallel Task System Scheduling. As               GPUs. Wedevelopoursolutionbyfollowingfourprinciples.\r\n                DuandLeunghaveproved(Du&Leung,1989),\ufb01ndingan                             The core of these principles is to organize tasks with less\r\n                optimal non-preemptive schedule for a Parallel Task Sys-                 variation and dependencies at the \ufb02otilla level (Principles 1\r\n                temwiththe precedence constraints consisting of chains is                and 2) and at the node level (Principles 3 and 4).\r\n                strongly NP-hard for each n > 2 (n is the number of proces-              Principle 1 DNNs in the same \ufb02otilla should be able to\r\n                sors). And, when the precedence constraints are empty, the               reach a similar training rate (e.g., images per sec) if a\r\n                problem is strongly NP-hard for each n \u2265 5. The Optimal                  proper number of GPUs are assigned to each of the DNNs.\r\n                Resource Allocation problem can be viewed as a parallel\r\n                task system scheduling problem with each DNN as a task                   This principle helps ensure a balanced pace of all GPUs,\r\n                and each GPU as a parallel processor. One subtle aspect                  which helps the DNNs in consuming the shared prepro-\r\n                is that even though the DNNs are independent, to leverage                cessed data in a similar rate to minimize the waiting time\r\n                sharedpreprocessingdataamongDNNs,anewlyfreedGPU                          of certain GPUs. This may result in multiple \ufb02otillas to be\r\n                does not take on a new DNN until the new round starts. It                created if not all DNNs in the ensemble are similar.\r\n                could be viewed as there are some pseudo precedence con-\r\n                straints between the DNNs in two adjacent rounds. So in                  Principle 2 Packing into one \ufb02otilla as many DNNs as pos-\r\n                general, the optimal solution is unlikely to be found in poly-           sible.\r\n                nomial time. Recall that our discussion has been assuming\r\n                that a new round starts only when the training of the DNNs               Thereason for this principle is two-fold. First, the through-\r\n                in the previous round is all done. If the condition is relaxed                                                             3\r\n                                                                                         put of multi-GPU training scales sublinearly with the num-\r\n                such that a round can be a time period of an arbitrary length,           ber of GPUs due to the communication overhead of ex-\r\n                the problem becomes even more complex to solve.                          changing gradients. The principle is to help maintain good\r\n                                                                                         ef\ufb01ciency of the DNNs. Second, it allows more DNNs to\r\n                4.3    GreedyAllocation Algorithm                                        share preprocessed data.\r\n                Motivated by the complexity in \ufb01nding optimal solutions                  Principle 3 When assigning multiple GPUs to a DNN, try\r\n                to the problem, we have designed a greedy algorithm for                  to use GPUs in the same node.\r\n                FLEET to assign DNNs to GPUs ef\ufb01ciently. It is worth\r\n                noting that, even though the Optimal Resource Allocation                     3If all DNNs in an ensemble has a perfect linear scaling in\r\n                problem connects with the classic Parallel Task System                   throughput, training the DNNs one after another would be the\r\n                Scheduling, several special aspects of it make it unique and             optimalstrategy. It is however often not the case in our observation.\r\n                demandnewalgorithmdesigns. First, unlike what is often                   Another practical reason for concurrently training multiple DNNs\r\n                                                                                         is hyperparameter tuning. By checking the intermediate training\r\n                assumed in the classic scheduling problems, the length of                results of those DNNs, the unpromising ones can be discarded.\r\n                                                                            FLEET\r\n               This principle is to reduce the variation in communication         are close, and M is the total number of GPUs that are\r\n                                                                                                     k\r\n               latency: inter-node communications are slower and have             already assigned to DNNs. In our experiments, \u03b4 is set to 20\r\n               morevariations than intra-node communications.                     (images/sec). The algorithm stops adding DNNs to a \ufb02otilla\r\n                                                                                  if no solution exists to Eq. 3.\r\n               Principle 4 Try to assign DNNs that need a small number            After a \ufb02otilla is formed, if there are still GPUs available,\r\n               of GPUs to the same node.                                          weassign the next GPU to the DNN in the \ufb02otilla that has\r\n               This principle is similar to Principle 2 but at the node level.    the smallest training rate iteratively until all the GPUs are\r\n               Therationale is that, although it is hard to reduce the com-       assigned. The DNN with the smallest training rate deter-\r\n               munication overhead of DNNs that need to be trained us-            mines the pipeline ef\ufb01ciency. Assigning extra GPUs to the\r\n               ing multiple nodes, we can minimize the communication              slowest DNN can improve pipeline ef\ufb01ciency.\r\n               overhead of DNNs that need a small number of GPUs by               The\ufb02otilla creation step produces a \ufb02otilla of DNNs as well\r\n               assigning them to the GPUs in the same node.                       as the GPU count vector that speci\ufb01es the number of GPUs\r\n               Basedonthefourprinciples, weproposeagreedyalgorithm                assigned to each DNN. We next explain how to properly\r\n               to solve the resource allocation problem, as described below.      assign GPUs to each DNN based on the GPU count vector\r\n                                                                                  and considering GPU locality.\r\n               4.3.2   Algorithm                                                  GPUAssignment. Thisprocedure assigns GPUs to DNNs\r\n               Thegreedy algorithm is shown in Algorithm 1 (Complexity            in a \ufb02otilla, following Principles 3 and 4. The goal of\r\n               analysis in Appendix A). It uses training rates of the DNNs,       this procedure is to \ufb01nd an assignment Ak to minimize the\r\n               R={ri(m)},i=1,\u00b7\u00b7\u00b7 ,N,m=1,\u00b7\u00b7\u00b7 ,M,whichareat-                        number of nodes involved in training each DNN. Let c(.)\r\n               tained through a short pro\ufb01ling process (line 1). We propose       be the function that counts the number of nodes involved in\r\n                                                                                                                                (k)\r\n                                                                                  training a DNN given its GPU assignment a        , which is the\r\n               pro\ufb01ling of fewer than 50 batches of training for each DNN.                                                      i\r\n               Wedeferthedetailed pro\ufb01ling process to Section 6.1.                i-th row of the assignment matrix Ak, the GPU assignment\r\n                                                                                  is an optimization problem:\r\n               Thegreedy algorithm dynamically determines the grouping\r\n               of the DNNs in an ensemble based on whether the DNN is                                  Nk      (k)\r\n               converged or not and the training rate of each DNN. Once                           minXc(ai ),\r\n                                                                                                  Ak        m(k)\r\n               a \ufb02otilla is created, an optimal GPU assignment can be                                  i=1     i\r\n               derived. Initially, all DNNs are considered as candidates                             M\r\n                                                                                                    Xk            (k)\r\n               (cands) when a new \ufb02otilla needs to be created (line 2). The                   s.t.       a   =m ,i=1,\u00b7\u00b7\u00b7 ,N,                  (4)\r\n                                                                                                          i,j     i\r\n               greedy algorithm then iterates over three main steps, \ufb02otilla                        j=1\r\n               creation (line 4), GPU allocation (line 5), and training (line\r\n                                                                                             (k)\r\n                                                                                          c(a  )\r\n               6), until all the DNNs in the ensemble are converged (i, e,        where      i   is the number of nodes involved to train the\r\n                                                                                           m(k)\r\n               cands are empty).                                                             i\r\n                                                                                  i-th DNN, scaled by m(k), the number of GPUs assigned.\r\n                                                                                                            i\r\n               Wenextdescribe the three steps in detail.                          Thesolution space is as large as Q      M!     .\r\n                                                                                                                       Nk    (k)\r\n                                                                                                                          (m !)\r\n               Flotilla Creation. This \ufb01rst step selects a set of DNNs from                                            i=1   i\r\n               candidates to create a new \ufb02otilla whose DNNs are trained          Instead of exhaustively searching for an optimal solution in\r\n               concurrently with data sharing, following Principles 1 and         the space, we propose a greedy approach that assigns GPUs\r\n               2. The Pseudo-code is in Appendix A. The algorithm \ufb01rst            to each DNN in an incremental fashion. For example, if the\r\n               identi\ufb01esthelargesttrainingratewithasingleGPU,r              =     j-th GPU is already assigned to a DNN, then the next GPU\r\n                                                                      fast        to be assigned to the DNN is the j+1-th GPU. The solution\r\n               max{r (1),\u00b7\u00b7\u00b7 ,r           (1)}, and the corresponding DNN,\r\n                       1           |cands|                                        space is reduced to the space of possible permutations of\r\n               D ,from the candidate set of DNNs. Then r                    is\r\n                 fast                                                 fast        the GPU count vector (N !). This algorithm assumes the\r\n               used as the reference training rate to search for other DNNs                                   k\r\n               that can be placed in the same \ufb02otilla. Mathematically, the        numberofGPUspernodeisthesameamongnodes,which\r\n               algorithm searches for the next DNN that can be placed into        holds in major supercomputers. It assigns GPUs to DNNs in\r\n               the \ufb02otilla by solving the following optimization problem:         the following order: (1) the DNNs whose required number\r\n                                                                                  of GPUs is a multiple of the number of GPUs per node; (2)\r\n                                     min       |ri(m)\u2212rfast|,                     the pairs of DNNs whose sum of the required number of\r\n                                Di\u2208cands\u2212F ,\r\n                                             k                                    GPUsisamultipleofthenumberofGPUspernode;(3)the\r\n                                  m=1,\u00b7\u00b7\u00b7,M                                       remaining DNNs by searching for an optimal assignment of\r\n                            s.t.   |ri(m)\u2212rfast| \u2264 \u03b4,                             GPUs. ThePseudo-code is in Appendix A.\r\n                                   m\u2264M\u2212Mk,                                (3)     The\ufb02otilla creation and GPU assignment steps ensure that\r\n               where \u03b4 is the threshold that determines if two training rates     DNNsinthesame\ufb02otillacanachievesimilar training rate\r\n                                                                        FLEET\r\n               to improve GPU utilization. We next describe how the           tails.\r\n               training step addresses the varying convergence speed issue    Communications between Preprocessors and Trainers.\r\n              via check-pointing.                                             Apreprocessor process is created through the fork opera-\r\n              Training. The training step trains the DNNs on their as-        tion. The number of preprocessors can be controlled by\r\n               signed GPUs concurrently with data sharing. Due to the         the number of trainer group masters that execute the fork\r\n               architectural difference of DNNs in a heterogeneous ensem-     operation. We establish the communications between a pre-\r\n               ble, these DNNs require a different number of epochs to        processor and its paired trainer group master through server\r\n               converge. With data sharing, converged models need to wait     process. A server process holds Python objects and allows\r\n               for the un-converged models to complete, leading to the        other processes to manipulate them using proxies. A proxy\r\n              waste of computing resources. We leverage check-pointing        is an object in the multiprocessingpackageofPython\r\n               to address the varying convergence speed issue. Speci\ufb01cally,   and refers to a shared object which lives (presumably) in\r\n               each \ufb02otilla is trained until only \u03b1 \u00b7 M GPUs remain ac-       another process. A preprocessor sends the processed data\r\n               tive for training. \u03b1 is set to 0.8 in all our experiments. We  to its training group master by writing to a Numpy object\r\n               monitor whether a model is converged at the end of each        using the object\u2019s proxy.\r\n               epoch. Once a model is converged, it is marked as complete     Data\ufb02ow Implementation.          Pipeline is the essential\r\n               and its GPUs are released. If the total number of GPUs that    schemeorganizing the different stages of DNN processing\r\n               are not released falls below \u03b1 \u00b7 M, the training of all the    together. It allows the stages to run in parallel. For example,\r\n               DNNsinthe\ufb02otilla stops. The parameters, loss history, and      while a DNN is trained on some set of data, preprocessors\r\n               epoch count of all the DNNs are check-pointed for recover-     can be preprocessing another set of data. Figure 4 illustrates\r\n               ing their training later. A DNN marked as complete will not    the data\ufb02ow implementation in FLEET.\r\n               be packed into any of the following \ufb02otillas.\r\n                                                                              The data\ufb02ow contains three pipelined steps: (1) Training\r\n               4.3.3  Application in the Local Paradigm                       group masters receive preprocessed data from their paired\r\n              Although the discussion has been assuming the global            preprocessor and put the data into a preprocessed queue QP.\r\n               paradigm, the greedy algorithm applies the local paradigm      (2) Preprocessed data from QP are broadcast to all the train-\r\n               of resource allocation as well. The training proceeds as       ing group masters through MPI. Each training group master\r\n               follows: (1) At the beginning, the algorithm forms the \ufb01rst    receives all the preprocessed data, but handles the data dif-\r\n               \ufb02otilla of DNNs and starts training them. (2) Whenever a       ferently, depending on whether data-parallel training is used.\r\n               DNNisdone, the algorithm \ufb01lls the released GPUs with           If a training group contains only one trainer (i.e., only one\r\n               newDNNs. IfnoDNNremainsuntrained,terminatewhen                 GPUisusedtotrainaDNN),thetraininggroupmasterputs\r\n                                                                              all the data into its trainer queue Q . Otherwise, the train-\r\n               all current training is done.                                                                      T\r\n                                                                              ing group master scatters the data to its trainer queue Q\r\n                                                                                                                                        T\r\n                                                                              and the distribution queue Q \u2217. The data in the distribution\r\n               5   IMPLEMENTATION                                                                          D\r\n                                                                              queue is sent to the trainer queue QT of each training group\r\n              This section describes an ef\ufb01cient training pipeline imple-     worker via MPI point-to-point communication in a separate\r\n               mentation of FLEET. We focus on the following two main         thread. (3) Each trainer (T1-T4) reads preprocessed data\r\n                                                                              fromthetrainerqueueQ toQ ,createsbatches,andfeeds\r\n               implementation challenges:                                                              T     D\r\n                                                                              each batch to the DNN model for training.\r\n               Challenge 1: Recall that FLEET has two types of pro-\r\n               cesses, preprocessor and trainer. The number of prepro-        6    EVALUATION\r\n               cessors needs to be set to meet the requirement of trainers\u2019\r\n               throughput. Thus, it is necessary for FLEET to support cre-    Weconduct a set of experiments to examine the ef\ufb01cacy\r\n               ating a different number of processes per node on a cluster    of FLEETbyansweringthefollowingquestions: (1) How\r\n               and also enable \ufb02exible communications between prepro-         much speedup can FLEET bring to ensemble training of\r\n               cessors and trainers.                                          heterogeneous DNNs? (2) How do the pros and cons of\r\n               Challenge 2: With data-parallel DNN training, prepro-          the two paradigms in FLEET designs, local and global,\r\n               cessed data from a processor is received by its paired train-  play out in handling the variations among DNNs? More\r\n               ing group master, scattered to trainers within the group (in-  speci\ufb01cally, does the greedy scheduling algorithm in FLEET\r\n               cluding the training group master), and broadcasted to the     produce favorable schedules? How much waiting time does\r\n               other training group masters. How do we build the data\ufb02ow      the round-by-round scheme in FLEET cause, compared\r\n               to enable ef\ufb01cient training pipeline?                          to eager scheduling schemes? (3) What is the overhead of\r\n                                                                              runtimepro\ufb01ling, scheduling, and checkpointing in FLEET?\r\n              Wenextdescribe the solutions and the implementation de-         We\ufb01rstdescribe the experiment settings (machines, base-\r\n                                                                                                                                                   FLEET\r\n                                                                                                                          Data Flow\u00a0 \r\n                               (if forked)                                                                                                                      comparisons), but use Caltech256 for the measurement of\r\n                                                                                                                          MPI Broadcast \r\n                                               Process 0       Process 0                       Process 0 \r\n                               Process 1 \r\n                                               Thread 2        Thread 0                        CPU Thread 1 \r\n                                                                                                                          MPI Point-to-point Communication \r\n                               Thread 0 \r\n                                                                                                                                                                end-to-end ensemble training times such that the training\r\n                                \r\n                                                                                                                     T1 \r\n                                   P1               Q                       Q                   Q                                                               can converge within the maximum 240 minutes limit that\r\n                                                      P                       T                  D\r\n                                                                                                                    (D1) \r\n                                                                                                                                     Rank 0                     SummitDevpermits.\r\n                                                                                                  \r\n                                                                                                                      T2                                        Pro\ufb01ling. To minimize the overhead of pro\ufb01ling, we only\r\n                                   P2               Q                        Q                  Q\r\n                                                      P                       T                  D\r\n                                                                                                                     (D2) \r\n                                                                                                                                                                pro\ufb01le the training rates of each DNN in the ensemble with\r\n                                                                    Q  \u2217\r\n                                                                      D\r\n                                                                                                                                     Rank 1                     the number of GPUs varying from one to M (M < M),\r\n                                                                                                                                                                                                                                                              t        t\r\n                                                                                                                                                                where M is determined based on the training rates of each\r\n                                                                            Q                   Q         T3(D2)\r\n                                                                              T                  D                                                                                t\r\n                                                                                                                                     Rank 2\r\n                                                                                                                                                                DNNonasingleGPU.Forpro\ufb01lingonm(m = 1,\u00b7\u00b7\u00b7 ,M )\r\n                                                                                                                                                                                                                                                                                      t\r\n                                                                                                                        T4 \r\n                                                                            Q                   Q\r\n                                                                              T                  D\r\n                                                                                                                       (D2)                                     GPUs,wetrain a DNNfor a maximum of 48 batches and\r\n                                                                                                                                     Rank 3\r\n                                                                                                                                                                use the training time of the last 20 batches to calculate the\r\n                             Figure 4: Illustration of the data\ufb02ow implementation. Two                                                                          exact training rate: ri(m),i = 1,\u00b7\u00b7\u00b7 ,N. Based on the\r\n                             DNNs,D1andD2,aretrainedusingfourGPUs(Ranks0-                                                                                       pro\ufb01led training rates, we estimate the training rates of each\r\n                                                                                                                                                                DNNwhen m > M . Pro\ufb01ling details are in Appendix\r\n                             3) by two training groups, (T1) and (T2,T3,T4). T1 and                                                                                                                         t\r\n                             T2are training group masters. Sizes of Q , Q                                                             and Q                     B.3. The pro\ufb01ling process also measures the throughput of\r\n                                                                                                                      P          T                 D            a range of preprocessors and uses it to set the number of\r\n                             are 2048 images, 2048 images and 10 batches respectively.                                                                          preprocessors.\r\n                             lines, etc.) in Section 6.1 and then report our experiment                                                                         Counterparts for Comparisons.\r\n                             results in Sections 6.2 and 6.3 to answer the questions.                                                                                \u2022 Baseline The baseline uses the default TensorFlow to\r\n                                                                                                                                                                          train each DNN on one GPU independently. Each\r\n                             6.1         ExperimentSettings                                                                                                               DNNtrainer has a preprocessor that preprocesses data\r\n                                                                                                                                                                          for itself independently. A GPU randomly picks one\r\n                             DNNs. TheDNNsusedinthisexperimentarederivedfrom                                                                                              yet-to-be-trained DNN whenever it becomes free until\r\n                             DenseNets (Huang et al., 2017) and ResNets (He et al.,                                                                                       there are no DNN left.\r\n                             2016). Both models are the state-of-the-art network archi-                                                                              \u2022 Homogeneous Training This is the state-of-the-art\r\n                             tectures that achieve high performance in various learning                                                                                   framework recently published (Pittman et al., 2018)\r\n                             tasks. We select these networks as the basis because, as                                                                                     for ensemble DNN training. This framework allows\r\n                             structural DNNs, they are composed of many Convolutional                                                                                     the DNNs that get trained at the same time to share\r\n                             blocks, which have a standard interface making a block                                                                                       the preprocessed data. But it is designed for homoge-\r\n                             ready to be connected with any other blocks. As a result, it                                                                                 neous DNN training, assuming no variations among\r\n                             is easy to derive new DNNs from them\u2014one just needs to                                                                                       DNNsorthesituationwherethenumberofDNNsisno\r\n                             remove or insert some Convolutional blocks. Based on the                                                                                     greater than the number of GPUs. In our experiments,\r\n                             public DNNs, we derive 100 experimental DNNs (50 from                                                                                        whenthere are more DNNs than GPUs, the framework\r\n                             DenseNet and 50 from ResNet) by randomly changing the                                                                                        randomly picks a subset of the remaining DNNs to\r\n                             block size of DenseNet and ResNet variations. The sizes of                                                                                   train, one DNN per GPU with shared preprocessed\r\n                             the DNNmodelsvaryfrom232MBto1.19GB.Thedistri-                                                                                                data. After that subset is done, it picks another subset\r\n                             bution of their training rates on a single GPU which vary                                                                                    and repeats the process until all DNNs are done.\r\n                             from 21 to 176 images/sec. (DNN details in Appendix B.1)\r\n                             System.                 All experiments are conducted on Summit-                                                                        \u2022 FLEET-GThisisFLEETintheglobalparadigm.\r\n                             Dev(Sum,2019)atOakRidgeNationalLab. Eachnodeis                                                                                          \u2022 FLEET-L This is FLEET in the local paradigm as\r\n                             equipped with two IBM POWER8 CPUs, 256GB DRAM,                                                                                               described in Section 4.3.3. Its difference from FLEET-\r\n                             and four NVIDIA Tesla P100 GPUs. FLEET is built on                                                                                           Gisthat as soon as a DNN is done, the released GPUs\r\n                             Tensor\ufb02ow 1.12 (as the core training engine), Horovod                                                                                        are immediately used to train some remaining DNNs;\r\n                             v0.15.2 (Sergeev & Del Balso, 2018) (as the basis for dis-                                                                                   which DNNs are picked is determined by the greedy\r\n                             tributed DNN training), and mpi4py v3.0.0 (for the pipeline                                                                                  algorithm as in FLEET-G, but only locally (for the\r\n                             construction). CUDA version is 9.2. (System details in                                                                                       newly released GPUs) rather than globally.\r\n                             Appendix B.2)\r\n                             Datasets. The datasets used in the experiments are Ima-                                                                            6.2        End-to-EndSpeedups\r\n                             geNet (Deng et al., 2009) and Caltech256 Object Category\r\n                             Dataset (Cal, 2007). ImageNet contains 1,261,406 training                                                                          Figure 5 reports the speedups of the three methods over\r\n                             images and Caltech256 contains 30,606 training images.                                                                             the baseline method, in terms of the end-to-end ensemble\r\n                             Weuse ImageNet whenever possible (e.g.,for throughput                                                                              training time of the 100 DNNs. All runtime overhead for\r\n                                                                                                                                                FLEET\r\n                                            2.5                                                                                                             Table 3: Mean and standard deviation of the running length\r\n                                                  Homogeneous[24]                                          FLEET-G                                          of DNNsinseconds. (80 GPUs, 100 DNNs)\r\n                                                                FLEET-L\r\n                                              2                                                                                                                                     Technique            Flotilla ID           Mean             Std. Dev.\r\n                                                                                                                                                                                     Baseline            -                     10372.2          4178.9\r\n                                                                                                                                                                                    FLEET-L              -                     6213             3580.0\r\n                                                                                                                                                                                                         0                     2067.9           54.7\r\n                                            1.5                                                                                                                                                          1                     335.6            48.4\r\n                                                                                                                                                                                                         2                     2291.9           26.0\r\n                                                                                                                                                                                    FLEET-G              3                     415.5            51.9\r\n                                                                                                                                                                                                         4                     1072.2           364.0\r\n                                          Speedup                                                                                                                                                        5                     2322.3           216.0\r\n                                              1\r\n                                                                                                                                                                      Table 4: Scheduling and checkpointing overhead.\r\n                                                                                                                                                                       (#GPU,              Total Training            Scheduling                  Checkpointing\r\n                                            0.5                                                                                                                        #DNN)               Time(in sec)              Overhead                    Overhead\r\n                                                                                                                                                                                                                     in sec        in %          in sec          in %\r\n                                                                                                                                                                       (20,100)            55200.1                   20.1          0.037         1496.0          2.7\r\n                                                                                                                                                                       (40,100)            30204.8                   15.8          0.054         1156.0          3.8\r\n                                                                                                                                                                       (60,100)            24495.0                   14.0          0.060         986.0           4.0\r\n                                              0                                                                                                                        (80,100)            21891.0                   12.0          0.057         816.0           3.7\r\n                                                  (20,100)   (40,100)   (60,100)   (80,100)   (100,100) (120,100) (140,100) (160,100)\r\n                                                                                (#GPUs, #DNNs)                                                                         (100,100)           18359.1                   10.1          0.058         782.0           4.3\r\n                             Figure 5: The averaged speedups over the baseline in terms                                                                                (120,100)           15323.9                   9.9           0.068         680.0           4.4\r\n                                                                                                                                                                       (140,100)           13366.3                   9.3           0.073         680.0           5.1\r\n                             of the end-to-end time for training a 100-DNN ensemble.                                                                                   (160,100)           11825.2                   10.2          0.092         748.0           6.3\r\n                            Theerror bars show the variations.\r\n                                                                                                                                                            \ufb01rst \ufb01ve \ufb02otillas in FLEET-G (80GPU,100DNN). In com-\r\n                                                             Homogeneous[24]                 FLEET-L           FLEET-G\r\n                                                                                                                                                            parison to the data in the baseline and FLEET-L (top rows in\r\n                                                                                                                                                            Table 3), the DNNs show much smaller variations in length,\r\n                                                                                                                                                            whichindicate the effectiveness of the GPU allocations in\r\n                                         100.0                                                                                                              FLEET-Gineveningoutthedifferences among DNNs. At\r\n                                           50.0                                                                                                             the beginning, we thought that the catch to FLEET-G is\r\n                                                                                                                                                            the waiting time of some GPUs after they are done with\r\n                                           10.0                                                                                                             their work in a round. Our experiments, however, show\r\n                                            5.0                                                                                                             the opposite effects. As Figure 6 shows, the average wait-\r\n                                   Wait time/#GPU in seconds                                                                                                ing time per GPU is smallest for FLEET-G. The reason is\r\n                                                                                                                                                            that the other methods all suffer long waiting time at the\r\n                                                      20 GPU      40 GPU      60 GPU     80 GPU  100 GPU  120 GPU  140 GPU  160 GPU \r\n                                                     100 DNN 100 DNN 100 DNN 100 DNN 100 DNN 100 DNN 100 DNN 100 DNN                                        end; because of their suboptimal resource allocation, some\r\n                                                       Figure 6: Waiting time per GPU.                                                                      GPUs have to work long after others to \ufb01nish up the last\r\n                                                                                                                                                            few DNNs. FLEET-Lgives notable but fewer speedups for\r\n                             FLEETisincluded. We repeat each measurement multiple                                                                           its less favorable decisions at resource allocation due to the\r\n                             times and report the average and error bars.                                                                                   local view.\r\n                             It shows the results in eight settings. The prior homoge-                                                                      Overall, FLEET gives larger speedups when #GPU >\r\n                             neous framework shows large slowdowns in the \ufb01rst four                                                                         #DNN. It is worth noting that in such a setting, there are\r\n                             settings where the number of GPUs is less than the number                                                                      still many \ufb02otillas to schedule and FLEET scheduler plays\r\n                             of DNNs. The slowdowns are due to the waiting of other                                                                         an important role. The reason is that in many cases, the\r\n                             GPUsfortheslowestDNNto\ufb01nishineachround,shown                                                                                   FLEETschedulerassigns multiple GPUs to one DNN. For\r\n                             in Figure 6. In the other four settings, the homogeneous                                                                       instance, 20 \ufb02otillas were created when training 100 DNN\r\n                             frameworkperformssimilarlyasthebaselinedoes: Asthere                                                                           on120GPUsand22\ufb02otillaswerecreatedwhentraining100\r\n                             are more GPUs than DNNs, there is only one round, in                                                                           DNNson160GPUs. When#GPU<#DNN,thespeedups\r\n                            which, the two methods use resource similarly. The shar-                                                                        from FLEET are not that signi\ufb01cant but still substantial:\r\n                             ing of preprocessing in the homogeneous framework does                                                                         113\u2013120%forthreeoutofthefoursuchsettings in Figure 5.\r\n                             not generate speedups for these DNN trainings because the\r\n                             preprocessing is not the bottleneck for them.                                                                                  6.3         Overhead\r\n                             FLEET-G gives the best overall performance, producing                                                                          Table 4 reports the breakdown of the runtime overhead of\r\n                            1.12-1.92X speedups over the baseline. The primary reason                                                                       FLEET-G.Theoverheadofschedulingandcheckpointingis\r\n                             for the speedups come from its better resource allocation                                                                      at most 0.1% and 6.3% of the end-to-end training time in all\r\n                             to the DNNs. The bottom of Table 3 reports the mean and                                                                        the settings. Recall that, due to wall-clock-time limitation\r\n                             standard deviations of the running lengths of DNNs in the                                                                      of SummitDev, we have used the small Caltech256 dataset.\r\n                                                                        FLEET\r\n              For large datasets (e.g., ImageNet), the overhead would be-     in FLEET. First, they treat different jobs as independent\r\n              comenegligible. The pro\ufb01ling overhead is independent of         black boxes. Without the MPI communication mechanisms\r\n              dataset size and solely depends on ensemble size. Recall        we put into FLEET to enable \ufb02exible data exchanges of\r\n              that, pro\ufb01ling needs the DNN to train for only a few steps in   TensorFlow-based workers and preprocessors, these sched-\r\n              parallel. Its overhead is marginal for typical DNN trainings    ulers cannot \ufb02exibly adjust the number of workers for a\r\n              on large datasets that take hours or days to train. On recent   DNNtraining. Second, as they treat the DNNs as separate\r\n              GPUs,afeature called Multi-Process Services (MPS) could         jobs, they cannot support the coordinations across DNNs\r\n              potentially allow multiple DNNs to be co-scheduled to a         in an ensemble, such as, the sharing of preprocessed data,\r\n              single GPU and run concurrently. It is not considered in the    cooperated checkpointing at the appropriate times.\r\n              current FLEET. To consider it, some co-run predictive mod-      Load balancing techniques for parallel computers such as\r\n              els could help, which could quickly predict the performance     nearest neighbor assignment to dynamically distribute work-\r\n              of a DNN when it co-runs with a set of other DNNs on one        loads have been studied in (Kumar et al., 1994). The way\r\n              GPU. The predictive performance can then be combined            FLEETdistributes DNN training workloads are fundamen-\r\n              with the existing performance models in FLEET to guide          tally different because an initial task assignment is not set\r\n              the scheduling of DNNs.                                         at the beginning but dynamically determined based on both\r\n               7   RELATEDWORK                                                the convergence status and the training rate of each DNN.\r\n              Muchresearch has been done to accelerate the training of        8    CONCLUSIONS\r\n              a single DNN over distributed systems, such as Tensor-          This paper presents a systematic exploration on enabling\r\n              \ufb02owfrom Google (Dean et al., 2012; Abadi et al., 2016),         \ufb02exibleef\ufb01cientensembletrainingforheterogeneousDNNs.\r\n              Project Adams from Microsoft (Chilimbi et al., 2014), Fire-     It addresses two-fold challenges. First, it formalizes the\r\n              Caffe(Iandolaetal., 2016), PipeDream(Harlapetal.,2018),         essence of the problem into an optimal resource alloca-\r\n              and GPipe (Huang et al., 2018). All those studies have fo-      tion problem, analyzes its computational complexity, and\r\n              cusedonimprovingthetrainingspeedofanindividualDNN               presents an ef\ufb01cient greedy algorithm to effectively map\r\n              rather than ensemble training.                                  DNNs to GPUs on the \ufb02y. Second, it develops a set of\r\n              Recently, ensemble training starts drawing more attention.      techniques to seamlessly integrate distributed data-parallel\r\n              Pittman et al. (Pittman et al., 2018) explored \ufb02exible com-     training of DNN, preprocessing sharing, and runtime DNN-\r\n              munication strategies for training DNN ensembles and pro-       to-GPU assignments together into a software framework,\r\n              posed data sharing to eliminate pipeline redundancy and         FLEET.Experiments on 100 heterogeneous DNNs on Sum-\r\n              improve pipeline ef\ufb01ciency.     The work is designed for        mitDevdemonstrate that FLEET can speed up the ensemble\r\n              DNNswithsimilar computational needs. There are some             training by 1.12-1.92X over the default training method,\r\n              other efforts (Garipov et al., 2018; Loshchilov & Hutter,       and1.23-1.97Xoverthestate-of-the-art framework that was\r\n              2016) on ensemble training, but they focus on designing         designed for homogeneous ensemble training.\r\n              lightweight methods to form high-performing ensembles in-\r\n              stead of improving pipeline ef\ufb01ciency of ensemble training.     ACKNOWLEDGEMENT\r\n              HiveMind (Narayanan et al., 2018) is a system designed\r\n              for accelerating the training of multiple DNNs on a single      Wethankthe anonymous MLSys reviewers for the helpful com-\r\n              GPUbyfusing common operations (e.g., preprocessing)             ments. This material is based upon work supported by the National\r\n              across models. It, however, lacks the essential support for     Science Foundation (NSF) under Grant No. CCF-1525609 and\r\n              distributed DNN training.                                       CCF-1703487. Any opinions, \ufb01ndings, and conclusions or rec-\r\n              Another line of research that is relevant to this work is task  ommendations expressed in this material are those of the authors\r\n              scheduling on clusters or work\ufb02ow management systems.           and do not necessarily re\ufb02ect the views of NSF. This manuscript\r\n              The scheduling of a set of tasks or workloads on clusters       has been authored by UT-Battelle, LLC, under contract DE-AC05-\r\n              or multiprocessor systems has been extensively studied in       00OR22725withtheUSDepartmentofEnergy(DOE).TheUS\r\n              the literature (Turek et al., 1992; Shmoys et al., 1995; Ur-    government retains and the publisher, by accepting the article for\r\n              gaonkar et al., 2002; Augonnet et al., 2011; Zaharia et al.,    publication, acknowledges that the US government retains a nonex-\r\n              2008; Grandl et al., 2015; Chowdhury et al., 2016; Xu et al.,   clusive, paid-up, irrevocable, worldwide license to publish or repro-\r\n              2018; Ousterhout et al., 2013; Cheng et al., 2016; Delim-       duce the published form of this manuscript, or allow others to do\r\n              itrou & Kozyrakis, 2014; Feitelson et al., 1997). Recent        so, for US government purposes. DOE will provide public access\r\n              workincluding Gandiva (Xiao et al., 2018) and Tiresias (Gu      to these results of federally sponsored research in accordance with\r\n              et al., 2019) design GPU cluster managers tailored for DNN      the DOE Public Access Plan (http://energy.gov/downloads/doe-\r\n              workloads. They, however, lack the \ufb02exibility supported         public-access-plan).\r\n                                                                     FLEET\r\n              REFERENCES                                                     Computer Architecture News, volume 42, pp. 127\u2013144.\r\n              Caltech256 object category dataset caltechauthors. https:      ACM,2014.\r\n                //authors.library.caltech.edu/7694/,12                     Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-\r\n                2007. Accessed On 4/3/2019.                                  Fei, L. ImageNet: A Large-Scale Hierarchical Image\r\n              Summit user guide oak ridge leadership comput-                 Database. In CVPR09, 2009.\r\n                ing    facility.        https://www.olcf.ornl.             Du,J.andLeung,J.Y.-T. Complexityofschedulingparallel\r\n                gov/for-users/system-user-guides/                            task systems. SIAM Journal on Discrete Mathematics, 2\r\n                summitdev-quickstart-guide/, 2019.                 Ac-       (4):473\u2013487, 1989.\r\n                cessed 3/3/2019.\r\n              Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean,  Feitelson, D. G., Rudolph, L., Schwiegelshohn, U., Sevcik,\r\n                J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al.   K. C., and Wong, P. Theory and practice in parallel job\r\n                Tensor\ufb02ow: A system for large-scale machine learning.        scheduling. In Workshop on Job Scheduling Strategies\r\n                In 12th {USENIX} Symposium on Operating Systems              for Parallel Processing, pp. 1\u201334. Springer, 1997.\r\n                Design and Implementation ({OSDI} 16), pp. 265\u2013283,        Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D. P., and\r\n                2016.                                                        Wilson, A. G. Loss surfaces, mode connectivity, and fast\r\n              Augonnet, C., Thibault, S., Namyst, R., and Wacrenier, P.-     ensembling of dnns. In Advances in Neural Information\r\n                A. Starpu: a uni\ufb01ed platform for task scheduling on          Processing Systems, pp. 8789\u20138798, 2018.\r\n                heterogeneous multicore architectures. Concurrency and     Grandl, R., Ananthanarayanan, G., Kandula, S., Rao, S., and\r\n                Computation: Practice and Experience, 23(2):187\u2013198,         Akella, A. Multi-resource packing for cluster schedulers.\r\n                2011.                                                        ACMSIGCOMMComputerCommunicationReview,44\r\n              Bergstra, J. and Bengio, Y.   Random search for hyper-         (4):455\u2013466, 2015.\r\n                parameter optimization. Journal of Machine Learning        Gu,J., Chowdhury, M., Shin, K. G., Zhu, Y., Jeon, M., Qian,\r\n                Research, 13(Feb):281\u2013305, 2012.                             J., Liu, H., and Guo, C. Tiresias: A {GPU} cluster man-\r\n              Canziani, A., Paszke, A., and Culurciello, E. An analysis      ager for distributed deep learning. In 16th {USENIX}\r\n                of deep neural network models for practical applications.    SymposiumonNetworkedSystemsDesignandImplemen-\r\n                arXiv preprint arXiv:1605.07678, 2016.                       tation ({NSDI} 19), pp. 485\u2013500, 2019.\r\n              Cheng,D.,Rao,J., Guo, Y., Jiang, C., and Zhou, X. Improv-    Harlap, A., Narayanan, D., Phanishayee, A., Seshadri, V.,\r\n                ing performance of heterogeneous mapreduce clusters          Devanur, N., Ganger, G., and Gibbons, P. Pipedream:\r\n                with adaptive task tuning. IEEE Transactions on Parallel     Fast and ef\ufb01cient pipeline parallel dnn training. arXiv\r\n                andDistributed Systems, 28(3):774\u2013786, 2016.                 preprint arXiv:1806.03377, 2018.\r\n              Chilimbi, T., Suzue, Y., Apacible, J., and Kalyanaraman,     He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn-\r\n                K. Project adam: Building an ef\ufb01cient and scalable deep      ing for image recognition. In Proceedings of the IEEE\r\n                learning training system. In 11th {USENIX} Sympo-            conference on computer vision and pattern recognition,\r\n                sium on Operating Systems Design and Implementation          pp. 770\u2013778, 2016.\r\n                (OSDI), pp. 571\u2013582, 2014.                                 Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger,\r\n              Chowdhury, M., Liu, Z., Ghodsi, A., and Stoica, I. {HUG}:      K. Q. Densely connected convolutional networks. In\r\n                Multi-resource fairness for correlated and elastic de-       Proceedings of the IEEE conference on computer vision\r\n                mands. In 13th {USENIX} Symposium on Networked               andpattern recognition, pp. 4700\u20134708, 2017.\r\n                Systems Design and Implementation ({NSDI} 16), pp.         Huang, Y., Cheng, Y., Chen, D., Lee, H., Ngiam, J., Le,\r\n                407\u2013424, 2016.                                               Q. V., and Chen, Z. Gpipe: Ef\ufb01cient training of giant\r\n              Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M.,         neuralnetworksusingpipelineparallelism. arXivpreprint\r\n                Mao,M.,Senior, A., Tucker, P., Yang, K., Le, Q. V., et al.   arXiv:1811.06965, 2018.\r\n                Large scale distributed deep networks. In Advances in      Iandola, F. N., Moskewicz, M. W., Ashraf, K., and Keutzer,\r\n                neural information processing systems, pp. 1223\u20131231,        K. Firecaffe: near-linear acceleration of deep neural\r\n                2012.                                                        network training on compute clusters. In Proceedings of\r\n              Delimitrou, C. and Kozyrakis, C. Quasar: resource-ef\ufb01cient     the IEEE Conference on Computer Vision and Pattern\r\n                and qos-aware cluster management. In ACM SIGARCH             Recognition, pp. 2592\u20132600, 2016.\r\n                                                                          FLEET\r\n               Kandasamy, K., Neiswanger, W., Schneider, J., Poczos, B.,        Salimans, T., Ho, J., Chen, X., Sidor, S., and Sutskever,\r\n                 and Xing, E. P. Neural architecture search with bayesian          I. Evolution strategies as a scalable alternative to rein-\r\n                 optimisationandoptimaltransport. InAdvancesinNeural               forcement learning. arXiv preprint arXiv:1703.03864,\r\n                 Information Processing Systems, pp. 2016\u20132025, 2018.              2017.\r\n               Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet        Sergeev, A. and Del Balso, M. Horovod: fast and easy\r\n                 classi\ufb01cation with deep convolutional neural networks.            distributed deep learning in tensor\ufb02ow. arXiv preprint\r\n                 In Advances in neural information processing systems,             arXiv:1802.05799, 2018.\r\n                 pp. 1097\u20131105, 2012.                                           Shmoys, D. B., Wein, J., and Williamson, D. P. Scheduling\r\n               Kumar, V., Grama, A. Y., and Vempaty, N. R. Scalable load           parallel machines on-line. SIAM J. Comput., 24(6):1313\u2013\r\n                 balancing techniques for parallel computers. Journal of           1331, December 1995. ISSN 0097-5397. doi: 10.1137/\r\n                 Parallel and Distributed computing, 22(1):60\u201379, 1994.            S0097539793248317. URL http://dx.doi.org/\r\n               Li, L. and Talwalkar, A.       Random search and repro-             10.1137/S0097539793248317.\r\n                 ducibility for neural architecture search. arXiv preprint      Sze, V., Chen, Y.-H., Yang, T.-J., and Emer, J. S. Ef\ufb01cient\r\n                 arXiv:1902.07638, 2019.                                           processing of deep neural networks: A tutorial and survey.\r\n               Loshchilov, I. and Hutter, F.       Sgdr:    Stochastic gra-        Proceedings of the IEEE, 105(12):2295\u20132329, 2017.\r\n                 dient descent with warm restarts.          arXiv preprint      Turek, J., Wolf, J. L., Pattipati, K. R., and Yu, P. S. Schedul-\r\n                 arXiv:1608.03983, 2016.                                           ing parallelizable tasks: Putting it all on the shelf. In\r\n               Mathuriya, A., Bard, D., Mendygral, P., Meadows, L., Arne-          ACMSIGMETRICSPerformanceEvaluationReview,vol-\r\n                                                \u00a8  \u00a8                               ume20,pp.225\u2013236.ACM,1992.\r\n                 mann, J., Shao, L., He, S., Karna, T., Moise, D., Penny-\r\n                 cook, S. J., et al. Cosmo\ufb02ow: using deep learning to learn     Urgaonkar, B., Shenoy, P., and Roscoe, T. Resource over-\r\n                 the universe at scale. In SC18: International Conference          booking and application pro\ufb01ling in shared hosting plat-\r\n                 for High Performance Computing, Networking, Storage               forms. ACM SIGOPSOperating Systems Review, 36(SI):\r\n                 andAnalysis, pp. 819\u2013829. IEEE, 2018.                             239\u2013254, 2002.\r\n               Narayanan, D., Santhanam, K., Phanishayee, A., and Za-           Xiao, W., Bhardwaj, R., Ramjee, R., Sivathanu, M., Kwatra,\r\n                 haria, M. Accelerating deep learning workloads through            N., Han, Z., Patel, P., Peng, X., Zhao, H., Zhang, Q.,\r\n                 ef\ufb01cient multi-model execution. In NIPS Workshop on               et al. Gandiva: Introspective cluster scheduling for deep\r\n                 Systems for Machine Learning (December 2018), 2018.               learning. In 13th {USENIX} Symposium on Operating\r\n               Ousterhout, K., Wendell, P., Zaharia, M., and Stoica, I. Spar-      Systems Design and Implementation ({OSDI} 18), pp.\r\n                 row: distributed, low latency scheduling. In Proceedings          595\u2013610, 2018.\r\n                 of the Twenty-Fourth ACM Symposium on Operating Sys-           Xu, L., Butt, A. R., Lim, S.-H., and Kannan, R.            A\r\n                 tems Principles, pp. 69\u201384. ACM, 2013.                            heterogeneity-aware task scheduler for spark. In 2018\r\n               Patton, R. M., Johnston, J. T., Young, S. R., Schuman, C. D.,       IEEE International Conference on Cluster Computing\r\n                 March, D. D., Potok, T. E., Rose, D. C., Lim, S.-H.,              (CLUSTER),pp.245\u2013256.IEEE,2018.\r\n                 Karnowski, T. P., Ziatdinov, M. A., et al. 167-p\ufb02ops           Zagoruyko, S. and Komodakis, N. Wide residual networks.\r\n                 deep learning for electron microscopy: from learning              arXiv preprint arXiv:1605.07146, 2016.\r\n                 physics to atomic manipulation. In Proceedings of the\r\n                 International Conference for High Performance Comput-          Zaharia, M., Konwinski, A., Joseph, A. D., Katz, R. H.,\r\n                 ing, Networking, Storage, and Analysis, pp. 50. IEEE              and Stoica, I. Improving mapreduce performance in het-\r\n                 Press, 2018.                                                      erogeneous environments. In Osdi, volume 8, pp. 7,\r\n               Pittman, R., Shen, X., Patton, R. M., and Lim, S.-H. Ex-            2008.\r\n                 ploring Flexible Communications for Streamlining DNN           Zoph, B. and Le, Q. V. Neural architecture search with\r\n                 Ensemble Training Pipelines. Proceedings of the Inter-            reinforcementlearning. arXiv preprint arXiv:1611.01578,\r\n                 national Conference for High Performance Computing,               2016.\r\n                 Networking, Storage, and Analysis (SC\u201918), 2018.\r\n                                                                                Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V. Learning\r\n               Ratnaparkhi, A. A. and Pilli, E.      Networks.     2016 In-        transferable architectures for scalable image recognition.\r\n                 ternational Conference on Emerging Trends in Com-                 In Proceedings of the IEEE conference on computer vi-\r\n                 munication Technologies (ETCT), pp. 1\u20136, 2012. doi:               sion and pattern recognition, pp. 8697\u20138710, 2018.\r\n                 10.1109/ETCT.2016.7882969.\r\n                                                                      FLEET\r\n                  180                                                       Algorithm 2 createFlotilla\r\n                  160\r\n                  140                                                       Input: cands,R,M // The indices of DNNs that are not con-\r\n                  120                                                           verged, the training rates of the DNNs, and the number of\r\n                  100\r\n                   80                                                           GPUsavailable\r\n                   60                                                       Output: F ,m //thek-th\ufb02otilla and the GPU count vector\r\n                   40                                                                 k   k\r\n                   20                                                        1: D    , r   =fastestDNN(cands,R)// Find the DNN with\r\n                 Training Rate (images/sec)                                      fast  fast\r\n                    0                                                           the largest training rate with a single GPU\r\n                     0         20        40        60        80       100\r\n                                      DNNs used in experiments               2: F ,M ,m =[D         ], 1, [1]\r\n                                                                                 k    k   k     fast\r\n                                                                             3: while |F | < |cands| do\r\n                Figure 7: Training rate of each DNN on a single GPU.                   k\r\n                                                                             4:   D ,r ,M =\ufb01ndNext(r              , R,cands,F ,M \u2212\r\n                                                                                    best best   best           fast           k\r\n                                                                                  M ) // Find the next DNN, its training rate and required\r\n                                                                                    k\r\n                                                                                  GPUcount\r\n              A COMPLEXITYANALYSISOFTHE                                      5:   if Dbest == \u22121 then\r\n                   GREEDYALLOCATIONALGORITHM                                 6:     break\r\n                                                                             7:   endif\r\n              Algorithm 2 shows the \ufb02otilla creation algorithm. Flotilla     8:   Fk.append(Dbest)\r\n                                                                             9:   m .append(M     )\r\n              creation \ufb01rst searches for the reference training rate (line 1,       k          best\r\n                                                                            10:   M +=M\r\n              timecomplexityO(N)),theniteratively\ufb01ndsthebestcandi-                  k       best\r\n                                                                            11: end while\r\n              date DNNtoaddinthe\ufb02otilla (lines 3-11, time complexity        12: while M < M do\r\n                                                                                       k\r\n                                                                            13:   D     =slowestDNN(F ,m ,R)//intermsofspeed\r\n              O(N \u00d7N \u00d7M), and \ufb01nally assigns all the remaining                      slow                   k   k\r\n                   k                                                              onthe currently assigned GPUs\r\n              GPUs available to the DNNs in the \ufb02otilla (lines 12- 15,      14:   m [slow]+ = 1\r\n              time complexity O(N \u00d7M)). Sothetimecomplexity of                      k\r\n                                    k                                       15:   M +=1\r\n              \ufb02otilla creation is O(N \u00d7 N \u00d7 M).                                     k\r\n                                     k                                      16: end while\r\n              GPUassignment\ufb01rst prunes the factorial solution space by\r\n              identifying and assigning GPUs to the DNNs whose training\r\n              rate meets certain requirements in O(N ) time complexity.                          Training Rate Epochs\r\n                                                     k\r\n              It then searches for the optimal GPU assignment strategy             200                                       25\r\n              for the remaining DNNs. The algorithm is shown in Al-                                                          20\r\n                                                                                   150\r\n              gorithm 3. This algorithm assumes the number of GPUs\r\n                                                                                                                             15\r\n              per node is the same among nodes (GPUsPerNode), which                100\r\n              holds in major supercomputers. It assigns GPUs to DNNs                                                         10\r\n                                                                                                                                     Epochs\r\n              in the following order:                                               50\r\n                                                                                                                             5\r\n                                                                                     0                                       0\r\n                1. the DNNs whose required number of GPUs is a multi-           Training Rate (#GPU=1)\r\n                                                                                       250       500       750       1000 \r\n                   ple of the number of GPUs per node; (lines 5-11)\r\n                2. the pairs of DNNs whose sum of the required number                                Model Size\r\n                   of GPUsisamultipleofthenumberofGPUspernode;              Figure8: CorrelationsbetweenmodelsizeofaDNNandthe\r\n                   (lines 12-24)                                            training rate and the number of epochs until convergence.\r\n                3. the remaining DNNs by searching for an optimal as-       201, ResNet-50, ResNet-101 and ResNet-152. The \ufb01rst\r\n                   signment of GPUs. (lines 25-39)                          three are variations of DenseNet (Huang et al., 2017). The\r\n              Let N\u2032 be the number of remaining DNNs. The solution          three variations share the same structure, but differ in the\r\n                    k                                                       number of DNN layers, indicated by their suf\ufb01xes. The\r\n              space is N\u2032!. Most of the time, N\u2032 is a small number less\r\n                         k                      k                           latter three are variations of ResNet (He et al., 2016).\r\n              than \ufb01ve. However, enumerating all the possible solutions\r\n              is still in factorial time complexity. We set the maximum     The 100 DNNs used in our experiments have a range of\r\n              numberofsolutions to explore as 1024, reducing the time       model sizes, from 232 MB to 1.19GB. Different DNNs\r\n              complexity to O(1). The time complexity of GPU assign-        have different GPU memory requirements and thus require\r\n              mentis thus O(N ).                                            different batch sizes to maximize GPU utilization. For each,\r\n                               k\r\n                                                                            we use the maximum batch size that can \ufb01t into GPU\u2019s\r\n              B EXPERIMENTDETAILS                                           memory. Figure 7 shows the distribution of their training\r\n                                                                            rates on a single GPU which vary from 21 to 176 images/sec.\r\n              B.1   Characteristics of Experimental DNNs                    Figure 8 outlines the relations between the training rates and\r\n              The DNNs used in this experiment are derived from six         modelsizes of the DNNs, as well as the relations between\r\n              popular DNNs, DenseNet-121, DenseNet-169, DenseNet-           convergence rates (i.e., the number of epochs needed for the\r\n                                                                           FLEET\r\n                                                                                 DNNstoconverge)andtheir model sizes. As model size in-\r\n               Algorithm 3 getGPUAssignment                                      creases, the training rate tends to drop as more computations\r\n               Input: F ,m //Thek-th\ufb02otilla and the GPU count vector             are involved in the DNN, but there are no clear correlations\r\n                        k    k                                                   with the convergence rate. It is the reason that the resource\r\n               Output: A\r\n                          k\r\n                1: j = 1 // The current available GPU with the smallest index.   allocation algorithm in FLEET primarily considers training\r\n                2: A = 0N ,M // The GPU assignment matrix of dimension\r\n                     k      k                                                    rate explicitly, and relies on the periodical (re)scheduling to\r\n                   Nk\u00d7M                                                          indirectly adapt to the variations of DNNs in the converging\r\n                3: remaining = {1,\u00b7\u00b7\u00b7 ,N} // The indices of DNNs to allo-        rates.\r\n                   cate GPUs\r\n                4: assigned = {} // The indices of DNNs that have assigned\r\n                   GPUs                                                          B.2   SystemSettings\r\n                5: for all i \u2208 remaining do\r\n                6:   if m(k)%GPUsPerNode==0then                                  All experiments are conducted on SummitDev (Sum, 2019),\r\n                         i\r\n                7:      assigned.add(i)                                          a development machine for Summit supercomputer at Oak\r\n                8:      j = assignGPUs(A ,i,j,m(k))\r\n                                         k        i                              Ridge National Lab. Each node is equipped with two IBM\r\n                9:   endif                                                       POWER8 CPUs and 256GB DRAM, and four NVIDIA\r\n               10: end for                                                       Tesla P100 GPUs. Each POWER8 CPU has 10 cores with 8\r\n               11: remaining -= assigned                                         HWthreadseach. ThedefaultSMTlevelissettooneunless\r\n               12: memo,assigned = {}, {}                                        noted otherwise. The number of cores allocated per GPU\r\n               13: for all i \u2208 remaining do\r\n               14:    if \u2212m(k)%GPUsPerNodenotinmemothen                          is \ufb01ve in all the experiments. NVLink 1.0 is the connection\r\n                            i\r\n                                 (k)                                             among all GPUs and between CPUs and GPUs within a\r\n               15:      memo[m %GPUsPerNode]=i\r\n               16:    else       i                                               node. EDR In\ufb01niBand connects different nodes in a full fat-\r\n                                              (k)\r\n               17:      for ii \u2208 {i,memo[\u2212m      %GPUsPerNode]}do\r\n               18:        assigned.add(ii)    i                                  tree. The \ufb01le system is an IBM Spectrum Scale \ufb01le system,\r\n               19:        j = assignGPUs(A ,ii,j,m(k))                           which provides 2.5 TB/s for sequential I/O and 2.2 TB/s for\r\n                                            k        ii                          randomI/O. Our experiments show that thanks to the large\r\n               20:      endfor\r\n               21:      del memo[m(k)%GPUsPerNode]                               I/O throughput of the \ufb01le system, I/O is not the bottleneck\r\n               22:    endif          i                                           of DNNtraining. The used CUDA version is 9.2.\r\n               23: end for                                                       FLEET is built on Tensor\ufb02ow 1.12 (as the core train-\r\n               24: remaining -= assigned                                         ing engine), Horovod v0.15.2 (Sergeev & Del Balso,\r\n               25: if |remaining| > 0 then\r\n                      \u02dc\r\n               26:    m(k),bestScore,bestA,jcopy,Acopy                    =      2018) (as the basis for distributed DNN training),\r\n                     [], \u221e,j,clone(A )\r\n                                      k                                          and mpi4py v3.0.0 (for the pipeline construction).\r\n               27:    for all i \u2208 remaining do                                   We set       inter op parallelism threads and\r\n                        \u02dc                 (k)\r\n               28:      m .append((i,m       ))\r\n                          (k)             i                                      intra op parallelism threads to # logical cores\r\n               29:    endfor                                                     for parallel TensorFlow operaitons on CPU. The used\r\n                                                          \u02dc\r\n               30:    for permutation in allPermutations(m(k)) do                CUDAversionis9.2.\r\n               31:      j,A =j       , clone(A    )\r\n                            k    copy         copy\r\n               32:      for i,m(k) in permutation do\r\n                               i\r\n               33:        j = assignGPUs(A ,i,j,m(k))                            B.3   Pro\ufb01ling Details\r\n                                            k       i\r\n               34:      endfor                                                   Tominimizetheoverheadofpro\ufb01ling, we only pro\ufb01le the\r\n               35:      score = calculateScore(A ) // Score is calculated based\r\n                                                k\r\n                        onthe loss function in Eq. 7                             training rates of each DNN in the ensemble with the num-\r\n               36:      if score < bestScore then                                ber of GPUs varying from one to M (M < M). For\r\n               37:        bestScore,bestA = score,A                                                                       t   t\r\n                                                       k                         pro\ufb01ling on m (m = 1,\u00b7\u00b7\u00b7 ,M ) GPUs, we train a DNN\r\n               38:      endif                                                                                     t\r\n               39:    endfor                                                     for a maximum of 48 batches and use the training time\r\n               40:    A =bestA                                                   of the last 20 batches to calculate the exact training rate:\r\n                       k\r\n               41: end if                                                        r (m),i = 1,\u00b7\u00b7\u00b7 ,N. Based on the pro\ufb01led training rates,\r\n                                                                                  i\r\n                                                                                 weestimate the training rates of each DNN when m > Mt.\r\n                                                                                 Speci\ufb01cally, the pro\ufb01ling has three steps:\r\n               Algorithm 4 assignGPUs                                              1. Collect the training rates of each DNN on a single\r\n               Input: A ,i,j,m(k)                                                     GPU,R(1)={r (1)},i = 1,\u00b7\u00b7\u00b7 ,N.\r\n                        k        i                                                                      i\r\n               Output: j // The index of the next available GPU to assign\r\n                           (k)\r\n                1: while m    >0 do                                                2. Estimate the number of GPUs required to make the\r\n                           i\r\n                       k                 (k)\r\n                2:   ai,j = 1; j+ = 1; m    \u2212=1                                       DNNthathasthesmallesttrainingrateonasingleGPU\r\n                3: end while             i\r\n                                                                                      achieve the largest single-GPU training rate, M       =\r\n                                                                                      \u0006           \u0007                                       a\r\n                                                                                        max(R(1)) .\r\n                                                                                        min(R(1))\r\n                                                               FLEET\r\n                1000\r\n                 800\r\n                 600\r\n                Training Rate400\r\n                 200\r\n                   01    2    3     4    5    6     7    8\r\n                                 Number of GPUs\r\n             Figure 9: The pro\ufb01led training rates (images/sec) of 100\r\n             DNNsinanensemblewithImagenet.\r\n              3. Collect the training rates of each DNN with the number\r\n                 of GPUs varying from two to M = max(M ,M ),\r\n                                               t          a   b\r\n                 where M =2\u00d7GPUsPerNode.\r\n                         b\r\n             Note that steps 1 and 3 can be done in parallel because the\r\n             trainings of different DNNs with different number of GPUs\r\n             are independent. The training rate of the i-th DNN with\r\n             the number of GPUs higher than M is estimated via the\r\n                                             t\r\n             following equation:\r\n                         r (M )   \u0010 r (M )     M \u22121\u0011m\u2212Mb\r\n              r (m) = m\u00d7 i    b \u00d7     i  b   \u00d7 b            . (5)\r\n               i           M       r (M \u22121)      M\r\n                             b      i  b           b\r\n             Theformula for Mb and Equation 5 are the result of perfor-\r\n             mance modeling on our observations on the DNN perfor-\r\n             mance trend as illustrated in Figure 9. It achieves a good\r\n             tradeoff between the pro\ufb01ling cost and the performance\r\n             prediction accuracy.\r\n             The pro\ufb01ling process also measures the throughput of a\r\n             range of preprocessors (#cores=1, 2, 4, 8, 16, 32) in the\r\n             pipeline. This step is quick since preprocessing does not\r\n             exhibit large variations. Based on the pro\ufb01led information,\r\n             FLEETcalculates the minimum number of preprocessors\r\n             that can meet the demands of an arbitrary M DNNs (with\r\n             one running on one GPU), and uses it to set the number of\r\n             preprocessors.\r\n", "award": [], "sourceid": 98, "authors": [{"given_name": "Hui", "family_name": "Guan", "institution": "North Carolina State University"}, {"given_name": "Laxmikant Kishor", "family_name": "Mokadam", "institution": "North Carolina State University"}, {"given_name": "Xipeng", "family_name": "Shen", "institution": "North Carolina State University"}, {"given_name": " Seung-Hwan", "family_name": "Lim", "institution": "Oak Ridge National Laboratory"}, {"given_name": "Robert", "family_name": "Patton", "institution": "Rensselaer Polytechnic Institute, Oak Ridge National Laboratory"}]}