{"title": "Ordering Chaos: Memory-Aware Scheduling of Irregularly Wired Neural Networks for Edge Devices", "book": "Proceedings of Machine Learning and Systems", "page_first": 44, "page_last": 57, "abstract": "Recent advances demonstrate that irregularly wired neural networks from Neural Architecture Search (NAS) and Random Wiring can not only automate the design of deep neural networks but also emit models that outperform previous manual designs. These designs are especially effective while designing neural architectures under hard resource constraints (memory, MACs, . . . ) which highlights the importance of this class of designing neural networks. However, such a move creates complication in the previously streamlined pattern of execution. In fact one of the main challenges is that the order of such nodes in the neural network significantly effects the memory footprint of the intermediate activations. Current compilers do not schedule with regard to activation memory footprint that it significantly increases its peak compared to the optimum, rendering it not applicable for edge devices. To address this standing issue, we present a memory-aware compiler, dubbed SERENITY, that utilizes dynamic programming to find a sequence that finds a schedule with optimal memory footprint. Our solution also comprises of graph rewriting technique that allows further reduction beyond the optimum. As such, SERENITY achieves optimal peak memory, and the graph rewriting technique further improves this resulting in 1.68\u00d7 improvement with dynamic programming-based scheduler and 1.86\u00d7 with graph rewriting, against TensorFlow Lite with less than one minute overhead.", "full_text": " ORDERINGCHAOS:MEMORY-AWARESCHEDULINGOF\r\n IRREGULARLYWIREDNEURALNETWORKSFOREDGEDEVICES\r\n ByungHoonAhn1\u2020 JinwonLee2 JamieMenjayLin2 Hsin-PaiCheng3\u2020 JileiHou2 HadiEsmaeilzadeh1\r\n ABSTRACT\r\n RecentadvanceonautomatingmachinelearningthroughNeuralArchitectureSearchandRandomNetworkGener-\r\n ators, has yielded networks that deliver higher accuracy given the same hardware resource constrains, e.g., memory\r\n capacity, bandwidth,numberoffunctionalunits. Manyoftheseemergentnetworks;however,compriseofirregular\r\n wirings(connections)thatcomplicatetheirexecutionbydeviatingfromtheconventionalregularpatternsoflayer,\r\n nodeconnectivity,andcomputation. Theirregularityleadstoanewproblemspacewherethescheduleandorder\r\n ofnodessigni\ufb01cantlyaffecttheactivationmemoryfootprintduringinference. Concurrently,thereisanincreasing\r\n generaldemandtodeployneuralmodelsontoresource-constrainededgedevicesduetoef\ufb01ciency,connectivity,and\r\n privacyconcerns. Toenablesuchatransitionfromcloudtoedgefortheirregularlywiredneuralnetworks,wesetout\r\n to devise a compiler optimization that caps and minimizesthefootprinttothelimitationsoftheedgedevice. This\r\n optimizationisasearchforthescheduleofthenodesinanintractablylargespaceofpossiblesolutions. Weoffer\r\n andleveragetheinsightthatpartialschedulesleadstorepeatedsubpathsforsearchandusethegraphpropertiesto\r\n generateasignaturefortheserepetition. ThesesignaturesenabletheuseofDynamicProgrammingasabasisforthe\r\n optimizationalgorithm. However,duetothesheernumberofneuronsandconnections,thesearchspacemayremain\r\n prohibitivelylarge. Assuch,wedeviseanAdaptiveSoftBudgetingtechniquethatduringdynamicprogrammingper-\r\n formsalight-weightmeta-searchto\ufb01ndtheappropriatememorybudgetforpruningsuboptimalpaths. Nonetheless,\r\n schedulesfromanyschedulingalgorithm,includingours,isstillboundtothetopologyoftheneuralgraphunder\r\n compilation. To alleviate this intrinsic restriction, we develop an Identity Graph Rewriting scheme that leads to\r\n evenlowermemoryfootprintwithoutchangingthemathematicalintegrityoftheneuralnetwork. Weevaluateour\r\n proposedalgorithmsandschemesusingrepresentativeirregularlywiredneuralnetworks. ComparedtoTensorFlow\r\n Lite, a widely used framework for edge devices, the proposed framework provides 1.86\u00d7reduction in memory\r\n footprint and 1.76\u00d7reductioninoff-chiptraf\ufb01cwithanaverageoflessthanoneminuteextracompilationtime.\r\n 1 INTRODUCTION ingedgedevices. Therefore,liftingtheregularitycondition\r\n Growing body of work focuses on Automating Machine providessigni\ufb01cantfreedomforNASandexpandsthesearch\r\n Learning (AutoML) using Neural Architecture Search space(Cortesetal.,2017;Zhangetal.,2019;Xieetal.,2019).\r\n (NAS)(Zoph&Le,2017;Cortesetal., 2017; Zoph et al., Thegeneralobjectiveistoenabledeploymentofneuralin-\r\n 2018; Liu et al., 2019a; Cai et al., 2019; Real et al., 2019; telligence even on stringently constrained devices by trading\r\n Zhangetal.,2019)andnoweven,RandomNetworkGenera- off regular wiring of neurons for higher resource ef\ufb01ciency.\r\n tors(Xieetal.,2019;Wortsmanetal.,2019)whichemitmod- Importantly,pushingneuralexecutiontoedgeisonewayto\r\n els with irregular wirings, and shows that such irregularly addressthegrowingconcernsaboutprivacy(Mireshghallah\r\n wiredneuralnetworkscansigni\ufb01cantlyenhanceclassi\ufb01ca- et al., 2020) and enable their effective use where connectivity\r\n tion performance. Thesenetworksthatdeviatefromregular to cloud is restricted (Wu et al., 2019). However, the new\r\n topology can even adapt to some of the constraints of the challengearisesregardingorchestratingexecutionofthese\r\n hardware(e.g.,memorycapacity,bandwidth,numberoffunc- irregularly wired neural networks on the edge devices as\r\n tionalunits),renderingthemselvesespeciallyusefulintarget- working memory footprint during execution frequently\r\n \u2020 1 surpass the strict cap on the memory capacity of these\r\n WorkdoneasinternatQualcommAIResearch. Universityof devices. Thelackofmulti-levelmemoryhierarchyinthese\r\n California, SanDiego2QualcommAIResearch3DukeUniversity.\r\n Correspondenceto: ByungHoonAhn. microdevicesexacerbatestheproblem,becausethenetwork\r\n cannotevenbeexecutedifthefootprintexceedsthecapacity.\r\n Proceedingsofthe3rd MLSysConference,Austin,TX,USA,2020. To that end, despite the signi\ufb01cant potential of irregularly\r\n Copyright2020bytheauthor(s). wiredneuralnetworks,theircomplicatedexecutionpattern,\r\n OrderingChaos: Memory-AwareSchedulingofIrregularlyWiredNeuralNetworksforEdgeDevices\r\n in contrast to previously streamlined execution of models\r\n with regular topology, renders conventional frameworks\r\n futile in taking these networks to edge due to their large peak\r\n memoryfootprint. Whilepeakmemoryfootprintislargely\r\n dependentonschedulingofneurons,currentdeeplearning\r\n compilers (Chen et al., 2018; Vasilache et al., 2018) and\r\n frameworks(Abadietal.,2016;Paszkeetal.,2019;Jiaetal.,\r\n 2014)relyonbasictopologicalorderingalgorithmsthatare\r\n oblivioustopeakmemoryfootprintandinsteadfocusonan\r\n orthogonalproblemoftilingandkernelleveloptimization.\r\n Thispaperisaninitialsteptowardsembeddingpeakmemory\r\n footprint as \ufb01rst-grade constraint in deep learning schedulers\r\n to unleash the potential of the emergent irregularly wired\r\n neural networks. As such, this paper makes the following (a) RandWire (b) SwiftNet\r\n contributions:\r\n (1) Memory-aware scheduling for irregularly wired Figure1.ArchitectureofnetworkmodelsfromNASandRandom\r\n NetworkGenerators. Topologyofsuchnetworksincludedistinctive\r\n neuralnetworks. Schedulingforthesenetworksisatopo- irregular wirings betweenthenodes.\r\n logical orderingproblem,whichenumeratesanintractably\r\n large space of possible schedules. We offer and leverage the 2 CHALLENGESANDOURAPPROACH\r\n insight that partial schedules leads to repeated subpaths for\r\n search and use the graph properties to generate a signature 2.1 IrregularlyWiredNeuralNetworks\r\n for these repetition while embeddinganotionoftherunning Recent excitement in Automated Machine Learning\r\n memoryusage. ThesesignaturesenabletheuseofDynamic (AutoML)(Feureretal.,2015;Dean,2017;Heetal.,2018;\r\n Programmingasabasisfortheoptimizationalgorithm. Elthakebetal.,2018;Wangetal.,2019;Laredoetal.,2019)\r\n (2) Adaptive soft budgeting for tractable compilation aimstoachievehumanoutoftheloopindevelopingmachine\r\n time. Evenwiththedynamicprogrammingasthebase,due learningsystems. ThisincludesNeuralArchitectureSearch\r\n to the sheer numberofneuronsandconnections,thesearch (NAS)(Zoph&Le,2017;Zophetal.,2018;Liuetal.,2019a;\r\n spacemayremaintoolarge(exponentiallylarge)inpractice. Cai et al., 2019; Real et al., 2019; Zhang et al., 2019) and\r\n Assuch, we devise an Adaptive Soft Budgeting technique RandomNetworkGenerators(Xieetal.,2019;Wortsman\r\n that uses a lightweight meta-search mechanism to \ufb01nd the et al., 2019) that focus on automation of designing neural\r\n appropriate memory budget for pruning the suboptimal architectures. Figure 1 demonstrates that networks of this\r\n paths. This technique aimsto\ufb01ndanin\ufb02ectionpointbeyond regimearecharacterizedbytheirdistinctiveirregulargraph\r\n which tighter budgets may lead to no solution and looser topology with much more irregular wirings (data\ufb02ow)\r\n budget prolongs the scheduling substantially, putting the compared to conventional networks with regular graph\r\n optimizationinapositionofquestionableutility. topology. Thispaperreferstothesenetworksasirregularly\r\n (3) Identity graph rewriting for enabling higher poten- wiredneuralnetworks.\r\n tial in memory reduction. Any scheduling algorithm, in-\r\n cludingours,isstillboundtothetopologyoftheneuralgraph\r\n under compilation. To relax this intrinsic restriction, we 85\r\n \"NPFCB/FU\u000e\"\r\n deviseanIdentityGraphRewritingschemethatexchanges 3BOE8JSF /\"4/FU\u000e\"\r\n subgraphs leading to a lower memory footprint without /\"4/FU\u000e# \"NPFCB/FU\u000e# %1/\u000e\u0012\u0014\u0012 4&/FU\r\n 80 3BOE8JSF *ODFQUJPO\u00013FT/FU\u00017\u0013 1PMZ/FU 3F/F9U\u000e\u0012\u0011\u0012\r\n altering the mathematicalintegrityoftheneuralnetwork. Inception 7\u0015\r\n 9DFQUJPO 3FT/FU\u000e\u0012\u0016\u0013\r\n Results show that our adaptive scheduling algorithm Inception 7\u0014\r\n improvespeakmemoryfootprintforirregularlywiredneural 75 Inception 7\u0013 top left means is better\r\n \"NPFCB/FU\u000e\" irregularly wired neural networks\r\n networksby1.68\u00d7comparedtoTensorFlowLite,thedefacto show better performance for\r\n frameworkforedgedevices. Ourgraphrewritingtechnique Shu\ufb04eNet same amount of compute than\r\n op-1 ImageNet Accuracy (%)70MobileNet regular topology neural networks \r\n providesanopportunitytolowerthepeakmemoryfootprint T Inception 7\u0012\r\n by an additional 10.7%. Furthermore, our framework can JSSFHVMBSMZ\u0001XJSFE\u0001OFVSBM\u0001OFUXPSLT\r\n SFHVMBS\u0001UPQPMPHZ\u0001OFVSBM\u0001OFUXPSLT\r\n evenbringabout1.76\u00d7reductioninoff-chiptraf\ufb01cforde- 65\r\n viceswithmulti-levelmemoryhierarchy,andeveneliminate 0 10 20 30 40\r\n Multiply-and-accumulate (#JMMJPOT )\r\n thetraf\ufb01cinsomecasesbycon\ufb01ningthememoryfootprint Figure2.ImageNet accuracy vs number of multiply-and-\r\n below the on-chip memory capacity. These gains come at accumulate,whereirregularlywiredneuralnetworksshowhigher\r\n averageoflessthanoneminuteextracompilationtime. performance for same compute than regular topology neural net-\r\n works. Plotfornumberofparametersalsodisplaysasimilartrend.\r\n OrderingChaos: Memory-AwareSchedulingofIrregularlyWiredNeuralNetworksforEdgeDevices\r\n 100 250 KB\r\n constraint \u2217\r\n 80 s =argmin\u00b5 (s,G), for s\u2208S (1)\r\n 0.04% of schedules peak\r\n DPODBU 60 are optimal s\r\n 40 4.1% of schedules\r\n DPOW of Schedules (%)20 satisfy the constraint Themoststraightforward way to schedule is a brute force\r\n Cumulative Distribution0 200 250 300 350 400 approach which just enumerates S and picks one with the\r\n Peak Memory Footprint (KB) minimum peak memory footprint. While this extreme\r\n (a) SwiftNetCellA. (b) CDFofpeakmemoryfor methodmay\ufb01ndanoptimalsolution,itistoocostlyinterms\r\n different possible schedules. of time due to its immense complexity: \u0398(|V |!) where |V |\r\n Figure3.CDF of the peak memory footprint for the different denotesnumberofnodesinthegraph. Onewaytoimprove\r\n possibleschedulesofagivenirregularlywiredneuralnetwork. is to narrow down the search space to just focus on only\r\n the topological orderings S \u2282S. However, this will still\r\n T\r\n From the performance perspective, these networks have suffer from a complexity with an upper bound of O(|V |!)\r\n shown to outperform manually designed architectures (takes daystoscheduleDAGwithmerely30nodes). Infact,\r\n in terms of accuracy while using less resources. In fact, previousworks(Bruno&Sethi,1976;Bernsteinetal.,1989)\r\n majority of winning neural architectures in competitions alreadyproveoptimalschedulingforDAGsisNP-complete.\r\n withprimarygoalofreducingresources(Gauenetal.,2017) Onanotherextremeareheuristicsfortopologicalordering\r\n rely on NAS, suggesting its effectiveness in that respect. suchasKahn\u2019salgorithm(Kahn,1962),withcomplexityof\r\n Figure 2 plots the accuracy of different models given their O(|V|+|E|)whereV andEarenumberofnodesandedges.\r\n computation. The \ufb01gure clearly shows that the Pareto However, as demonstrated in Figure 3, such method may\r\n frontier of irregularly wired neural networks from NAS yield suboptimalscheduleofnodeswhichwillnotrunonthe\r\n and RandomNetworkGeneratorsarebetterthanthehand target hardware. Tothisend,weexploredynamicprogram-\r\n designedmodelswithregulartopology. Thisindicatesthat mingcombinedwithadaptivesoftbudgetingforscheduling\r\n theef\ufb01ciencyintermsofaccuracygiven\ufb01xedresourcesare to achieveanoptimalsolutionwhilekeepingthegraphcon-\r\n \u2217\r\n better with the irregularly wired neural networks. stant s , without adding too much overheadintermsoftime.\r\n WeexplainouralgorithmsindepthinSection3.1and 3.2.\r\n 2.2 Challenges Graph rewriting. Any scheduling algorithm including\r\n Manyexistingcompilers(Chenetal.,2018;Vasilacheetal., ours is intrinsically bounded by the graph topology.\r\n 2018)andframeworks(Paszkeetal.,2019;Abadietal.,2016; Therefore, we explore to transform the search space\r\n Jia et al., 2014) rely on basic topological ordering algorithms through graph rewriting (Plump, 1999). Graph rewriting\r\n toschedulethegraph. Whilethecurrentapproachmaybesuf- is generally concerned with substituting a certain pattern\r\n \ufb01cienttorunconventionalnetworksonserver-classmachines, in the graph with a different pattern to achieve a certain\r\n such scheme may be un\ufb01t for running irregularly wired objective. For a computational data\ufb02ow graph, leveraging\r\n neuralnetworksonresource-constrainededgedevices. This distributive, associative, and commutative properties within\r\n is because, unlike running networks with regular topology, thecomputationofthegraph,graphrewritingcanmaintain\r\n runningirregularnetworksresultsinvariedrangeofmemory the semantics while bringing signi\ufb01cant improvements\r\n footprint depending on the schedule. For instance, given regardingsomeobjective. Forexample,ingeneralprograms,\r\n the constraints of a representative edge device (SparkFun Plogx canberepresentedasP logx +P logx\r\n i i oddi i eveni i\r\n Edge: 250KBweight/activationmemoryand60MMACs), orlogQ x ,whilex+xcanbetranslatedtox\u00d72orx<<1.\r\n Figure3(b)showsthat4.1%oftheschedulesbarelymeetsthe i i\r\n Likewise,webringthisinsighttoneuralnetworksto\ufb01nda\r\n hardmemoryconstraint,whileonly0.04%wouldachievethe setofpossibletransformationsX thatcanrewritetheoriginal\r\n optimalpeakmemory. Inreality,suchlimitationwillprevent graphG toanewgraphG\u2032thatwouldalsochangeoursearch\r\n further exploration regarding the diversity and innovation spaceS toonewithalowerpeakmemoryfootprint:\r\n of network design, and in order to allow edge computing\r\n regimetotakefulladvantageoftheirregularlywiredneural X\u2217=argmin(\u00b5peak(s\u2217,X(G))) (2)\r\n networks,thislimitationshouldbealleviatedifnotremoved. X\r\n 2.3 DesignObjectives Weidentify a set of candidate patterns for transformation\r\n \u03c7:g\u2192g\u2032(g\u2208Gandg\u2032\u2208G\u2032),whichconstitutesX. While\r\n Scheduling algorithm. To address this issue, our work transformingthegraph,ourmethodkeepsthemathematical\r\n aims to \ufb01nd a schedule of nodes s\u2217 from the search space integrity of the graph intact, thus not an approximation\r\n S that would minimize peak memory footprint \u00b5peak. S method. We embed this systematic way to improve peak\r\n enumeratesallpossibleorderingsofthenodesv\u2208V where memory footprint and the search space as identity graph\r\n VisthesetofallnodeswithinagraphG. rewriting, and weaddressthistechniqueinSection3.3.\r\n OrderingChaos: Memory-AwareSchedulingofIrregularlyWiredNeuralNetworksforEdgeDevices\r\n \ufb02ag = {\u2018no solution\u2019, \u2018timeout\u2019, \u2018solution\u2019}\r\n 1\r\n 2 7\r\n Identity Adaptive Soft Dynamic 5 10 8\r\n Graph Rewriter Budgeting \u03c4 Programming- s* 3 11 13 16\r\n G , T based Scheduler 4 12 14 9\r\n Rewrite graph to alleviate Adaptively manage Find memory-optimal 15\r\n activation memory footprint Rewritten soft budget to speed schedule given an 17\r\n Graph G of the graph Graph G up scheduling input graph Schedule s* \r\n Figure4. Overallwork\ufb02owofSERENITY,memory-awareschedulingofirregularlywiredneuralnetwork.\r\n 3 SERENITY: MEMORY-AWARE ingoptimalsolutionbysweepingtheentiresearchspace.\r\n SCHEDULINGOFIRREGULARLY\r\n WIREDNEURALNETWORKS Identifying signature to enable dynamic programming.\r\n As discussed in Section 2, the objective is reducing the The \ufb01rst step to applying dynamic programming to a\r\n peak memory footprint while executing irregularly wired newproblem is characterizing the structure of an optimal\r\n \u2217 \u2217 \u2217\r\n solution: s =s (s isanoptimalsolutionfornnumberof\r\n neural networks. We propose SERENITY, memory-aware n n\r\n scheduling that targets devices with restricted resources nodes). Then,itrequiresidentifyingarecursiverelationship\r\n between the optimal solution of a subproblem s\u2217 and the\r\n (e.g., edge devices). Figure 4 summarizes the overall i\r\n original problem s\u2217 , and we do this by analyzing the\r\n schedulingprocess,highlightingthemajorcontributionsof i+1\r\n our approach. Input to SERENITY is a graph of irregularly straightforward recursive topological ordering, which\r\n wiredneuralnetworkG,whichinfactactsasanintermediate whileinef\ufb01cientsweepstheentiresearchspace. Inessence,\r\n representation (IR) during the scheduling process. We topological ordering algorithm is a repeated process of\r\n augmentthisIRwiththemetadataofthenodessuchasthe identifying a set of nodes that are available for scheduling\r\n operation type, input/output edges, input/output shapes, and iterating the set for recursion. In graph theory such a\r\n and memory cost. Then the graph rewriter transforms set of nodes available for scheduling is called zero-indegree\r\n the graph G \u2192 G\u2032 to relax the memory costs of memory set z, where z is a set of nodes which all of their incoming\r\n intensivepatternswiththegoalofreducingthepeakmemory edgesandthecorrespondingpredecessornodes(indegree)\r\n footprint \u00b5 of G. SERENITY schedulesthegraphtoan havebeenscheduled. Figure5demonstratestherecursion\r\n peak tree of the different topological ordering algorithms, where\r\n optimalschedules\u2217 usingthedynamicprogramming-based theheightofthetreeisthesearchstepandeverypathfrom\r\n scheduler. However, since the scheduling may be slow the root to the leaf is a topological ordering s \u2208 S . The\r\n due to the complexity, we scale down search space by T\r\n leveraging divide-and-conquer which partitions the graph \ufb01gurehighlightstheredundantz intherecursivetopological\r\n into multiple subgraphs. Them, we augmentthescheduler orderingintherecursiontree,thenmergesthesez tomake\r\n with an adaptive soft budgeting which prunes suboptimal themunique,identifyingitasthesignatureforrepetition,and\r\n pathsbyadaptively\ufb01ndingabudgetforthresholdingthrough preventtheaforementionedre-computation. Thismakesthe\r\n aswiftmeta-searchtospeeduptheschedulingprocess. This schedulingforz intoauniquesubproblem,thatconstitutes\r\n sectionfocusesontheinnovationsofSERENITY: dynamic\r\n programming-based scheduling, divide-and-conquer, X Scheduled X Schedulable X X For memoization\r\n adaptive soft budgeting, and graph rewriting, which are A s A\r\n explainedindetailinSection3.1, 3.2,and 3.3,respectively. B C B C J\r\n D E F ch StepG D C J B E F J B C\r\n 3.1 DynamicProgramming-basedScheduling: G H I J Sear \u2026 G D C \u2026 G D C \u2026\r\n AchievingOptimalPeakMemoryFootprint z K\r\n Our goal for the scheduling algorithm is to minimize the L Redundant zero-indegree set z\r\n peakmemoryfootprint\u00b5peak(s,G). AsstatedinSection2.3, Graph G Recursive Topological Ordering\r\n recursive algorithms that covers the entire search space S A\r\n or the subspace of all topological orderings S \u2282Stakes B C J\r\n T\r\n impracticallylongtime. Thisisprimarilyduetotherepetitive ch StepG D A,B,C A,B,J E F A,C,J\r\n re-computationofsubproblemsthatupperboundsthealgo- Sear \u2026 G D C \u2026\r\n rithmbyO(|V|!). Therefore,weleveragedynamicprogram- Unique zero-indegree set z\r\n ming (Bellman,1961;1966;Held&Karp,1962)whichin-\r\n cludesamemoizationschemethathasbeenshowntobeeffec- Dynamic Programming-based Topological Ordering\r\n tive in reducing the complexity of time-intensive algorithms Figure5.Illustration of identifying redundant zero-indegree set z\r\n byreusingsolutionsfromtheirsubproblems,whilestill\ufb01nd- andmakingzunique(square)throughoutthetopologicalordering\r\n algorithmtoreducere-computation.\r\n OrderingChaos: Memory-AwareSchedulingofIrregularlyWiredNeuralNetworksforEdgeDevices\r\n A Scheduled B To Schedule/Allocate B To Deallocate Algorithm1DynamicProgramming-basedScheduling\r\n i = 8 (0) Initial State Activation Memory 1: Input: graphG \u2217\r\n z8 = G H D E F I J 2: Output: optimalschedules\r\n u8 = H s8 = \u00b5 8 3: // initialize memoization\r\n A B D C E F I J\r\n \u00b5 peak,8 from M8 4: s \u2190[],\u00b5 ,\u00b5 \u21900,z \u2190zero-indegree(s ,G)\r\n A 0 0 peak,0 0 0\r\n (1) Schedule/Allocate H D E F I J H 5: M [z ]\u2190(s ,\u00b5 ,\u00b5 )\r\n B C \u00b5peak 0 0 0 0 peak,0\r\n D E F s9 = A B D C E F I J H 6: // iterate search step\r\n G H I J \u00b5peak,9 = max(\u00b5peak,8, \u00b5peak) 7: fori=0ton\u22121do\r\n outdegree of : 1D \u2192 0 D E 8: // iterate (schedule, current memory, peak memory)\r\n K outdegree of : 1E \u2192 0 H I 9: forz ,(s ,\u00b5 ,\u00b5 ) in M do\r\n i i i peak i\r\n L D E F I J H 10: foru inz do\r\n Graph G (2) Deallocate \u00b59 i i\r\n 11: s \u2190s.append(u )//allocate\r\n Figure6.Visualization of scheduling the node u = H during the i+1 i i\r\n 8 12: z \u2190zero-indegree(s ,G)\r\n search step i = 8. Starting from s , \u00b5 , and \u00b5 the \ufb01gure i+1 Q i+1\r\n 8 8 peak,8 13: \u00b5 ,\u00b5 \u2190\u00b5+ (u.shape)\r\n showshowthealgorithmcalculatess ,\u00b5 ,and\u00b5 i+1 peak i i\r\n 9 9 peak,9 14: \u00b5peak,i+1\u2190max(\u00b5peak,i,\u00b5peak)\r\n 15: forp inu .predsdo\r\n thedynamicprogramming-basedtopologicalordering. i i\r\n 16: if p is in zero-outdegree(s ,G)then\r\n i Q i+1\r\n 17: \u00b5 \u2190\u00b5 \u2212 (p.shape)//deallocate\r\n Integratingthepeakmemoryfootprintconstraint. On i+1 i+1 i\r\n top of the dynamic programming formulation that shows 18: endif\r\n potential for optimizing the search space signi\ufb01cantly, we 19: endfor\r\n 20: // memoizeschedulewithleastpeakmemory\r\n overlay the problem speci\ufb01c constraints to achieve the 21: if \u00b5 \u2264M [z ].\u00b5 then\r\n optimal solution. In particular, we calculate the memory peak,i+1 i+1 i+1 peak,i+1\r\n 22: M [z ]\u2190(s ,\u00b5 ,\u00b5 )\r\n footprint \u00b5 anditscorrespondingpeak\u00b5 in each i+1 i+1 i+1 i+1 peak,i+1\r\n i+1 peak,i+1 23: endif\r\n search step i to select optimal path s\u2217 for memoization.\r\n i+1 24: endfor\r\n Here, we clarify the process of a search step, explaining 25: endfor\r\n thedetailsofcalculating\u00b5peak,i+1 andsavingsi+1 foreach 26: endfor\r\n search step i. In each search step, we start with number of 27: s\u2217,\u00b5\u2217 \u2190M[\u00b7] .s ,M[\u00b7] .\u00b5 // solution\r\n uniquezero-indegreesets z (signature), saved in ith entry peak n n n peak,n\r\n i\r\n ofmemoizationM . Foreachz ,weappendthescheduleup\r\n i i\r\n to the point si, sum of activations in the memory \u00b5i for the activation memoryhereisrecordedas\u00b5 .\r\n signaturez ,andthepeakmemoryfootprintofthes denoted 9\r\n i i\r\n \u00b5 . Therefore, in each search step i, we start with s , \u00b5 ,\r\n peak,i i i Finding schedule with optimal peak memory footprint.\r\n and\u00b5 fors . Then,whenweiteratez toscheduleanew\r\n peak,i i i After scheduling u , we save the new signature into the\r\n nodeu ,itsoutputactivationisappendedtos toforms , i\r\n i i i+1\r\n Q M fornextsearchstepi+1.Sincethegoalofthisworkis\r\n andisallocated inthememory. Sizeofu isproduct( ) of i+1\r\n i to minimizetheoverall\u00b5 , weidentifythecorresponding\r\n u .shape,whereshapeisapropertyoftheactivationtensor peak\r\n i optimalschedules\u2217 foreachz byonlysavings with\r\n that includes channels, height, width, and the precision (e.g., i+1 i+1 i+1\r\n byte, \ufb02oat), is added to \u00b5 , so \u00b5 \u2190\u00b5 +Q(u.shape). the minimum\u00b5peak,i+1. We integrate the aforementioned\r\n i i+1 i i step of scheduling u and updating M to complete\r\n Then we use \u00b5 as \u00b5 to update \u00b5 (peak i i+1\r\n i+1 peak peak,i+1 the proposed dynamic programming-based scheduling\r\n memory footprint for si+1). Since some predecessors of algorithm. Algorithm1summarizesthethealgorithm. Asa\r\n u will not be used anymore after allocating u , we update\r\n i i \ufb01rststep,thealgorithmstartsbyinitializingthememoization\r\n the outdegrees of the node by decrementing them. Having table M ,thenthealgorithmiteratesdifferentsearchsteps.\r\n updatedtheoutdegree,wewillbeleftwithazero-outdegree 0\r\n set that denotes the nodes that are ready for deallocation. We In each search step i, the algorithm performs the above\r\n illustratedmemoryallocationforallu inz ,andsavings ,\r\n deallocatethenodesinthesetandupdate\u00b5 accordingly. i i i+1\r\n i+1 \u00b5 , and \u00b5 . After iterating all search steps to n\u22121,\r\n i+1 peak,i+1\r\n Todemonstrateschedulingofanodeu ,Figure6simulates s issavedinM withauniqueentry,fornbeingnumber\r\n i \u2217 n\r\n schedulinganodeu = H ini=8. Inthe\ufb01gure,(1) H isap- ofnodesinG. Weprovidetheprooffortheoptimalityofthe\r\n 8\r\n pendedtos8 andallocatedtomemoryasitisscheduled,and peakmemoryfootprintinthesupplementarymaterial.\r\n thentheschedulerrecordsmaximumofthe\u00b5peak,8 andthe\r\n sumofallactivationsinthememoryatthispointas\u00b5peak,9. Complexity of the algorithm. The complexity of the\r\n Then,itrecalculatestheoutdegreesofthepredecessornodes proposed dynamic programming-based scheduling is\r\n of H: D and E\u2019s outdegree are decremented from one to O(|V|\u00d72|V|),whichissigni\ufb01cantlyfasterthantheexhaus-\r\n zero. (2) Then these nodes are deallocated and sum of the tivesearchofS withanupperboundcomplexityofO(|V|!).\r\n T\r\n OrderingChaos: Memory-AwareSchedulingofIrregularlyWiredNeuralNetworksforEdgeDevices\r\n A g1 Schedule A sg1 A s* Adaptive soft budgeting. While divide-and-conquer\r\n B C B B approach scales down the number of nodes, the algorithm\r\n D C C\r\n D Concatenate D maystillnotbefastenoughduetotheexponentialcomplexity\r\n E g2 E sg2 E ofthealgorithm. Therefore,weexploreavoidingsuboptimal\r\n F G Schedule F F\r\n G G solutions during the early stage of scheduling without affect-\r\n H H H ingtheoptimalityoftheoriginalalgorithm. Sinceourgoalis\r\n Divide Conquer Combine to \ufb01ndasinglesolutionthatcanrunwithinagivenmemory\r\n Figure7.Illustration of divide-and-conquer, which divides the budget\u03c4\u2217=\u00b5\u2217whileallothersolutionscanbediscarded,set-\r\n graphs into multiple subgraphs (divide), schedules each of them ting somebudget\u03c4 thatisgreaterorequalto\u00b5\u2217 andpruning\r\n using the optimal scheduler (conquer), then concatenates the suboptimalscheduleswithwhichtheir\u00b5 exceeds\u03c4 can\r\n sub-schedulestogetthe\ufb01nalschedule(combine). peak\r\n focusthesearchtoasmallersearchspaceS\u2032 \u2282S whilestill\r\n T T\r\n achievingtheoptimalschedules\u2217. Ontopofthis,wedevelop\r\n Duetothespacelimitation,wepresentthederivationofthe ameta-searchfor\u03c4. Thisisinspiredfromengineersbuying\r\n algorithmcomplexityinthesupplementarymaterial. alargermemory(increase\u03c4)ifaprogramfailsduetostack\r\n over\ufb02ow(=\u2019nosolution\u2019duetoanoverlyaggressivepruning)\r\n 3.2 OptimizingSchedulingSpeed: Speedingup and selling out excess memory (decrease \u03c4) if the current\r\n theDynamicProgramming-basedScheduling budget is prohibitive (= \u2019timeout\u2019 due to lack of pruning).\r\n Whiletheaboveschedulingalgorithmimprovescomplexity SERENITY takes advantage of this insight to develop an\r\n of the search, search space may still be intractable due to adaptivesoftbudgetingschemewhileschedulingtocutdown\r\n theimmenseirregularity. Therefore,wedevisedivide-and- theoverallnumberofexploredschedules. Figure8illustrates\r\n conquerandadaptivesoftbudgetingtoacceleratethesearch the overall idea by \ufb01rst showing how some schedules are\r\n byeffectivelyshrinkingandpruningthesearchspace. pruned with regard to a given budget \u03c4 in Figure 8(a) then\r\n implicationofdifferent\u03c4 onschedulingtimeinFigure8(b).\r\n Divide-and-conquer. We can observe from Figure 1 Figure8(a)depictsacertainpointwhileschedulingG,where\r\n that the topology of irregularly wired neural networks are nodes G, H, F,and J canbescheduled. Inparticular,the\r\n hourglass shaped ( \u22b3 ), because many NAS and Random\r\n \u22b2\r\n NetworkGeneratorsdesigncellswithsingleinputandsingle X Scheduled X Schedulable X X For memoization\r\n outputthenstackthemtoformanhourglassshapetopology.\r\n (Wilkenetal.,2000)showsthat,duringgeneralpurposecode A output activation size\r\n scheduling,graphscanbepartitioned(divide)intomultiple B C \u03c4 = 36 C 6 D 6 E 6 F 6\r\n subgraphsandthecorrespondingsolutions(conquer)canbe G 3 H 3 I 3 J 6\r\n D E F 35\r\n concatenated(combine)toformanoptimalsolutionforthe G H I J \u2026 z A,B,C,D,E,I 32 \u2026\r\n overall problem. Whilethecomplexityoftheschedulingal- K ch Step 35 35 38 38> \u03c4 s3\r\n gorithmremainsthesame,thisdivide-and-conquerapproach Sear G H23 32F J\r\n canreducethenumberofnodesineachsubproblem,speed- L \u2026 s1 s2 \u2026\r\n Graph G z A,B,C,D,E,I,F,H 23\r\n inguptheoverallschedulingtime. Forinstance,foragraph (a) Whilebothpaths ands schedulesleadtosamez\u2032,their\r\n that can be partitioned into N equal subgraphs, the schedul- 1 2\r\n \u00b5and\u00b5 varies and we can prune schedules that yield\r\n |V| |V|/N peak\r\n ing time will decrease from |V |\u00d72 to |V |\u00d72 that higher\u00b5 thanagivenbudget\u03c4. Numbersnexttoboxor\r\n wecanspeedupschedulingbymultipleordersofmagnitude peak\r\n circle are \u00b5 and numbersnexttoedgesare\u00b5\r\n peak\r\n compared to the naive approach, depending on the size of Optimal Budget \u03c4* Hard Budget \u03c4max\r\n thegraphandthenumberofpartitions. Soft Budget \u03c4\r\n )\r\n Assuch,Figure7showsthisinsightcanbeextendedtoour ' )\r\n on '\r\n ed Schedules i out\r\n problemsetting,wherewecan\ufb01rstperformschedulingon ut e\r\n ol m\r\n i\r\n t\r\n '\r\n no s Prohibitive (\r\n each cell and merge those solutions together to form the '\r\n Scheduling Time ( Scheduling Time\r\n \ufb01nalsolution. First, stage is partitioning the original graph \u221d Scheduling Failure \"EBQUJWF\u00014PGU\u0001#VEHFUJOH\r\n G into multiple subgraphs g (divide). Then, utilizing the No. of Explor\r\n Budget\r\n independence among the subgraphs, each subgraph g can (b) Adaptive soft budgeting starts by setting a hard budget\r\n be scheduled separately for their corresponding optimal \u03c4 as the maximum value for the soft budget \u03c4. Then,\r\n max\r\n schedule sg (conquer). Considering that the number of conductsabinarysearchfor\u03c4,higherthan\u03c4\u2217 thatit\ufb01ndsa\r\n nodesinthesubgraphgismuchsmallerthantheentiregraph solutionyetnottoohighthatschedulingcompletesquickly.\r\n G, the scheduling time will decrease signi\ufb01cantly. Finally, Figure8.Illustrationoftheadaptivesoftbudgeting. (a)showshow\r\n the schedules of the subgraphs are concatenated to give schedules are pruned, and (b) illustrates how the soft budget \u03c4\r\n \u2217 relates to the number of explored schedules.\r\n optimalschedules oftheentiregraph(combine).\r\n OrderingChaos: Memory-AwareSchedulingofIrregularlyWiredNeuralNetworksforEdgeDevices\r\n x th y wij th th\r\n \ufb01gure compares two possible solutions s1 and s2 which i i Input Output j Channel of i Kernel\r\n schedules H\u2192F and F\u2192H, respectively given \u03c4 = 36. x1 x2 xn x1 x2 xn\r\n Whiles ands bothstartsfromz with\u00b5=32,scheduling\r\n 1 2\r\n H leads to \u00b5 =32+3(H)=35,whereasscheduling F DPODBU QBSUJBM\r\n peak Channel-wise x = DPOW w?1 w?2 w?n\r\n or J leads to \u00b5peak=32+6(ForJ)=38. Therefore,since Partitioning DPOW w1\u2026wm BEE\r\n weassume\u03c4 =36,s ands willfailbecause\u00b5 =38\r\n 2 3 peak y y\r\n for s and s exceeds 36. So, as long as we set the budget\r\n 2 3 \u00b5peak = \u03a3size(x ) + size(y) \u00b5peak = max(size(x ) + size(y))\r\n \u2217 i i \r\n \u03c4 higher than \u00b5 , the scheduler still \ufb01nds a single optimal\r\n x1 x2 xn x2\r\n solution while avoiding many suboptimal paths. On the x1 xn\r\n other hand, too small a \u03c4 <\u00b5\u2217 leads to no solution because Kernel-wise DPODBU QBSUJBM\r\n theoptimalpathwouldbeprunedaway. x = EFQUI\u000eDPOW w1 w2 wn\r\n Partitioning EFQUI\u000eDPOW w1\u2026wn DPODBU\r\n Havingestablishedthepossibilityofpruning,ourquestion y y\r\n boils down to discovering \u03c4 that is greater or equal to \u00b5\u2217 \u00b5peak = \u03a3size(x ) + size(y) \u00b5peak = max(size(x ) + size(w x ))\r\n which we call an optimal budget \u03c4\u2217, yet close enough to i i i i \r\n shrink the search space effectively. Figure 8(b) and Algo- Figure9.Illustrationofthegraphrewritingpatterns: channel-wise\r\n rithm 2 summarizes the proposed adaptive soft budgeting. partitioning and kernel-wise partitioning can reduce the memory\r\n Since we start with no information about the approximate costofconvolutionanddepthwiseconvolutionrespectively.\r\n range for \u03c4, we resort to a commonly used topological\r\n ordering algorithm called Kahn\u2019s algorithm (Kahn, 1962) soonasit\ufb01ndsaschedule(\u2019solution\u2019),andthismethodusing\r\n (O(|V|+|E|))toadaptivelygainideaoftherangefor\u03c4. We binarysearchisguaranteedtoworkduetothemonotonically\r\n usethepeakmemoryfootprintfromthissequenceanduse increasingnumberofexploredscheduleswith\u03c4.\r\n it as our hard budget \u03c4 , and in contrast we call adaptively\r\n max\r\n changing\u03c4 asasoftbudget. Since\u03c4 \u2265\u00b5\u2217,weknowthat 3.3 IdentityGraphRewriting: Improvingthe\r\n max\r\n any\u03c4\u2265\u03c4 donotneedtobeexplored. Havingthisupper SearchSpaceforBetterPeakMemoryFootprint\r\n max\r\n bound for the search, adaptive soft budgeting implements Reorganizing the computational graph of the irregularly\r\n abinarysearchto\ufb01rstruntheschedulingalgorithmwith\u03c4 wiredneuralnetworksmayleadtosigni\ufb01cantreductioninthe\r\n andT asinput,whereT isanhyperparameterthatlimitsthe peakmemoryfootprint\u00b5 duringcomputation. Forexam-\r\n schedulingtimepersearchstep. Thebinarysearchincreases peak\r\n \u03c4 (\u03c4 \u2190(\u03c4 +\u03c4 )/2) if it \ufb01nds \u2019no solution\u2019 and ple, it is notable that large stream of NAS-based works (Liu\r\n new new old et al., 2019a; Zhang et al., 2019) rely on extensive use of con-\r\n decreases\u03c4 (\u03c4 \u2190\u03c4 /2)ifasearchstepreturns\u2019timeout\u2019\r\n new new catenationasanaturalapproachtomergeinformationfrom\r\n (searchstepdurationexceedsT). Thebinarysearchstopsas multiple branches of the input activations and expand the\r\n searchspaceoftheneuralarchitectures. However,concatena-\r\n Algorithm2AdaptiveSoftBudgeting tion with manyincomingedgesmayprolongthelivenessof\r\n 1: Input: graphG theinputactivationandincreasethememorypressure,which\r\n is unfavorable especially for resource constrained scenarios.\r\n \u2217\r\n 2: Output: optimalschedules Toaddress this issue, we propose identity graph rewriting\r\n 3: \u03c4 \u2190\u00b5(Kahn\u2019sAlgorithm(G),G)//hardbudget\r\n max to effectively reduce \u00b5peak aroundtheconcatenationwhile\r\n 4: \u03c4 ,\u03c4 \u2190\u03c4\r\n old new max keeping the arithmetic outputs identical. To this end, we\r\n 5: flag\u2190\u2019nosolution\u2019 presenttwomainexamplesofthegraphpatternsinirregularly\r\n 6: repeat wiredneuralnetworksthatbene\ufb01tsfromourtechnique:\r\n 7: // binary search for \u03c4: decrease \u03c4 if \u2019timeout\u2019\r\n 8: // and increase \u03c4 if \u2019no solution\u2019 Channel-wise partitioning (convolution). One typical\r\n 9: if flag is \u2019timeout\u2019 then pattern in irregularly wired neural networks is concatenation\r\n 10: // simultaneous (concat: [\u00b7]) that takes multiple branches of the input prior to\r\n 11: \u03c4 \u2190\u03c4 ,\u03c4 \u2190\u03c4 /2\r\n old new new new aconvolution(conv: \u2217). Whileexecutingsuchpattern,peak\r\n 12: else if flag is \u2019no solution\u2019 then memoryfootprint \u00b5 occurs when the output y \u2208Rn is\r\n 13: // simultaneous peak\r\n beingcomputedwhileconcatenatedbranchesofinputx\u2208Rn\r\n 14: \u03c4 \u2190\u03c4 ,\u03c4 \u2190(\u03c4 +\u03c4 )/2\r\n old new new new old are also mandated to reside in the memory. Our objective\r\n 15: endif is to achieve the same arithmetic results and logical effect\r\n 16: if flag is \u2019solution\u2019 then as concat yet sidestep the corresponding seemingly exces-\r\n 17: s\u2217\u2190schedule//optimalschedule sive memory cost. To this end, we channel-wise partition\r\n 18: endif theconvthatfollowstheconcatsothatthepartitionedconv\r\n 19: untilflag is\u2019solution\u2019 canbecomputedassoonastheinputx becomesavailable.\r\n i\r\n OrderingChaos: Memory-AwareSchedulingofIrregularlyWiredNeuralNetworksforEdgeDevices\r\n Equation3-6detailthemathematicalderivationofthissub- Table1. Speci\ufb01cationofthenetworksusedforevaluation.\r\n stitution. Speci\ufb01cally, as shown in Equation 3, each kernel NETWORK TYPE DATASET # MAC # WEIGHT TOP-1\r\n iteratesandsumsuptheresultofconvolvingchannelsinconv. ACCURACY\r\n However,usingthedistributivepropertyofP and\u2217,these DARTS NAS IMAGENET 574.0M 4.7M 73.3%\r\n i SWIFTNET NAS HPD 57.4M 249.7K 95.1%\r\n transformtosummationofchannel-wisepartitionedconvo- RANDWIRE RAND CIFAR10 111.0M 1.2M 93.6%\r\n lution, whichwecallpartialconv. Thispartialconvremoves RANDWIRE RAND CIFAR100 160.0M 4.7M 74.5%\r\n concat from the graph leading to lower memory cost. As\r\n illustrated in Figure 9, the memory cost of same computation 4 EVALUATION\r\n P\r\n reducesfrom xi+ytomax(w\u22c6i\u2217xi)+y,whichbecomes WeevaluateSERENITYwithfourrepresentativeirregularly\r\n moreeffectivewhentherearemoreincomingedgestoconcat.\r\n h i wired neural networks graphs. We \ufb01rst compare the\r\n X X peak memory footprint of SERENITY against TensorFlow\r\n y= w \u2217x ,..., w \u2217x (concat+conv) (3)\r\n 1i i mi i Lite(Google)whileusingthesamelinearmemoryallocation\r\n ih i i 1\r\n =Xw \u2217x,...,w \u2217x (4) scheme for both. Furthermore, we also experiment\r\n 1i i mi i the impact of such peak memory footprint reduction on\r\n i h i off-chip memory communication. We also conduct an\r\n =Xw ,...,w \u2217x (5) in-depth analysis of the gains from the proposed dynamic\r\n 1i mi i\r\n i h i programming-based scheduler and graph rewriting using\r\n =Xw\u22c6i\u2217xi (partial conv+add) (6) SwiftNetCellA(Zhangetal.,2019). Lastly, westudythe\r\n i impactofadaptivesoftbudgetingontheschedulingtime.\r\n Kernel-wise partitioning (depthwise convolution). 4.1 Methodology\r\n Depthwiseconvolution(depthconv)(Sifre&Mallat,2014; Benchmarks and datasets. Table 1 lists the details of\r\n Howard et al., 2017) has been shown to be effective in thenetworks\u2013representativeoftheirregularlywiredneural\r\n reducingcomputationyetachievecompetitiveperformance, networks from Neural Architecture Search (NAS) and\r\n henceitswideuseinnetworksthattargetextremeef\ufb01ciency RandomNetworkGenerators(RAND)\u2013usedforevaluation:\r\n as its primary goal. For concatenation (concat) followed DARTS(Liuetal.,2019a)forImageNet,SwiftNet(Zhang\r\n by a depthwise convolution (depthconv), similar to above et al., 2019) for a dataset comprised of human presenceorab-\r\n concat+conv case, peak memory footprint \u00b5 occurs\r\n peak sence(HPD),andRandWire(Xieetal.,2019)forCIFAR10\r\n when the concatenated x is inside the memory and the and CIFAR100. DARTS (Liu et al., 2019a) is a gradient-\r\n result y additionally gets saved to the memory before x basedNASalgorithm. Inparticularwefocusonthelearned\r\n is deallocated. This time, we leverage the independence normalcellforimageclassi\ufb01cationonImageNet: onlythe\r\n among different kernels to kernel-wise partition the \ufb01rstcellbecauseithasthehighestpeakmemoryfootprintand\r\n depthconv that follows the concat so that each input x\r\n i theresetofthenetworkisjustrepeatedstackingofthesame\r\n is computed to smaller feature maps without residing in cell following the practice in NASNet (Zoph et al., 2018).\r\n the memory too long. As such, Equation 7-8 derives this SwiftNet(Zhangetal.,2019)isnetworkfromNASbytarget-\r\n substitution. Equation7showsthateverycomponentinthe inghumandetectiondataset. RandWire(Xieetal.,2019)are\r\n yisindependent(differentsubscriptindex)andisviablefor fromRandomNetworkGeneratorsforimageclassi\ufb01cation\r\n partitioning. In other words, this rewriting simply exposes onCIFAR10andCIFAR100.Thetablealsoliststheirdataset,\r\n the commutative property between depthconv and concat multiply-accumulatecount(# MAC),numberofparameters\r\n pluskernel-wisepartitioningtoreduce\u00b5peak signi\ufb01cantly. (# WEIGHT),andtop-1accuracyontheirrespectivedataset.\r\n h i\r\n y= w \u2217x ,...,w \u2217x (concat+depthconv) (7)\r\n 1 1 n n 4.2 ExperimentalResults\r\n h i\r\n = [w \u2217x ],...,[w \u2217x ] (partialdepthconv+concat)\r\n 1 1 n n ComparisonwithTensorFlowLite. Figure10evaluates\r\n (8) SERENITY over TensorFlow Lite on different cells of the\r\n aforementionednetworksintermsofreductioninmemory\r\n Implementation. Followingthegeneralpracticeofusing footprint. The \ufb01gures illustrate that SERENITY\u2019s dynamic\r\n patternmatchingalgorithmsincompilers(Lattner&Adve, programming-basedschedulerreducesthememoryfootprint\r\n 2004; Rotem et al., 2018; Jia et al., 2019), we implement by a factor of 1.68\u00d7without any changes to the graph. In\r\n identity graph rewriting using pattern matching to identify 1TensorFlowLiteimplementsalinearmemoryallocatornamed\r\n regionsofthegraphwhichcanbesubstitutedtoanoperation simplememoryarena: https://github.com/tensorflo\r\n withlowercomputationalcost. Likewise,wemakeuseofthis w/tensorflow/blob/master/tensorflow/lite/s\r\n techniquetoidentifyregionsthatleadstolowermemorycost. imple memory arena.cc\r\n OrderingChaos: Memory-AwareSchedulingofIrregularlyWiredNeuralNetworksforEdgeDevices\r\n 4.00 Higher the better 45x TensorFow Lite 250\r\n y Dynamic Programming+Memory Allocator )\r\n r 3. B 200\r\n mo 70x Dynamic Programming+Graph Rewriting+Memory Allocator K\r\n 3.00 (\r\n e 44x 2. \r\n 39x t\r\n M 20x 20x 2. n\r\n 2. 09x i\r\n k 2. 2. r 150\r\n a 83x 86x p\r\n e 2. t\r\n P 1. 68x 68x 68x 1. o\r\n 2.00 40x 40x 1. 1. 1. o\r\n n 39x 39x F\r\n i 27x 27x 25x 25x \r\n 1. 1. 1. 1. y 100 25.1KB reduction in peak memory\r\n n 1. 1. 1. 1. r\r\n o o\r\n i\r\n t m footprint with Memory Allocator\r\n c 1.00\r\n u\r\n d Me 50 Dynamic Programming+Memory Allocator\r\n Re .FNPSZ\u0001'PPUQSJOU\u0001\t,#\n Dynamic Programming+Graph Rewriting+Memory Allocator\r\n 3FEVDUJPO\u0001JO\u00011FBL\u0001.FNPSZ 0.00 0\r\n /PSNBM Normal $FMM\u0001\" Cell A $FMM\u0001# Cell B $FMM\u0001$ Cell C $FMM\u0001\" Cell A $FMM\u0001# Cell B $FMM\u0001\" Cell A $FMM\u0001# Cell B $FMM\u0001$ Cell C (FPNFBO Geomean 5JNF Time\r\n %\"354 DARTS 4XJGU/FU SwiftNet 3BOE8JSF RandWire 3BOE8JSF RandWire (a) Memoryfootprintwiththememoryallocator(peakmem-\r\n ImageNet Visual Wake Words Dataset CIFAR10 CIFAR100\r\n *NBHF/FU )VNBO\u00011SFTFODF $*'\"3\u0012\u0011 $*'\"3\u0012\u0011\u0011 oryfootprintofTensorFlowLite=551.0KB).\r\n Figure10.Reduction in peak memory footprint of SERENITY 200\r\n againstTensorFlowLite(nomemoryhierarchy). )\r\n B\r\n K 150\r\n (\r\n \r\n 4.00 32KB 64KB 128KB 256KB t\r\n 56x n\r\n i\r\n 3. r\r\n SERENITY removes o\ufb00-chip communication p\r\n t\r\n o 100\r\n p 68x o\r\n