{"title": "Searching for Winograd-aware Quantized Networks", "book": "Proceedings of Machine Learning and Systems", "page_first": 14, "page_last": 29, "abstract": "Lightweight architectural designs of Convolutional Neural Networks (CNNs) together with quantization have paved the way for the deployment of demanding computer vision applications on mobile devices. Parallel to this, alternative formulations to the convolution operation such as FFT, Strassen and Winograd, have been adapted for use in CNNs offering further speedups. Winograd convolutions are the fastest known algorithm for spatially small convolutions, but exploiting their full potential comes with the burden of numerical error, rendering them unusable in quantized contexts. In this work we propose a Winograd-aware formulation of convolution layers which exposes the numerical inaccuracies introduced by the Winograd transformations to the learning of the model parameters, enabling the design of competitive quantized models without impacting model size. We also address the source of the numerical error and propose a relaxation on the form of the transformation matrices, resulting in up to 10% higher classification accuracy on CIFAR-10. Finally, we propose wiNAS, a neural architecture search (NAS) framework that jointly optimizes a given macro-architecture for accuracy and latency leveraging Winograd-aware layers. A Winograd-aware ResNet-18 optimized with wiNAS for CIFAR-10 results in 2.66\u00d7 speedup compared to im2row, one of the most widely used optimized convolution implementations, with no loss in accuracy.", "full_text": " SEARCHING FOR WINOGRAD-AWARE QUANTIZED NETWORKS\r\n Javier Fernandez-Marques1 Paul N. Whatmough2 AndrewMundy2 MatthewMattina2\r\n ABSTRACT\r\n Lightweight architectural designs of Convolutional Neural Networks (CNNs) together with quantization have\r\n paved the way for the deployment of demanding computer vision applications on mobile devices. Parallel to this,\r\n alternative formulations to the convolution operation such as FFT, Strassen and Winograd, have been adapted for\r\n use in CNNs offering further speedups. Winograd convolutions are the fastest known algorithm for spatially small\r\n convolutions, but exploiting their full potential comes with the burden of numerical error, rendering them unusable\r\n in quantized contexts. In this work we propose a Winograd-aware formulation of convolution layers which exposes\r\n the numerical inaccuracies introduced by the Winograd transformations to the learning of the model parameters,\r\n enabling the design of competitive quantized models without impacting model size. We also address the source of\r\n the numerical error and propose a relaxation on the form of the transformation matrices, resulting in up to 10%\r\n higher classi\ufb01cation accuracy on CIFAR-10. Finally, we propose wiNAS, a neural architecture search (NAS)\r\n framework that jointly optimizes a given macro-architecture for accuracy and latency leveraging Winograd-aware\r\n layers. A Winograd-aware ResNet-18 optimized with wiNAS for CIFAR-10 results in 2.66\u00d7 speedup compared\r\n to im2row, one of the most widely used optimized convolution implementations, with no loss in accuracy.\r\n 1 INTRODUCTION chosen in favour of depthwise convolutions for applications\r\n The rise in popularity of deep CNNs has spawned a re- like image super-resolution (Zhang et al., 2018; Lee et al.,\r\n search effort to \ufb01nd lower complexity networks to increase 2019), image segmentation (Takikawa et al., 2019) and,\r\n inference ef\ufb01ciency. This is desirable for inference in the GANs(Brock et al., 2019). Therefore, alternative forms\r\n cloud and becomes crucial on mobile and IoT devices with of speeding up standard convolutions are required to run\r\n muchmoreconstrained hardware (Lane & Warden, 2018). these applications in mobile CPUs, which often come with\r\n Over the last few years, multiple approaches have been constrained compute and energy budgets (Whatmough et al.,\r\n proposed to alleviate the compute-bound nature of convo- 2019). Model quantization and the use of alternative convo-\r\n lutions (Sze et al., 2017). Arguably, the use of depthwise lution algorithms instead of direct convolution are two ways\r\n convolutions, as popularized by the family of MobileNet of accomplishing this task.\r\n architectures (Howard et al., 2017; Sandler et al., 2018; Lower-precision networks result in smaller model sizes,\r\n Howard et al., 2019), has become the most widely em- faster inference, lower energy consumption and smaller\r\n braced design choice to make lightweight networks. These chip area (Sze et al., 2017; Li et al., 2019). Concretely, 8-bit\r\n layers are used in state of the art image classi\ufb01cation net- quantizatized models achieve comparable performance to\r\n works (Stamoulis et al., 2019; Tan & Le, 2019). However, full-precision models (Jacob et al., 2018; Krishnamoorthi,\r\n beyond image classi\ufb01cation, normal convolutions are still 2018) while being ready for deployment on off-the-shelf\r\n This project has received funding from the European Union\u2019s hardware as 8-bit arithmetic is widely supported. In addi-\r\n Horizon 2020 research and innovation programme under grant tion to resulting in a direct 4\u00d7 model size reduction, 8-bit\r\n agreement No 732204 (Bonseyes). This work is supported by the integer-only arithmetic bene\ufb01ts from up to 116\u00d7 and 27.5\u00d7\r\n Swiss State Secretariat for Education Research and Innovation chip area reduction compared to full precision additions and\r\n (SERI) under contract number 16.0159. The opinions expressed multiplies respectively, requiring 30\u00d7 and 18.5\u00d7 less en-\r\n and arguments employed herein do not necessarily re\ufb02ect the ergy (Horowitz, 2014; Whatmough et al., 2018). Because of\r\n of\ufb01cial views of these funding bodies. these desirable bene\ufb01ts, 8-bit quantization has been widely\r\n 1Department of Computer Science, University of Oxford (UK) adopted in both compute-constrained devices (Liberis &\r\n 2ArmMLResearchLab. Correspondence to: Javier Fernandez- Lane, 2019; Chowdhery et al., 2019; Wang et al., 2019) and\r\n Marques. accelerators (Whatmough et al., 2019).\r\n Proceedings of the 3rd MLSys Conference, Austin, TX, USA, Orthogonal to lightweight architectural designs and quanti-\r\n 2020. Copyright 2020 by the author(s). zation, fast convolution algorithms in replacement of direct\r\n Searching for Winograd-aware Quantized Networks\r\n convolutions can provide further speedups. These come Input\r\n with their own trade-offs (Anderson & Gregg, 2018), but in Output\r\n this work we focus on the Winograd algorithm since it is the\r\n fastest known algorithm for convolutions of the dimensions 3x3 \r\n convolution\r\n often found in CNNs. The Winograd convolution performs\r\n the bulk of the computation as a Hadamard product between\r\n weights and input in the Winograd space requiring O(n)\r\n operations. Unlike normal convolutions, that generate a\r\n single output per convolution, a Winograd convolution com- Figure 1. Standard convolutions operate on a tile (blue) de\ufb01ned by\r\n putes several outputs simultaneously. This property makes the \ufb01lter size, generating a single output. Winograd convolutions\r\n Winograd convolutions minimal in the number of general operate on larger tiles without modifying the \ufb01lter dimensions. A\r\n 1 3\u00d73Winogradconvolutionoperatingona4\u00d74tile(red)generates\r\n multiplications (Winograd, 1980). Normal convolutions op-\r\n erate on tile sizes matching the width and height of the \ufb01lter, a 2 \u00d7 2 output. This is often expressed as F(2 \u00d7 2,3 \u00d7 3).\r\n on the other hand, Winograd convolutions can operate on\r\n larger tiles. This is illustrated in Figure 1. In this way, while \u2022 WeshowthatWinograd-awarenetworks enable Wino-\r\n normal convolutions would require 8.1K multiplications to grad convolutions in quantized networks, including\r\n densely convolve a 32 \u00d7 32 input with a 3 \u00d7 3 \ufb01lter, Wino- 8-bit networks with little accuracy drop. To the best\r\n gradconvolutionsoperatingona4\u00d74tilerequireonly3.6K. of our knowledge, this is the \ufb01rst time this has been\r\n ThespeedupsWinogradconvolutionsofferincreasewithtile empirically demonstrated.\r\n size. However, exploiting these speedups exposes a problem\r\n inherent in current Winograd convolution implementations: \u2022 WedemonstratethatlearningtheWinogradtransforms,\r\n numerical error. This error, which grows exponentially with as opposed to keeping these \ufb01xed, results in better\r\n tile size, is the primary reason why Winograd is generally network accuracy \u2013 up to 10% improvement when\r\n only deployed with 32-bit \ufb02oating point and for compara- using 6 \u00d7 6 and 8 \u00d7 8 tiles in 8-bits CNNs with 3 \u00d7 3\r\n tively small tile sizes, rarely larger than 6 \u00d7 6. In practice, \ufb01lters. This improvement is more pronounced with\r\n an architecture with standard convolutional layers would larger 5 \u00d7 5 \ufb01lters.\r\n \ufb01rst be trained before replacing standard convolution with \u2022 Wepresent wiNAS as a tool that can \ufb01nd Winograd-\r\n Winograd convolution for deployment. aware networks jointly optimised for both high accu-\r\n In this paper, we focus on alleviating the problem of nu- racy and low latency given a real hardware model.\r\n merical error that arises when using Winograd convolutions\r\n in quantized neural networks. Achieving this ultimately\r\n enables us to combine the speedups of Winograd with those 2 RELATEDWORK\r\n that reduced precision arithmetic is known to offer, among\r\n other bene\ufb01ts in terms of energy and area. To this end, we Convolutions have become the de facto spatial feature ex-\r\n present an end-to-end training pipeline that exposes the nu- tractor in neural networks. As a result, a number of ap-\r\n merical inaccuracies introduced by Winograd to the learning proaches have emerged to reduce the computational costs\r\n of the model parameters. We also address the source of the of using this operator.\r\n numerical error and propose a relaxation on the form of\r\n the transformation matrices used in the Winograd convolu- CompactCNNArchitectures. Theseincludealternative\r\n tion algorithm. We achieve this by adding these matrices formulations to the dense convolutional layer, such as bot-\r\n to the set of learnable parameters in a layer, after initial- tleneck layers (He et al., 2016) that perform the 3 \u00d7 3 con-\r\n izing them via Cook-Toom (L. Toom, 1963). Finally, we volutions in a lower-dimensional space, or the depth-wise\r\n describe wiNAS, a Winograd-aware Neural Architecture convolutional layers (Howard et al., 2017) which replace\r\n Search framework which leverages Winograd-aware layers the standard convolutions with a channel-wise convolution\r\n and latency measurements on Arm Cortex-A73 and A53 followed by a 1 \u00d7 1 point-wise convolution. More recently,\r\n cores, to jointly optimize for high accuracy and low latency. Chen et al. (2019) proposed a compact multi-resolution\r\n Our framework transforms a given macro-architecture by convolutional block that reduces spatial redundancy of low\r\n replacing each convolution with either im2row or Winograd frequencies resulting in faster inference, memory savings\r\n convolutions of different tile sizes. and slightly higher accuracy. This reinforces the proposi-\r\n Thecontributions of this work are summarized below: tion that current networks rely more on texture than shape\r\n for image/object discrimination (Brendel & Bethge, 2019;\r\n 1Generalmultiplications is a term commonly used in Winograd Geirhos et al., 2019). In this work, instead of presenting\r\n jargon referring to element-wise or Hadamard product stage. a new architecture, we propose an optimization for an ex-\r\n isting known-good architecture to speed up inference. Our\r\n Searching for Winograd-aware Quantized Networks\r\n optimization can be applied to existing pre-trained models the use of integer arithmetic for complex Winograd con-\r\n without the need for end-to-end training. volutions (Meng & Brothers, 2019); a general formulation\r\n for the Winograd algorithm (Barabasz & Gregg, 2019) that\r\n Quantization. The most extreme form of quantization shows promising results in FP16 and BFLOAT16 when\r\n is binary networks (Courbariaux et al., 2015; Lin et al., using higher degree polynomials; an ef\ufb01cient region-wise\r\n 2017; Xiang et al., 2017), which replace convolutions with multi-channel implementation of Winograd convolutions\r\n bit-shifts resulting in 58\u00d7 inference speed-ups (Rastegari using General Matrix Multiplications (GEMMs) (Maji et al.,\r\n et al., 2016). Ternary and 2-bit models (Li et al., 2016; 2019) that achieves 4\u00d7 speedups on Arm Cortex-A CPUs;\r\n Wanetal., 2018; Gong et al., 2019) achieve higher accura- and, a technique (Liu et al., 2018) that enables up to 90%\r\n cies while alleviating some the challenges of training binary sparsity in the Hadamard product stage of the Winograd\r\n networks (Alizadeh et al., 2019). However, it is 8-bit quan- algorithm, effectively reducing by 10\u00d7 the number of mul-\r\n tization (Jacob et al., 2018; Krishnamoorthi, 2018; Wang tiplications with no accuracy loss in FP32 models. Our\r\n et al., 2019) that has achieved high popularity due to its bal- work\ufb01lls the gap of using Winograd convolutions in quan-\r\n ance between accuracy, model size reduction and inference tized neural networks, enabling even faster convolutions in\r\n speedup. Newer data formats, such as Posit (Carmichael current off-the-shelf hardware, such as mobile CPUs.\r\n et al., 2019) aim to close the accuracy gap between INT8\r\n and FP32 networks, however hardware supporting it is un- Neural Architecture Search. Automating the process of\r\n available. For training, BFLOAT16 (Kalamkar et al., 2019) designing neural network architectures has drawn consid-\r\n has been validated as an alternative to FP32, enabling faster erable attention. Early attempts relied on reinforcement\r\n training. In this work, we adopt INT8 and INT16 uniform learning (Zoph & Le, 2017; Brock et al., 2018; Real et al.,\r\n quantization during training and study how lowering preci- \u00b4\r\n sion impacts on the lossy nature of Winograd convolutions. 2019;Tanetal.,2019)orBayesianoptimization(Hernandez-\r\n Lobato et al., 2016; Fedorov et al., 2019) and required thou-\r\n Fast Convolution Algorithms. Alternative formulations sandsofGPUhourstoconvergeduetotheircomputationally\r\n of the convolution operation such as: the use of FFTs, which expensive and exhaustive search stages. Other works opted\r\n replace convolution with its multiplication-only counterpart instead for a gradient-based search by framing the problem\r\n in the frequency domain resulting in faster inference (Math- as a single over-parameterized network where all candidate\r\n ieu et al., 2013; Abtahi et al., 2018) and training (Highlander operations at a particular node (e.g. a layer) are taken into\r\n &Rodriguez,2015);theStrassenalgorithm(Strassen,1969), consideration. The main aspect differentiating gradient-\r\n which when applied to convolutions (Cong & Xiao, 2014; based NASapproaches is the way the output of a layer com-\r\n Tschannen et al., 2018) signi\ufb01cantly reduces the number of bines the contribution of each candidate operation. While\r\n multiplications at the cost of more additions; or the Wino- Bender et al. (2018) de\ufb01nes it as the sum and DARTS (Liu\r\n grad algorithm (Winograd, 1980), which replaces convolu- et al., 2019) as a weighted sum, ProxylessNAS (Cai et al.,\r\n tions with a set of matrix transformations and point-wise 2019)relies on path-level binarization, making it possible to\r\n multiplications and, results in signi\ufb01cantly faster inference perform the search on the entire architecture directly using\r\n stages (Lavin & Gray, 2016). a single GPU. In addition to architecture discovery, NAS\r\n has also been successfully used for automated network prun-\r\n Winograd Convolution. The Winograd algorithm for ing (He et al., 2018) and quantization (Wang et al., 2019).\r\n fast convolution was \ufb01rst applied to CNNs by Lavin Our work leverages NAS to \ufb01nd the optimal convolution\r\n & Gray (2016), showing 2.2\u00d7 speedup compared to algorithm (i.e. im2row or different Winograd implemen-\r\n cuDNN(Chetluretal., 2014) on a VGG (Simonyan & Zis- tations) for each layer in the model while preserving the\r\n serman, 2015) network, with no loss in accuracy on small overall network macro-architecture and model size.\r\n 4\u00d74tiles, batch 1. However, exploiting Winograd convo- 3 WINOGRAD-AWARENETWORKS\r\n lutions on larger input tiles is challenging due to numerical\r\n instability. In response to this limitation, Barabasz et al. This section introduces Winograd convolutions and their\r\n (2018) showed that the error introduced by the Winograd trade-offs in terms of compute, memory and accuracy. Then,\r\n algorithm grows at least exponentially with tile size, which wepresent the Winograd-aware layers used in our networks.\r\n can be partially alleviated by choosing better polynomial\r\n pointsforconstructingthetransformationmatricesviaCook- 3.1 Winogradimplementationtrade-offs\r\n Toom(L. Toom, 1963). An alternative formulation using\r\n trimmed Vandermonde matrices was described by Vincent TheWinogradalgorithm for convolutions using linear poly-\r\n et al. (2017). More recently, several works studying the nomials guarantees to use the minimum number of element-\r\n suitability of Winograd convolutions in memory and com- wise multiplications to compute m \u00d7 m outputs using an\r\n pute constrained setups have been proposed. These include: r \u00d7 r \ufb01lter. Lavin & Gray (2016) refer to this minimal\r\n Searching for Winograd-aware Quantized Networks\r\n algorithm as F(m\u00d7m,r\u00d7r)andpresentitsmatrixform: ResNet-18 Accuracy\r\n Convolution method 32-bit 16-bit 8-bit\r\n \u22a4h \u22a4 \u22a4 i Direct 93.16 93.60 93.22\r\n Y =A [GgG ]\u2299[B dB] A (1) Winograd F2 93.16 93.48 93.21\r\n Winograd F4 93.14 19.25 17.36\r\n Winograd F6 93.11 11.41 10.95\r\n whereG,BandAaretransformationmatricesappliedtothe Table 1. Replacing the convolutional layers in pre-trained ResNet-\r\n \ufb01lter g, input and output respectively and \u2299 is the Hadamard 18 models on CIFAR-10 with F2, F4 and F6. This works well\r\n or element-wise multiplication. in full precision, but accuracy drops drastically with quantization\r\n 2 for con\ufb01gurations beyond F2. Note that prior to evaluating the\r\n These transformation matrices are commonly constructed\r\n as described in the Cook-Toom algorithm which requires quantized con\ufb01gurations we performed a warmup of all the mov-\r\n choosing a set of so-called polynomial points from R2. This ing averages involved in Eq.1 using the training set but without\r\n choice is not trivial, but for small Winograd kernels e.g. modifying the weights. Without this relaxation (which requires a\r\n F(2 \u00d7 2,3 \u00d7 3) or F(4 \u00d7 4,3 \u00d7 3), there is a common Winograd-aware pipeline as in Fig. 2), F2 would be unusable.\r\n consensus. While a standard convolution using a r \u00d7r \ufb01lter\r\n gwouldoperateonar\u00d7rinputtile,aWinogradconvolution However, attempting this with larger Winograd tiles, or in\r\n expressed as Eq. 1 expects an input patch d with dimensions combination with quantization, results signi\ufb01cant accuracy\r\n (m+r\u22121)\u00d7(m+r\u22121).Thekeydifferenceisthatwhile 3\r\n loss. The root of the problem is the increasing numerical\r\n the standard convolution would generate a 1 \u00d7 1 output, range in G, B and A as d increases. As a consequence, the\r\n the Winograd convolution would compute a m \u00d7 m output. multiple matrix multiplications in Eq.1 contribute consid-\r\n In this way, a standard 3 \u00d7 3 convolution requires 9 mult. erable error, ultimately reducing accuracy. This problem is\r\n per output (mpo), F(2 \u00d7 2,3 \u00d7 3) and F(4 \u00d7 4,3 \u00d7 3) exacerbated in networks using quantized weights and acti-\r\n require 4 mpo and 2.25 mpo respectively. Theoretically, vations, where the range and precision of values is reduced.\r\n these savings grow as we increase the tile or \ufb01lter sizes. WeshowtheselimitationsinTable1. Thenumericalerroris,\r\n For the remaining of this work and, unless stated otherwise, to a large extent, the main limiting factor for adopting large-\r\n wewill be considering 3 \u00d7 3 \ufb01lters and therefore refer to tiled Winograd and for adopting Winograd convolutions in\r\n F(2\u00d72,3\u00d73)asF2,F(4\u00d74,3\u00d73)asF4,andsoon. general for reduced precision networks.\r\n Thechallenges associated with the use of Winograd convo- In this work we focus on minimizing the numerical errors\r\n lutions span three dimensions: that arise when using the Winograd algorithm in quantized\r\n Compute. Winogradconvolutions require the transforma- networks. Our approach does not aggravate the compute\r\n tion of both tile d and \ufb01lter g to the Winograd domain. The and memory challenges previously mentioned. Instead, it\r\n cost of these transformations grows with m, and can repre- indirectly alleviates these by making use of quantization.\r\n sent a signi\ufb01cant portion of the total computation of up to\r\n 75%(Sec. 6.2). This suggests that Winograd offers little to 3.2 AWinograd-awaretrainingpipeline\r\n \u22a4\r\n nospeedupinlayers with few \ufb01lters. The cost of GgG is Neural networks have proven to be resilient to all kinds of\r\n often ignored as it is amortized across inferences. approximations, e.g. pruning and quantization. When ap-\r\n \u22a4 plying these techniques, consistently better models are gen-\r\n Memory. In Eq.1, GgG transforms the \ufb01lter g to the\r\n Winograddomain,matchingthedimensionsoftheinputtile erated if these approximations are present during training.\r\n d. This results in an increase of run-time memory associated In other words, when the training is aware of quantization,\r\n withtheweights: 1.78\u00d7and4\u00d7forF2andF4respectively. or when training is aware of pruning.\r\n This is especially undesirable on memory-constrained de- Following this intuition, we propose an end-to-end\r\n vices such as microcontrollers. Winograd-aware pipeline as shown in Figure 2. In the\r\n Numerical Error. Small F2 and F4 perform well in forward pass we apply Eq.1 to each patch of the activa-\r\n single and double precision (FP32/64) networks and are tions from the previous layer. We can apply standard back-\r\n available in production-ready libraries such as NVIDIA propagation, since Eq.1 is only a collection of matrix-matrix\r\n cuDNN (Chetlur et al., 2014) and Arm Compute Li- multiplications. This implementation allows us to:\r\n brary (Arm Software). Because these introduce only\r\n marginal numerical error, a network can \ufb01rst be trained \u2022 Learn better \ufb01lters. Building an explicit implemen-\r\n using conventional convolutions before replacing appro- tation of each of the stages involved in the Winograd\r\n priate layers with Winograd, without impacting accuracy. 3\r\n Werefer the interested reader to Barabasz et al. (2018) for an\r\n 2See Section 5.2 in Blahut (2010) for a step-by-step example. analysis on the nature of the errors in Winograd convolutions.\r\n Searching for Winograd-aware Quantized Networks\r\n Cook-Toom \r\n Initialization\r\n BT G AT\r\n Transforming\r\n input patch\r\n d T d B T\r\n B B dB\r\n Qx\r\n T\r\n A yA\r\n Layer Input Hadamard Summation AT y A\r\n across channels y\r\n Qx Qx Layer \r\n Output\r\n g G g GT T Transforming output patch\r\n GgG\r\n Layer Qx\r\n Weights\r\n Transforming weights\r\n \u22a4 \u22a4\r\n Figure 2. Forward pass of Winograd-aware layers. Transformation matrices G, B and A areconstructed via Cook-Toom. If these are\r\n included in the set of model parameters, they would be updated with every batch via back-progation (this is represented with the coloured\r\n \u22a4 \u22a4\r\n arrows going back to matrices G, B and A , carrying the gradients to update the values of each transform). In its default con\ufb01guration,\r\n each intermediate output throughout the pipeline quantized to the same level as the input and weights, this is represented by Qx.\r\n transform exposes the numerical errors introduced in algorithms for each layer, but without modifying the net-\r\n Eq.1 to the learning of the \ufb01lters. This prevents the work\u2019s macro-architecture (e.g. number or order of layers,\r\n accuracy drops shown in Table 1. hyper-parameters, etc). Keeping the macro-architecture\r\n \u2022 Learnthetransforms. Traditionally, matrices G, B\u22a4 \ufb01xedallows us to fairly compare the standard model to its\r\n andA\u22a4are\ufb01xed. Instead,wecantreatthemasanother Winograd-aware counterpart in terms of latency and accu-\r\n set of learnable parameters in the layer. This relaxation racy. We call our framework wiNAS.\r\n leads to much improved performance in quantized net- 4.1 Winograd-awareNASpipeline\r\n works while still maintaining the overall structure of\r\n the Winograd convolution algorithm and its speedups. Introducing latency measurements into the optimization ob-\r\n \u2022 Quantization diversity. Unlike standard convolu- jective requires knowing the shape of the input tensor, i.e.\r\n tion, which does not require intermediate computation, the activations from the previous layer, at each layer of\r\n Winogradconvolution requires at least four of them for the network. We design wiNAS as a variation of Proxy-\r\n \u22a4 \u22a4 lessNAS (Cai et al., 2019), leveraging path sampling while\r\n GgG ,B dB,theHadamardproductandtheoutput performing the search. This technique, enables the alloca-\r\n transformation. Each of these can be quantized to a dif- tion of the entire network on a single GPU by evaluating no\r\n ferent number of bits depending on the bit-width of the morethan two candidate operations at each layer per batch.\r\n input, that of the weights, and the overall complexity\r\n of the problem the network is designed to solve. Similarly to ProxylessNAS, wiNAS formulates the search\r\n as a two-stage process, alternating the update of model\r\n 4 SEARCHING FOR WINOGRAD-AWARE parameters (the weights), where the loss is de\ufb01ned as\r\n NETWORKS\r\n 2\r\n L =Loss +\u03bb kwk (2)\r\n Simultaneously maximizing accuracy and minimizing la- weights CE 0 2\r\n tency with Winograd convolution isn\u2019t trivial. The reason\r\n for this is that large tiles result in low latency, but come at and the update of architecture parameters (the weight as-\r\n the cost of higher numerical error. This presents a good op- signed to each operation on a given layer), where the loss\r\n portunity to jointly optimize network accuracy and latency. introduces the latency metrics is de\ufb01ned as\r\n To this end, we implement a NAS-based approach that\r\n automatically transforms an existing architecture into a 2\r\n L =Loss +\u03bb kak +\u03bb E{latency} (3)\r\n Winograd-aware version. We perform NAS at the micro- arch CE 1 2 2\r\n architecture level by selecting from different convolution\r\n Searching for Winograd-aware Quantized Networks\r\n Input quantized networks, where both weights and activations are\r\n uniformly quantized (including all the intermediate outputs\r\n showninFigure2). Wefollowtheper-layersymmetricquan-\r\n a a a a tization as described in Krishnamoorthi (2018).We repeated\r\n 0 1 2 3 each experiment while enabling the Winograd transforms\r\n G, B\u22a4 and A\u22a4 to be learnt, which we denote using the\r\n im2row F(2x2, 3x3) F(4x4, 3x3) F(6x6, 3x3) additional suf\ufb01x -\ufb02ex.\r\n Winograd-awarelayersdonotrequireanover-parameterized\r\n model to perform well. We also varied the model size by\r\n using a width-multiplier, as used by the MobileNets family,\r\n Output ranging from 0.125 to 1.0, meaning that when the multiplier\r\n is 1.0 the network is the full ResNet-18. This leads to mod-\r\n Figure 3. With wiNAS, each 3\u00d73 convolution in a given architec- els ranging between 215K and 11M parameters. Winograd-\r\n ture is implemented with either im2row or with Winograd convolu- aware layers with learnable transformations marginally in-\r\n tions varying tile size. While the former is lossless and faster than crease (< 0.1%) the model size, since the transforms them-\r\n direct convolution, Winograd offers lower latencies but introduce selves need to be saved for model deployment. We repeated\r\n numerical instability that could ultimately impact in accuracy. the experiment for CIFAR-100(Krizhevskyetal.,2009), but\r\n without varying the depth-multiplier. CIFAR-100 is consid-\r\n where a are the architecture parameters and \u03bb controls erably more challenging that CIFAR-10, as it is comprised\r\n 2 of 100 classes with only 600 images per class.\r\n the impact of latency in the loss. The expected latency,\r\n E{latency}, for a given layer is the weighted combination Additionally, we use an INT8 LeNet (Lecun et al., 1998),\r\n ofthelatencyestimateofeachcandidateoperationwiththeir trained on the MNIST dataset, to evaluate the suitability of\r\n respective probability of being sampled. Intuitively, search- Winograd-aware layers with learnable transforms for 5 \u00d7 5\r\n ing for Winograd convolutions with high \u03bb would result in \ufb01lters. This is a more challenging case than 3 \u00d7 3 \ufb01lters,\r\n 2 because a larger tile tile is required (de\ufb01ned by F(m \u00d7\r\n faster models, potentially at the detriment of accuracy.\r\n Unlike ProxylessNAS, wiNAS focuses on simply selecting m,r\u00d7r)),withlargertransformationmatriceswhichrequire\r\n the optimal convolution algorithm for each of the 3\u00d73 con- the choice of more good polynomial points.\r\n volutional layers. Therefore, the set of candidate operations For experiments on ResNet-18, we replace 2\u00d72-stride con-\r\n for a given conv2d layer contains im2row and Winograd- volution layers with a 2 \u00d7 2 max-pooling layer followed\r\n aware layers in their F2, F4 and F6 con\ufb01gurations. This by a dense 3 \u00d7 3 convolution layer. Altering the network\r\n search space is illustrated in Figure 3. Each candidate oper- in this way is necessary since there is no known equivalent\r\n ation comes with its respective latency, which is a function for strided Winograd convolutions, which remains an open\r\n of the output dimensions and quantization level. research question. This is a common strategy when eval-\r\n uating Winograd (Liu et al., 2018; Choi et al., 2018). We\r\n 5 EXPERIMENTAL SETUP also modi\ufb01ed the number of output channels of the input\r\n layer from 64 to 32. We did this to reduce the memory peak\r\n Weconductvariousexperimentsgroupedinthreecategories. during training. We use the Adam (Kingma & Ba, 2015)\r\n In this section we describe each experiment we conducted. optimizer and train for 120 epochs. Both CIFAR-10/100\r\n WeusedPyTorch(Paszkeetal., 2017) for training and Arm use the same ResNet-18 architecture, differing only in the\r\n ComputeLibrary for deployment. numberofoutputs of the fully connected layer. Results for\r\n other architectures are shown in A.1.\r\n 5.1 Vanilla Winograd-aware networks 5.2 wiNAS:Winograd-awareNAS\r\n Webegin our study of Winograd-aware networks by per-\r\n forming an extensive evaluation on the ResNet-18 (He et al., To evaluate wiNAS, we de\ufb01ne two different sets of can-\r\n 2016) architecture using the CIFAR-10 (Krizhevsky et al., didate operations. These spaces are: wiNASWA and\r\n 2009) dataset. In this experiment we train the network end- wiNASWA-Q,bothallowing each 3\u00d73convolutional layer\r\n to-end using standard convolutions, and F2, F4 and F6 to be implemented with either im2row or each of the Wino-\r\n Winograd convolutions. For each experiment with Wino- grad con\ufb01gurations, F2, F4 or F6. The former uses a \ufb01xed\r\n grad, all layers in the network use the same tile size, except bit-width for all elements in the architecture, while the latter\r\n the last two residual blocks which are kept \ufb01xed to F2.The introduces in the search space candidates of each operation\r\n input convolutional layer uses normal convolutions. We quantized to FP32, INT16 and INT8.\r\n run the experiments for FP32, INT16, INT10 and INT8\r\n Searching for Winograd-aware Quantized Networks\r\n ThehyperparametersusedforwiNASareasfollows: forthe high accuracy. Second, we present the results from our\r\n learning of model parameters we use mini-batch SGD with dense benchmark for winograd convolutions on mobile\r\n Nesterov momentum(Sutskever et al., 2013). In the stage CPUs. Third, we show that wiNAS can jointly optimize a\r\n where we update the architecture parameters we use instead given macro-architecture for accuracy and latency.\r\n Adamwiththe\ufb01rstmomentumscaling,\u03b21,settozero,so\r\n the optimizer only updates paths that have been sampled. 6.1 Vanilla Winograd-aware networks\r\n For both stages we use Cosine Annealing (Loshchilov & Figure 4 (left) shows Winograd-aware networks in FP32\r\n Hutter, 2017) scheduling and a batch size of 64. We perform perform as well as direct convolutions, with both \ufb01xed and\r\n the search for 100 epochs in each search space at different learned (-\ufb02ex) transformation matrices. With quantization\r\n \u03bb2 values ranging from 0.1 to 1e-3. Once the search is com- (all other plots), winograd-aware layers are essential to en-\r\n pleted, we trained the architecture end-to-end with the same able the use of fast Winograd convolutions. This is not\r\n hyperparameters as the rest of winograd-aware networks. possible if switching to Winograd convolutions after train-\r\n 5.3 Winogradconvolutions on mobile CPUs ing, as is commonly done in practice (see Table 1).\r\n For our study, we chose Arm A73 and A53 cores on a Furthermore, we show that learning the Winograd trans-\r\n HuaweiHiKey960developmentboardwiththebig.LITTLE forms (-\ufb02ex) results in 10% and 5% better accuracies for\r\n CPU architecture. These cores are good candidates for F4andF6inINT8scenarios. Wearguethatenablingthis\r\n validating the speedups that are achievable with Winograd relaxation helps in easing the numerical instability inher-\r\n convolutions in today\u2019s off-the-shelf mobile hardware. ent to Winograd convolutions, which is further exacerbated\r\n byquantization. The accuracy of Winograd-aware models\r\n scales linearly with network width, suggesting that these can\r\n CPU Clock L1 L2 be exploited in conjunction with architecture compression\r\n A73 2.4 GHz 64KB 2048 KB techniques such as channel pruning.\r\n A53 1.8 GHz 32KB 512 KB Results from LeNet (5\u00d75 \ufb01lters), provides further evidence\r\n Table 2. Key hardware speci\ufb01cations for the high-performance that larger tiles result in higher numerical error. In Figure 5,\r\n Cortex-A73 and the high-ef\ufb01ciency Cortex-A53 cores found on a weshowthateveninrelatively small datasets like MNIST,\r\n HiKey960developmentboard. keeping the transformations G, B\u22a4 and A\u22a4 \ufb01xed, leads\r\n to poor results as the output tile size is increased. This\r\n While both A73 and A53 are implemented as 16nm quad- difference is almost 47% in the case of F(6 \u00d7 6,5 \u00d7 5)\r\n core CPUs, the former is a high-performance processor and layers, which uses 10 \u00d7 10 tiles.\r\n the latter implements a high-ef\ufb01ciency processor. In Table Winograd-aware layers do not structurally modify the net-\r\n 2wesummarisethemaindifferences between these CPUs. work architecture, since Winograd is just an algorithm to\r\n The memory bandwidth would be the primary factor that perform convolution. We demonstrate it is possible to trans-\r\n ultimately sets the upper limit to the speedup achievable form a pre-trained model with standard convolution into\r\n byWinogradsinceit requires operating in larger tiles than its Winograd-aware counterpart within a few epochs. Con-\r\n direct convolution algorithms such as im2row or im2col. cretely, in Figure 6 we show that an INT8 ResNet-18 F4\r\n In our study, we measured the time taken for 3 \u00d7 3 convolu- can be adapted from a model of the same network that was\r\n tions using im2row, im2col and each of the Winograd con- trained end-to-end with standard convolutions in 20 epochs\r\n \ufb01gurations (F2, F4, F6) when varying output width/height of retraining. This represents a 2.8\u00d7 training time reduction\r\n (from 112\u00d7112downto2\u00d72)andinCh\u2192outCh(from for Winograd-aware models. This is only possible when al-\r\n 3 \u2192 32 to 512 \u2192 512). We performed the benchmark lowingthetransformationmatricestoevolveduringtraining.\r\n in controlled conditions and in single thread mode. Each Adapting FP32 models can be done in a single epoch.\r\n combination was run \ufb01ve times with \ufb01ve seconds delay in Webelieve both F4 and F6 performance could be raised\r\n between to prevent thermal throttling. We implemented with alternative quantization implementations, closing the\r\n Winograd convolutions using GEMMs (Maji et al. (2019)), accuracy gap with F2 and direct convolutions.\r\n and performed the same experiment separately on A73 and\r\n A53forbothFP32andINT8. INT16measurementsarenot 6.2 ImpactofWinogradonLatency\r\n currently supported in Arm Compute Library.\r\n The speedups associated to with use of Winograd convo-\r\n 6 EXPERIMENTAL RESULTS lutions often only account for the point-wise stage while\r\n assuming negligible costs for the input, weights and output\r\n Theresults of this work are arranged as three subsections. transformations. Furthermore, these also assume that the\r\n First, we show that winograd-aware networks can achieve larger the input patch, d, the larger the speedup compared to\r\n Searching for Winograd-aware Quantized Networks\r\n im2row F2 F2-flex F4 F4-flex F6 F6-flex\r\n 95 ResNet-18 32-bit 95 ResNet-18 16-bit 95 ResNet-18 10-bit 95 ResNet-18 8-bit\r\n 94\r\n 93 90 90 90\r\n 92\r\n 91 85 85 85\r\n 90\r\n Accuracy (%)89 Accuracy (%)80 Accuracy (%)80 Accuracy (%)80\r\n 88\r\n 87 75 75 75\r\n 86\r\n 850.1250.25 0.5 0.75 1.0 700.1250.25 0.5 0.75 1.0 700.1250.25 0.5 0.75 1.0 700.1250.25 0.5 0.75 1.0\r\n width multiplier width multiplier width multiplier width multiplier\r\n Figure 4. Performance of a winograd-aware ResNet-18 at different bit-widths and trained with different Winograd con\ufb01gurations. We\r\n showhowwinograd-awarelayers scale with network\u2019s width. We can observe that in quantized networks, models that learn the Winograd\r\n transforms (-\ufb02ex con\ufb01gurations), strictly outperforms those models that keep them \ufb01xed with the values obtained via Cook-Toom.\r\n 100 LeNet 8-bit ResNet-18 8-bit\r\n 95 92.66%\r\n 98 85\r\n 96\r\n im2row 75\r\n 94 F2 im2row\r\n 92 F2-flex 65 F4\r\n F4 F4-flex\r\n Validation Accuracy90 F4-flex Validation Accuracy55 F4 (adapted)\r\n F6 F4-flex (adapted)\r\n 88 F6-flex 45\r\n 0 5 10 15 20 25 30 0 20 40 epoch60 80 100 120\r\n epoch\r\n Figure 5. Performance of INT8 LeNet on MNIST using standard Figure 6. Transforming a standard model (trained with default con-\r\n convolutions (im2row) or winograd-aware layers. Letting the trans- volutions) to its Winograd-aware counterpart can be done in very\r\n formations to evolve during training (-\ufb02ex) always results in better few epochs of retraining. We found this technique works best if\r\n models. F4 and F6 con\ufb01gurations (not shown) reach an accu- the Winograd transformations are learnt during retraining (-\ufb02ex),\r\n racy of 73% and 51%, respectively. All con\ufb01gurations reach otherwise adaptation becomes much more challenging.\r\n 99.25%\u00b10.1%infullprecision.\r\n of this, this \ufb01rst layer accounts for a marginal portion of the\r\n normal convolutions. However, although these assumptions total latency of the model, often below 1ms.\r\n are true for large tensors, they are not necessarily true when Optimal m is a function of input width and height. For\r\n working with tensors of the sizes often found in CNNs for an input with suf\ufb01cient number of channels, e.g. 32 chan-\r\n image classi\ufb01cation or object detection. nels and up, we observe a consistent pattern alternating\r\n Figure 7 shows a small portion of the obtained latency mea- between F4 and F6 as the channel dimension of the output\r\n surements for our benchmark in FP32. An almost identical increase. This alternation comes as a result of the impos-\r\n pattern appears when using 8-bit arithmetic. In Figure 8 sibility of subdividing the input into an integer number of\r\n weshowthespeedupsthatWinogradconvolutions offer at (m+r\u22121)\u00d7(m+r\u22121)patches,andthereforehavingto\r\n different layers of a ResNet-18 network. Our observations waste calculations when operating around the matrix edges.\r\n can be summarized in three points: This pattern is invariant to different inCh \u2192 outCh con\ufb01g-\r\n Input layers do not bene\ufb01t from Winograd. This is pri- urations and fades away as input dimensions exceed 40\u00d740,\r\n marily because the matrices in the element-wise GEMM where F6 consistently becomes the fastest.\r\n stage are not large enough to compensate for the costs of Winograd transforms are costly. Excluding inputs with\r\n transforming the input and output patches (see Figure 5.3 very few channels, the cost of performing the transforma-\r\n and 8). They represent a signi\ufb01cant portion (up to 65% and tions to/from the Winograd domain can exceed 25% of the\r\n 75%respectively on the A73 and A53) of the total costs for overall costs. These costs become negligible as the input\r\n convolving a RGB 32\u00d732 input expanded to 32 channels. width and height decrease, but the rate at which this hap-\r\n Similar ratios can be observed for other input sizes. In spite pens also depends on the hardware. Our Winograd-aware\r\n Searching for Winograd-aware Quantized Networks\r\n inCh --> outCh\r\n 3 --> 32 32 --> 64 128 --> 192 192 --> 256 256-->512\r\n outW im2row F2 F4 F6 im2row F2 F4 F6 im2row F2 F4 F6 im2row F2 F4 F6 im2row F2 F4 F6\r\n 2 0.007 0.008 0.016 0.029 0.070 0.043 0.082 0.167 0.659 0.407 1.219 2.196 1.463 1.082 2.378 4.407 3.912 2.932 6.619 11.853\r\n 4 0.011 0.029 0.016 0.030 0.154 0.078 0.081 0.167 1.642 0.802 1.170 2.195 2.884 1.731 2.502 4.486 7.450 4.962 6.588 11.947\r\n 6 0.021 0.053 0.065 0.029 0.328 0.199 0.174 0.165 4.137 2.229 2.040 2.148 6.780 4.559 4.135 4.327 17.450 13.858 11.452 11.919\r\n 8 0.031 0.059 0.064 0.133 0.519 0.280 0.175 0.408 5.306 2.993 2.004 3.899 10.932 6.145 4.167 7.907 28.238 14.930 11.499 21.241\r\n 10 0.058 0.101 0.119 0.144 0.910 0.475 0.482 0.412 9.466 5.054 5.321 3.973 17.808 10.198 10.318 7.904 44.656 27.597 32.685 21.437\r\n 12 0.066 0.133 0.129 0.132 1.208 0.621 0.475 0.424 11.625 6.601 5.382 3.971 24.196 12.995 10.272 7.955 61.236 35.702 32.164 21.478\r\n 14 0.087 0.186 0.154 0.267 1.610 0.868 0.695 1.043 16.177 9.277 7.498 9.846 33.702 18.154 14.220 19.082 85.809 48.590 34.306 60.003\r\n 16 0.111 0.235 0.153 0.283 2.592 1.191 0.723 1.051 20.845 12.158 7.551 10.002 42.362 23.147 14.310 19.263 109.943 57.083 34.190 60.504\r\n 18 0.169 0.281 0.263 0.281 3.315 1.379 1.133 1.031 26.785 15.125 12.159 9.961 55.085 29.292 23.178 19.476 142.460 75.505 63.799 60.987\r\n 20 0.184 0.325 0.249 0.400 3.416 1.695 1.131 1.728 32.851 18.450 12.115 15.108 67.300 35.276 23.274 27.723 173.488 90.041 65.349 67.923\r\n 22 0.210 0.398 0.331 0.410 4.164 2.070 1.506 1.690 40.245 22.207 16.010 15.114 82.028 43.166 30.697 27.781 213.326 110.160 82.434 67.228\r\n 24 0.247 0.452 0.324 0.409 4.783 2.453 1.498 1.729 47.961 26.600 16.126 15.035 97.706 51.064 30.954 27.923 251.771 125.604 83.167 67.047\r\n Figure 7. Latencies (in milliseconds) of convolving increasingly larger input tensors in the width/height dimensions (y axis) and in depth\r\n (x axis). We compare the time needed for im2row and each of the Winograd con\ufb01gurations with 32-bit arithmetic on a Cortex-A73. We\r\n showthat (1) im2row is the consistently the optimal algorithm for the input layer to a network, (2) the choice between F2, F4 and F6\r\n should be done based on the output\u2019s width/height and, (3) this choice should not generally be altered based on inCh \u2192 outCh.\r\n pipeline formulation doesn\u2019t impose any constrains on how 3.0\r\n the transforms are learnt. This results in dense transforms im2row\r\n (as opposed to the default transforms witch contain zeros) 2.5 im2col\r\n F2\r\n andtherefore applying them require additional compute. Ta- 2.0 F4\r\n ble 3 includes this latency overhead in models making use F6\r\n Ratio1.5\r\n of learned transforms. In Appendix A.2 we provide more\r\n details on how dense transforms impact overall latency. 1.0\r\n OnA53,thespeedupsfromFP32Winogradconvolutions 0.5\r\n are smaller than On A73. We argue this comes as a results 0.0 32x32 16x16 8x8\r\n of the differences in the memory subsystem, limiting the inCh:3 outCh:32 inCh:128 outCh:128 inCh:256 outCh:256\r\n lower-end CPU to ef\ufb01ciently operate with larger tensors. im2row\r\n These speedups grow signi\ufb01cantly when leveraging INT8 3.5 im2col\r\n arithmetic, made possible by winograd-aware training. Con- 3.0 F2\r\n cretely, INT8 Winograd increases the speedup on the A53 2.5 F4\r\n F6\r\n by a factor of almost 1.5\u00d7 compared to Winograd in FP32, Ratio2.0\r\n as shown in WAF4 con\ufb01gurations in Table 3 \u2013 at the cost of 1.5\r\n 1.1%accuracy in CIFAR-10. In the case of the more chal- 1.0\r\n lenging CIFAR-100 dataset, the drop in accuracy is more 0.5\r\n severe. However, our WAF2 layers offer attractive speedups 0.0 32x32 16x16 8x8\r\n for INT8 with no drop in accuracy. We rely on wiNAS to inCh:3 outCh:32 inCh:128 outCh:128 inCh:256 outCh:256\r\n minimize this degradation with small impact on latency. Figure 8. Latencies (normalized w.r.t im2row) to execute different\r\n layers of the ResNet-18 network measured in A73 (above) and\r\n 6.3 wiNASNetworks A53(below). ForWinograds,solidcolourbarregionsrepresentthe\r\n element-wise GEMM stage, below and above it are respectively\r\n Choosing the convolution algorithm that minimizes overall the input and output transformation costs. Measured for FP32 and\r\n network latency can be easily done by looking at the bench- with default Winograd transforms.\r\n mark results. However, since the accuracy of Winograd\r\n convolutions degrade rapidly in reduced precision networks, increase in the worst case, an INT8 model on the A53 for\r\n selecting the fastest algorithm for each layer without sacri- CIFAR-100. These models resulted in similar accuracies\r\n \ufb01cing accuracy, is not straight forward. in FP32 and reached 0.32% and 1.1% higher accuracies\r\n WhenusingwiNASWA,values of \u03bb2 larger than 0.05 con- in INT8 for CIFAR-10 and CIFAR-100 respectively. De-\r\n sistently result in models with the same layer con\ufb01guration spite CIFAR-100modelssacri\ufb01cingmorelatencyinorderto\r\n recover accuracy, they remained faster than WA at INT8.\r\n as those in WAF4 (described in section 5.1). When lowering F2\r\n the impact of latency in Eq. 3 loss function, we observed Whenintroducing quantization in the search performed by\r\n several F4 Winograd-aware layers were replaced with ei- wiNAS , the accuracy gap is almost closed for CIFAR-\r\n ther im2row or F2, at the cost of less than 9 ms latency WA-Q\r\n 10andfurther reduced for CIFAR-100. This comes primar-\r\n Searching for Winograd-aware Quantized Networks\r\n Conv. Bits Accuracy (%) Cortex-A53 Cotex-A73\r\n Type act. / param. CIFAR-10 CIFAR-100 Latency (ms) Speedup Latency (ms) Speedup\r\n im2row 93.16 74.62 118 - 85 -\r\n im2col 93.16 74.62 156 0.76\u00d7 102 0.83\u00d7\r\n WF2 93.16 74.60 126 0.94\u00d7 56 1.52\u00d7\r\n WF4 32/32 93.14 74.53 97 1.22\u00d7 46 1.85\u00d7\r\n WAF2* 93.46 74.69 126 0.94\u00d7 56 1.52\u00d7\r\n \u2020 \u2020\r\n WA 93.54 74.98 122 0.92\u00d7 54 1.58\u00d7\r\n F4\r\n \u2020 \u2020\r\n wiNAS 93.35 74.71 123 0.96\u00d7 56 1.52\u00d7\r\n WA\r\n im2row 93.20 74.11 117 1.01\u00d7 54 1.57\u00d7\r\n im2col 93.20 74.11 124 0.95\u00d7 59 1.45\u00d7\r\n WAF2* 8 / 8 93.72 73.71 91 1.30\u00d7 38 2.24\u00d7\r\n \u2020 \u2020\r\n WAF4 92.46 72.38 82 1.44\u00d7 35 2.43\u00d7\r\n \u2020 \u2020 \u2020 \u2020\r\n wiNASWA 92.71 73.42 88 / 91 1.34\u00d7/1.30\u00d7 35 / 36 2.43\u00d7/2.36\u00d7\r\n \u2020 \u2020 \u2020 \u2020\r\n wiNASWA-Q auto 92.89 73.88 74 / 97 1.60\u00d7/1.22\u00d7 32 / 43 2.66\u00d7/1.98\u00d7\r\n Table 3. Performance in terms of accuracy and latency (ms) of ResNet-18 when convolutions are implemented with different algorithms\r\n and for different quantization levels. We show that Winograd-aware (WA ) layers combine the speedups of Winograd convolutions with\r\n F2/4\r\n those of INT8 arithmetic, with little to no accuracy loss in some cases. This is not possible with existing Winograd (WF2/4) formulations.\r\n Latency is measured on Arm Cortex-A73 and A53 cores. For the last two rows, wiNAS found different optimizations for each dataset. We\r\n showlatencies for CIFAR-10 on the left and CIFAR-100 on the right. Speedups are shown against im2row in FP32. (*) With default\r\n Winograd transforms. (\u2020) Includes worst case latency increase due to be using learned transform, which are often dense.\r\n ily as a result of relying on higher bit-widths for the \ufb01rst Winograd could alleviate some of the degradation that we\r\n layers in the network. In both cases, we maintain attractive observed with increased tile size.\r\n speedups compared to im2row and Winograd convolutions In this work we focused on mobile CPUs, but we expect\r\n in FP32, specially on the A73. All the ResNet-18 architec- these bene\ufb01ts to be also applicable to GPUs. However, to\r\n tures optimized with wiNAS are described in A.3. further maximize the speedups that Winograd-aware layers\r\n 7 DISCUSSION for quantized CNNs offer, a custom hardware implementa-\r\n tion in the form of an accelerator would be preferable.\r\n In this section we present some of the challenges of training 8 CONCLUSION\r\n Winograd-aware networks and propose lines of future work.\r\n Adirect implementation of Eq. 1 requires saving the inter- Running CNN-based applications that require standard con-\r\n mediate outputs of each matrix-matrix multiplication, since volutional layers is challenging in compute-constrained de-\r\n these are needed for back-propagation. This results in high vices such as mobile CPUs. This paper presents Winograd-\r\n memory usage. In this work we had to rely on gradient aware layers as the building block to combine the bene\ufb01ts\r\n checkpointing (Chen et al., 2016) to lower the memory peak of quantized networks and fast Winograd convolutions. We\r\n duringtraining, at the cost of additional computation. We be- studiedWinograd-awarelayerswithdifferenttilesizes,three\r\n lieve a native CUDAimplementationoftheWinograd-aware quantization levels and on three popular datasets. We found\r\n layers with better memory reuse would ease this problem. that allowing the transformation matrices to evolve during\r\n Learning larger models (with width multipliers 0.75 and training resulted in signi\ufb01cantly better models. With wiNAS\r\n 1.0) proved challenging for F4 and F6 when introducing weleveraged Winograd-aware layers and latency metrics\r\n quantization. Using other types of quantization would likely from off-the-shelf mobile CPUs and found architectures\r\n help. In particular per-channel af\ufb01ne quantization, as in that helped minize the numerical instability of Winograd. A\r\n Jacob et al. (2018). Also, enabling different bit-widths Winograd-aware ResNet-18 quantized to INT8 offers up to\r\n throughout Eq. 1 could help mitigate the accuracy drop. 1.32\u00d7 faster inference for only a marginal accuracy drop\r\n compared to existing Winograd implementations, which are\r\n It is known that bad polynomial points for constructing G, limited to FP32. This network is also 1.54\u00d7 faster than an\r\n \u22a4 \u22a4\r\n B andA introducesigni\ufb01cantdeviations in the result of optimized im2row implementation using INT8 arithmetic.\r\n computing Winograd convolutions compared to that of nor-\r\n malconvolutions. We observed that good starting points are REFERENCES\r\n also important even when learning the Winograd transforma-\r\n tions. Polynomial points speci\ufb01cally tailored for quantized Abtahi, T., Shea, C., Kulkarni, A., and Mohsenin, T. Accelerating\r\n Searching for Winograd-aware Quantized Networks\r\n convolutional neural network with fft on embedded hardware. Choi, Y., El-Khamy, M., and Lee, J. Compression of deep convo-\r\n IEEETransactions on Very Large Scale Integration (VLSI) Sys- lutional neural networks under joint sparsity constraints, 2018.\r\n tems, 26(9):1737\u20131749, Sep. 2018. ISSN 1063-8210. doi:\r\n 10.1109/TVLSI.2018.2825145. Chowdhery, A., Warden, P., Shlens, J., Howard, A., and Rhodes,\r\n Alizadeh, M., Fernandez-Marques, J., Lane, N. D., and Gal, Y. A R. Visual wake words dataset, 2019.\r\n systematic study of binary neural networks\u2019 optimisation. In Cong, J. and Xiao, B. Minimizing computation in convolutional\r\n International Conference on Learning Representations, 2019. neural networks. In Wermter, S., Weber, C., Duch, W., Honkela,\r\n Anderson, A. and Gregg, D. Optimal dnn primitive selection with T., Koprinkova-Hristova, P., Magg, S., Palm, G., and Villa, A.\r\n partitioned boolean quadratic programming. Proceedings of E. P. (eds.), Arti\ufb01cial Neural Networks and Machine Learning \u2013\r\n the 2018 International Symposium on Code Generation and ICANN2014,pp.281\u2013290,Cham,2014.SpringerInternational\r\n Optimization - CGO 2018, 2018. doi: 10.1145/3168805. Publishing. ISBN 978-3-319-11179-7.\r\n Arm Software. Arm compute library. https:// Courbariaux, M., Bengio, Y., and David, J.-P. Binaryconnect:\r\n developer.arm.com/ip-products/processors/ Training deep neural networks with binary weights during prop-\r\n machine-learning/compute-library. Accessed: agations. In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama,\r\n 2019-08-01. M., and Garnett, R. (eds.), Advances in Neural Information\r\n Processing Systems 28, pp. 3123\u20133131. Curran Associates, Inc.,\r\n Barabasz, B. and Gregg, D. Winograd convolution for dnns: Be- 2015.\r\n yond linear polynomials, 2019. Fedorov, I., Adams, R. P., Mattina, M., and Whatmough, P. N.\r\n Barabasz, B., Anderson, A., and Gregg, D. Improving accuracy of SpArSe: Sparse Architecture Search for CNNs on Resource-\r\n winograd convolution for dnns. CoRR, abs/1803.10986, 2018. Constrained Microcontrollers. In Proceedings of the Neural\r\n Bender, G., Kindermans, P.-J., Zoph, B., Vasudevan, V., and Le, Information Processing Systems (NeurIPS) Conference 2019,\r\n Q. Understanding and simplifying one-shot architecture search. 2019.\r\n In Dy, J. and Krause, A. (eds.), Proceedings of the 35th Inter- Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann,\r\n national Conference on Machine Learning, volume 80 of Pro- F. A., and Brendel, W. Imagenet-trained CNNs are biased to-\r\n ceedings of Machine Learning Research, pp. 550\u2013559, Stock- wards texture; increasing shape bias improves accuracy and\r\n \u00a8\r\n holmsmassan, Stockholm Sweden, 10\u201315 Jul 2018. PMLR. robustness. In International Conference on Learning Represen-\r\n Blahut, R. E. Fast Algorithms for Signal Processing. Cambridge tations, 2019.\r\n University Press, 2010. doi: 10.1017/CBO9780511760921. Gong, R., Liu, X., Jiang, S., Li, T., Hu, P., Lin, J., Yu, F., and Yan,\r\n Brendel, W. and Bethge, M. Approximating CNNs with bag-of- J. Differentiable soft quantization: Bridging full-precision and\r\n local-features models works surprisingly well on imagenet. In low-bit neural networks, 2019.\r\n International Conference on Learning Representations, 2019. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for\r\n Brock, A., Lim, T., Ritchie, J., and Weston, N. SMASH: One-shot image recognition. 2016 IEEE Conference on Computer Vision\r\n modelarchitecture search through hypernetworks. In Interna- and Pattern Recognition (CVPR), Jun 2016. doi: 10.1109/cvpr.\r\n tional Conference on Learning Representations, 2018. 2016.90.\r\n Brock, A., Donahue, J., and Simonyan, K. Large scale GAN train- He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., and Han, S. Amc:\r\n ing for high \ufb01delity natural image synthesis. In International Automl for model compression and acceleration on mobile\r\n Conference on Learning Representations, 2019. devices. Lecture Notes in Computer Science, pp. 815\u2013832,\r\n Cai, H., Zhu, L., and Han, S. ProxylessNAS: Direct neural ar- 2018.\r\n chitecture search on target task and hardware. In International \u00b4\r\n Conference on Learning Representations, 2019. Hernandez-Lobato, J. M., Gelbart, M. A., Reagen, B., Adolf, R.,\r\n \u00b4\r\n Hernandez-Lobato, D., Whatmough, P. N., Brooks, D., Wei,\r\n Carmichael, Z., Langroudi, H. F., Khazanov, C., Lillie, J., G.-Y., and Adams, R. P. Designing neural network hardware\r\n Gustafson, J. L., and Kudithipudi, D. Deep positron: A deep accelerators with decoupled objective evaluations. In NIPS\r\n neural networkusingthepositnumbersystem. 2019Design,Au- workshop on Bayesian Optimization, 2016.\r\n tomationandTestinEuropeConferenceandExhibition(DATE), Highlander, T. and Rodriguez, A. Very ef\ufb01cient training of con-\r\n Mar2019. doi: 10.23919/date.2019.8715262. volutional neural networks using fast fourier transform and\r\n Chen, T., Xu, B., Zhang, C., and Guestrin, C. Training deep nets overlap-and-add. Procedings of the British Machine Vision\r\n with sublinear memory cost, 2016. Conference 2015, 2015. doi: 10.5244/c.29.160.\r\n Chen, Y., Fan, H., Xu, B., Yan, Z., Kalantidis, Y., Rohrbach, M., Horowitz, M. 1.1 computing\u2019s energy problem (and what we can\r\n Yan, S., and Feng, J. Drop an octave: Reducing spatial redun- do about it). In 2014 IEEE International Solid-State Circuits\r\n dancyinconvolutionalneuralnetworkswithoctaveconvolution, Conference Digest of Technical Papers (ISSCC), pp. 10\u201314, Feb\r\n 2019. 2014. doi: 10.1109/ISSCC.2014.6757323.\r\n Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J., Howard, A., Sandler, M., Chu, G., Chen, L.-C., Chen, B., Tan,\r\n Catanzaro, B., and Shelhamer, E. cudnn: Ef\ufb01cient primitives M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., Le, Q. V., and\r\n for deep learning. CoRR, abs/1410.0759, 2014. Adam,H. Searching for mobilenetv3, 2019.\r\n Searching for Winograd-aware Quantized Networks\r\n Howard,A.G.,Zhu,M.,Chen,B.,Kalenichenko,D.,Wang,W., Lin, X., Zhao, C., and Pan, W. Towards accurate binary convo-\r\n Weyand,T., Andreetto, M., and Adam, H. Mobilenets: Ef\ufb01cient lutional neural network. In Advances in Neural Information\r\n convolutional neural networks for mobile vision applications, Processing Systems, pp. 345\u2013353, 2017.\r\n 2017. Liu, H., Simonyan, K., and Yang, Y. DARTS: Differentiable\r\n Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., architecture search. In International Conference on Learning\r\n and Keutzer, K. Squeezenet: Alexnet-level accuracy with 50x Representations, 2019.\r\n fewer parameters and \u00a10.5mb model size, 2016. Liu, X., Pool, J., Han, S., and Dally, W. J. Ef\ufb01cient sparse-\r\n Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., winograd convolutional neural networks. In International Con-\r\n Adam,H.,andKalenichenko, D. Quantization and training of ference on Learning Representations, 2018.\r\n neural networks for ef\ufb01cient integer-arithmetic-only inference. Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent\r\n 2018IEEE/CVFConferenceonComputerVisionandPattern with warm restarts. In International Conference on Learning\r\n Recognition, Jun 2018. doi: 10.1109/cvpr.2018.00286. Representations (ICLR) 2017 Conference Track, April 2017.\r\n Kalamkar, D., Mudigere, D., Mellempudi, N., Das, D., Banerjee, Maji, P., Mundy, A., Dasika, G., Beu, J., Mattina, M., and Mullins,\r\n K., Avancha, S., Vooturi, D. T., Jammalamadaka, N., Huang, R. Ef\ufb01cient winograd or cook-toom convolution kernel imple-\r\n J., Yuen, H., Yang, J., Park, J., Heinecke, A., Georganas, E., mentation on widely used mobile cpus, 2019.\r\n Srinivasan, S., Kundu, A., Smelyanskiy, M., Kaul, B., and\r\n Dubey, P. A study of b\ufb02oat16 for deep learning training, 2019. Mathieu, M., Henaff, M., and LeCun, Y. Fast training of convolu-\r\n Kingma,D.P.andBa,J. Adam: Amethodforstochasticoptimiza- tional networks through ffts, 2013.\r\n tion. In International Conference on Learning Representations Meng, L. and Brothers, J. Ef\ufb01cient winograd convolution via\r\n (ICLR), 2015. integer arithmetic. CoRR, abs/1901.01965, 2019.\r\n Krishnamoorthi, R. Quantizing deep convolutional networks for Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito,\r\n ef\ufb01cient inference: A whitepaper. CoRR, abs/1806.08342, 2018. Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic\r\n differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.\r\n Krizhevsky, A. et al. Learning multiple layers of features from tiny\r\n images. Technical report, Citeseer, 2009. Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. Xnor-\r\n net: Imagenet classi\ufb01cation using binary convolutional neural\r\n L. Toom, A. The complexity of a scheme of functional elements networks. CoRR, abs/1603.05279, 2016.\r\n realizing the multiplication of integers. Doklady Akademii Nauk Real, E., Aggarwal, A., Huang, Y., and Le, Q. V. Regularized\r\n SSSR, 3, 01 1963. evolution for image classi\ufb01er architecture search. Proceedings\r\n Lane, N. D. and Warden, P. The deep (learning) transformation of the AAAI Conference on Arti\ufb01cial Intelligence, 33(01):4780\u2013\r\n of mobile and embedded computing. Computer, 51(5):12\u201316, 4789, Jul. 2019. doi: 10.1609/aaai.v33i01.33014780.\r\n May2018. ISSN0018-9162. doi: 10.1109/MC.2018.2381129. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen,\r\n Lavin, A. and Gray, S. Fast algorithms for convolutional neural L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks.\r\n networks. 2016 IEEE Conference on Computer Vision and 2018IEEE/CVFConferenceonComputerVisionandPattern\r\n Pattern Recognition (CVPR), Jun 2016. doi: 10.1109/cvpr.2016. Recognition, Jun 2018. doi: 10.1109/cvpr.2018.00474.\r\n 435. Simonyan,K.andZisserman,A.Verydeepconvolutionalnetworks\r\n Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based for large-scale image recognition. In International Conference\r\n learning applied to document recognition. In Proceedings of onLearning Representations, 2015.\r\n the IEEE, pp. 2278\u20132324, 1998. Stamoulis, D., Ding, R., Wang, D., Lymberopoulos, D., Priyantha,\r\n Lee, R., Venieris, S. I., Dudziak, L., Bhattacharya, S., and Lane, B., Liu, J., and Marculescu, D. Single-path mobile automl:\r\n N. D. Mobisr: Ef\ufb01cient on-device super-resolution through Ef\ufb01cient convnet design and nas hyperparameter optimization,\r\n heterogeneous mobile processors. In The 25th Annual In- 2019.\r\n ternational Conference on Mobile Computing and Network- Strassen, V. Gaussian elimination is not optimal. Numer. Math.,\r\n ing, MobiCom \u201919, New York, NY, USA, 2019. Associa- 13(4):354\u2013356, August 1969. ISSN 0029-599X. doi: 10.1007/\r\n tion for Computing Machinery. ISBN 9781450361699. doi: BF02165411.\r\n 10.1145/3300061.3345455.\r\n Sutskever, I., Martens, J., Dahl, G., and Hinton, G. On the im-\r\n Li, F., Zhang, B., and Liu, B. Ternary weight networks, 2016. portance of initialization and momentum in deep learning. In\r\n Dasgupta, S. and McAllester, D. (eds.), Proceedings of the 30th\r\n Li, H., Bhargav, M., Whatmough, P. N., and Philip Wong, H. . International Conference on Machine Learning, volume 28 of\r\n On-ChipMemoryTechnologyDesignSpaceExplorationsfor Proceedings of Machine Learning Research, pp. 1139\u20131147,\r\n Mobile Deep Neural Network Accelerators. In 2019 56th Atlanta, Georgia, USA, 17\u201319 Jun 2013. PMLR.\r\n ACM/IEEE Design Automation Conference (DAC), pp. 1\u20136,\r\n June 2019. Sze, V., Chen, Y.-H., Yang, T.-J., and Emer, J. S. Ef\ufb01cient process-\r\n ing of deep neural networks: A tutorial and survey. Proceedings\r\n Liberis, E. and Lane, N. D. Neural networks on microcontrollers: of the IEEE, 105(12):2295\u20132329, Dec 2017. ISSN 1558-2256.\r\n saving memory at inference via operator reordering, 2019. doi: 10.1109/jproc.2017.2761740.\r\n Searching for Winograd-aware Quantized Networks\r\n Takikawa, T., Acuna, D., Jampani, V., and Fidler, S. Gated-scnn: Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., and Fu, Y. Im-\r\n Gated shape cnns for semantic segmentation. arXiv preprint age super-resolution using very deep residual channel attention\r\n arXiv:1907.05740, 2019. networks. CoRR, abs/1807.02758, 2018.\r\n Tan, M. and Le, Q. V. Ef\ufb01cientnet: Rethinking model scaling for Zoph, B. and Le, Q. V. Neural architecture search with rein-\r\n convolutional neural networks, 2019. forcement learning. In International Conference on Learning\r\n Representations, 2017.\r\n Tan, M., Chen, B., Pang, R., Vasudevan, V., and Le, Q. V. Mnasnet:\r\n Platform-aware neural architecture search for mobile. In CVPR,\r\n 2019.\r\n Tschannen, M., Khanna, A., and Anandkumar, A. StrassenNets:\r\n Deeplearningwithamultiplicationbudget. InDy,J.andKrause,\r\n A. (eds.), Proceedings of the 35th International Conference\r\n on Machine Learning, volume 80 of Proceedings of Machine\r\n \u00a8\r\n Learning Research, pp. 4985\u20134994, Stockholmsmassan, Stock-\r\n holmSweden,10\u201315Jul2018.PMLR.\r\n Vincent, K., Stephano, K., Frumkin, M., Ginsburg, B., and J.,\r\n D. Onimproving the numerical stability of winograd convolu-\r\n tions. In International Conference on Learning Representations\r\n (Workshop track), 2017.\r\n Wan,D.,Shen,F., Liu, L., Shao, L., Qin, J., and Shen, H. T. Tbn:\r\n Convolutional neural network with ternary inputs and binary\r\n weights. In ECCV, 2018.\r\n Wang, K., Liu, Z., Lin, Y., Lin, J., and Han, S. Haq: Hardware-\r\n aware automated quantization with mixed precision. In The\r\n IEEEConference on Computer Vision and Pattern Recognition\r\n (CVPR), June 2019.\r\n Whatmough, P. N., Lee, S. K., Brooks, D., and Wei, G. DNN\r\n Engine: A 28-nm Timing-Error Tolerant Sparse Deep Neural\r\n NetworkProcessorforIoTApplications. IEEEJournalofSolid-\r\n State Circuits, 53(9):2722\u20132731, Sep. 2018. ISSN 1558-173X.\r\n doi: 10.1109/JSSC.2018.2841824.\r\n Whatmough, P. N., Lee, S. K., Donato, M., Hsueh, H., Xi, S.,\r\n Gupta, U., Pentecost, L., Ko, G. G., Brooks, D., and Wei,\r\n G. A 16nm 25mm2 SoC with a 54.5x Flexibility-Ef\ufb01ciency\r\n RangefromDual-CoreArmCortex-A53toeFPGAandCache-\r\n Coherent Accelerators. In 2019 Symposium on VLSI Circuits,\r\n pp. C34\u2013C35, June 2019. doi: 10.23919/VLSIC.2019.8778002.\r\n Whatmough,P.N.,Zhou,C.,Hansen,P., Venkataramanaiah, S. K.,\r\n Seo, J.-S., and Mattina, M. FixyNN: Ef\ufb01cient Hardware for\r\n Mobile Computer Vision via Transfer Learning, 2019.\r\n Winograd, S. Arithmetic Complexity of Computations. Society\r\n for Industrial and Applied Mathematics, 1980. doi: 10.1137/1.\r\n 9781611970364.\r\n Winograd, S. Signal processing and complexity of computation.\r\n In ICASSP \u201980. IEEE International Conference on Acoustics,\r\n Speech, and Signal Processing, volume 5, pp. 94\u2013101, April\r\n 1980. doi: 10.1109/ICASSP.1980.1171044.\r\n Xiang, X., Qian, Y., and Yu, K. Binary deep neural networks for\r\n speech recognition. In Proc. Interspeech 2017, pp. 533\u2013537,\r\n 2017. doi: 10.21437/Interspeech.2017-1343.\r\n \u00b4\r\n Xie, S., Girshick, R. B., Dollar, P., Tu, Z., and He, K. Aggre-\r\n gated residual transformations for deep neural networks. 2017\r\n IEEEConference on Computer Vision and Pattern Recognition\r\n (CVPR), pp. 5987\u20135995, 2017.\r\n Searching for Winograd-aware Quantized Networks\r\n A APPENDIX Conv. Bits WA Accuracy (%)\r\n A.1 Winograd-awarelayersforotherarchitectures type act. / param. trans. CIFAR-10 CIFAR-100\r\n im2row - 93.17 74.54\r\n WA static 93.19 74.66\r\n Theresults of our study of Winograd-aware networks pre- F2\r\n WA 32/32 \ufb02ex 93.08 74.58\r\n sented Section 6 showed multiple con\ufb01gurations of the F2\r\n WA static 93.24 74.47\r\n ResNet-18 architecture at different width-multipliers, bit- F4\r\n WA \ufb02ex 93.15 74.62\r\n widths, quantization levels and convolution algorithms. F4\r\n Here, we present a similar analysis for two other popu- im2row - 93.40 74.89\r\n WA static 92.93 75.32\r\n lar architectures for image classi\ufb01cation. We limit our F2\r\n WA 8 / 8 \ufb02ex 93.11 75.80\r\n study to the full models (i.e. mult=1.0) We show results F2\r\n WA static 76.73 51.20\r\n F4\r\n WA \ufb02ex 93.29 75.35\r\n for SqueezeNet (Iandola et al., 2016) in Table 4 and for F4\r\n ResNeXt(Xieetal., 2017) in Table 5. These results align\r\n with what was observed for ResNet-18: In the presence of Table 5. Comparison between standard convolutions (im2row) and\r\n quantization, learning the Winograd transformations (\ufb02ex Winograd-awarelayersforResNeXt 20(8\u00d716). WithINT8quanti-\r\n con\ufb01gurations) resulted in superior performance than using zation and using the default transformation matrices (static), larger\r\n the default (static) transformations. All experiments used tile sizes (F4) introduce substantial numerical error and result in a\r\n the same hyper-parameters as described in Section 5. sever accuracy drop. This drop in accuracy is signi\ufb01cantly reduced\r\n if the transformation matrices are learnt (\ufb02ex).\r\n Conv. Bits WA Accuracy (%)\r\n Type act. / param. trans. CIFAR-10 CIFAR-100\r\n im2row - 91.13 69.06 as these transforms are adjusted for larger input patches. For\r\n WA static 91.31 69.42 example, for the default transforms F4 these ratios are 22%,\r\n F2 22%and25%. Forimplementationsofmatrix-matrix mul-\r\n WA 32/32 \ufb02ex 91.25 69.36\r\n F2\r\n WA static 91.23 69.14 tiplications that can exploit data sparsity, as is the case of\r\n F4\r\n WA \ufb02ex 91.41 69.32\r\n F4 Arm\u2019sComputeLibrary, having zeros means less compute\r\n im2row - 91.15 69.34 which often translate into lower latencies.\r\n WA static 90.88 70.06\r\n F2 The Winograd-aware formulation here presented doesn\u2019t\r\n WA 8 / 8 \ufb02ex 91.03 70.18\r\n F2 impose restrictions on how the learnt transform should look\r\n WA static 79.28 55.84\r\n F4\r\n WA \ufb02ex 90.72 69.73\r\n F4 like. As a consequence, the resulting transforms rarely con-\r\n tain zeros. This translates in additional compute for input\r\n Table 4. Comparison between standard convolutions (im2row) and BTdBandoutputATyAtransforms. Theimpactofusing\r\n Winograd-aware layers for SqueezeNet. With INT8 quantization dense, learnt, transforms for WA models running on a\r\n and using the default transformation matrices (static), larger tile F4\r\n sizes (i.e. F4) introduce substantial numerical error and result in a Cortex-A73 is a latency increase of 17% (+8ms) and 20%\r\n sever accuracy drop. This drop in accuracy is signi\ufb01cantly reduced (+6ms) for FP32 and INT8 respectively for a ResNet18 net-\r\n if the transformations are learnt (\ufb02ex). work. This increase in latency is higher on the Cortex-A53\r\n since the Winograd transforms are proportionally more ex-\r\n For both architectures, INT8 Winograd-aware F4 models pensive on this core. These penalties represent the worst\r\n with learnt Winograd transformations did not result in a ac- case performance increase, assuming the transforms are\r\n curacy gaps as pronounced as the ones reported for ResNet- compute bound. However, we believe that due to the ac-\r\n 18inSection6. Thesemodelsevensurpasstheim2rowbase- cess patterns of the Winograd transform kernels (gather and\r\n lines for CIFAR-100. We argue this is because SqueezeNet scatter across a wide area of memory) at least some of the\r\n andResNeXt (8\u00d716)havefewer3\u00d73convolutionallayers performance of the transforms results from misses in the\r\n (8 and 6, respecitvely) compared to ResNet-18, which has cache hierarchy and so some additional computation can be\r\n 16. Therefore, the succession of fewer convolutional layers tolerated without necessarily increasing execution time.\r\n implemented as Winograd convolutions reduces the overall We note that the impact for F2 models is considerably\r\n impact of numerical error. higher especially since the original transforms G and A\r\n are, not only sparse, but binary and the learnt ones are not.\r\n A.2 OverheadofLearntWinogradTransforms However, these penalties are never met in practice since F2\r\n Thedefault Winograd transformation matrices contain vary- Winograd-aware models with default transforms can per-\r\n ing amounts of 0\u2019s. For F2 the sparsity ratios are 50%, 33% form equally well as those with learnt transforms (as shown\r\n T T in Figure 4 and Tables 4 and 5) even in INT8.\r\n and 25% respectively for B , G and A . From the con-\r\n struction process of these matrices and specially the choice Evenwiththeperformanceloss due to the learnt transforms,\r\n of polynomial points, we would expect lower sparsity ratios we\u2019re still demonstrating some (non-negligible) 1.54\u00d7 and\r\n Searching for Winograd-aware Quantized Networks\r\n 1.43\u00d7speedupcomparedtoINT8im2rowforA73andA53\r\n respectively. To the best of our knowledge this is the \ufb01rst\r\n time INT8 Winograd convolutions are empirically proven\r\n to work.\r\n A.3 Architectures optimized with wiNAS\r\n Our framework wiNAS, takes a given macro-architecture\r\n and optimizes each 3 \u00d7 3 convolutional layer by choosing\r\n from direct convolution or different Winograd con\ufb01gura-\r\n tions. For the search, all 1 \u00d7 1 convolutions were \ufb01xed to\r\n use im2row.\r\n For wiNAS in FP32, the resulting architecture only sub-\r\n WA\r\n stituted the last convolution layer with im2row instead of\r\n F2. The rest of the layers remained unchanged from the\r\n WAF4 con\ufb01guration (which was described in Section 5.1).\r\n The same micro-architecture was used in CIFAR-10 and\r\n CIFAR-100.\r\n For wiNAS with 8-bit quantization and CIFAR-10,\r\n WA\r\n wiNASreplacedthe5th and second last convolutional lay-\r\n ers with im2row, instead of F4 and F2 respectively. For\r\n CIFAR-100, more optimization was compared to WA .\r\n F4\r\n Theresulting micro-architecture optimization is shown in\r\n Figure 9 (left).\r\n When introducing quantization in the search space,\r\n wiNASWA-Q,the resulting architectures are shown in Figure\r\n 9 for both CIFAR-10 (middle) and CIFAR-100 (right).\r\n Searching for Winograd-aware Quantized Networks\r\n Input Input Input\r\n im2row FP32 im2row FP32 im2row FP32\r\n im2row INT8 F4 FP32 im2row\u00a0FP32\r\n im2row INT8 F4 INT16 im2row\u00a0FP32\r\n im2row INT8 F4\u00a0INT16 F2 FP32\r\n im2row INT8 F4\u00a0INT16 F2\u00a0FP32\r\n im2row INT8 F4\u00a0INT8 F2\u00a0FP32\r\n im2row INT8 im2row INT8 im2row FP32\r\n im2row INT8 F4\u00a0INT8 F4\u00a0FP32\r\n F4 INT8 F4\u00a0INT8 F4\u00a0INT8\r\n F4 INT8 F4 INT8 F4 INT8\r\n F4 INT8 F4 INT8 F4 INT8\r\n im2row INT8 im2row INT8 im2row INT8\r\n F4 INT8 F4 INT8 F4 INT8\r\n F4 INT8 F4 INT8 F4 INT8\r\n F2\u00a0INT8 F2\u00a0INT8 F2\u00a0INT8\r\n F2\u00a0INT8 F2\u00a0INT8 F2\u00a0INT8\r\n im2row INT8 im2row INT8 im2row INT8\r\n F2\u00a0INT8 F2\u00a0INT8 F2\u00a0INT8\r\n im2row\u00a0INT8 F2\u00a0INT8 im2row\u00a0INT8\r\n im2row\u00a0INT8 F2\u00a0INT8 im2row\u00a0INT8\r\n FC FC FC\r\n Output Output Output\r\n Figure 9. Resulting architectures after optimizing a ResNet-18 macro-architecture using wiNAS. For wiNASWA and CIFAR-100, the\r\n architecture resulted is shown on the left. With wiNAS , that introduces quantization in the search space, the optimization resulted in\r\n WA-Q\r\n different architectures for CIFAR-10 (middle) and CIFAR-100 (right), evidencing the difference in complexity of the latter.\r\n", "award": [], "sourceid": 9, "authors": [{"given_name": "Javier", "family_name": "Fernandez-Marques", "institution": "University of Oxford"}, {"given_name": "Paul", "family_name": "Whatmough", "institution": "Arm ML Research Lab"}, {"given_name": "Andrew", "family_name": "Mundy", "institution": "Arm ML Research Lab"}, {"given_name": "Matthew", "family_name": "Mattina", "institution": "Arm ML Research Lab"}]}