Part of Proceedings of Machine Learning and Systems 5 (MLSys 2023) mlsys2023
Tim Kaler, Alexandros Iliopoulos, Philip Murzynowski, Tao Schardl, Charles E. Leiserson, Jie Chen
Training and inference with graph neural networks (GNNs) on massive graphs in a distributed environment has been actively studied since the inception of GNNs, owing to the widespread use and success of GNNs in applications such as recommendation systems and financial forensics. This paper is concerned with minibatch training and inference with GNNs in distributed settings, where the necessary partitioning of vertex features across distributed storage causes feature communication to become a major bottleneck that hampers scalability.To significantly reduce the communication volume without compromising prediction accuracy, we propose a policy for caching data associated with frequently accessed vertices in remote partitions. The proposed policy is based on an analysis of vertex-wise inclusion probabilities (VIP) during multi-hop neighborhood sampling, which may expand the neighborhood far beyond the partition boundary of the graph. The VIP analysis not only enables the elimination of the communication bottleneck, but also offers a means to organize in-memory data by prioritizing GPU storage for the most frequently accessed vertex features. We present SALIENT++, which extends the prior state-of-the-art SALIENT system to work with partitioned feature data and leverages the VIP-driven caching policy. SALIENT++ retains the local training efficiency and scalability of SALIENT by using a deep pipeline and drastically reducing communication volume while consuming only a fraction of the storage required by SALIENT. We demonstrate experiments on the Open Graph Benchmark data sets and show that training a 3-layer GraphSAGE model with SALIENT++ on 8 single-GPU machines is 7.1x faster than with SALIENT on 1 single-GPU machine, and 12.7x faster than with DistDGL on 8 single-GPU machines.