Importance

Loading Importance Indicators...

Abstract

Various embodiments concern a computer-implemented method for training a neural network model utilizing zero bubble pipeline parallelism, the computer-implemented method including: performing a plurality of forward passes through the neural network model, wherein each forward pass of the plurality of forward passes transforms a corresponding input x to a corresponding output y; performing a plurality of backward passes through the neural network model, wherein the plurality backward passes are split into a plurality of gradient computation passes B and a plurality of parameters computation passes W; performing a plurality of gradient computation passes B for the corresponding input x and the corresponding output y; and determining pipeline bubbles and performing the plurality of parameters computation passes W during the pipeline bubbles.

Loading the Abstract Image...

First Claim