Uses of Class
ai.beestream.jitnn.nn.DataParallel
Packages that use DataParallel
Package
Description
Neural-network modules after PyTorch's
torch.nn: Module and
Sequential, linear and convolution layers, pooling, normalisation,
attention, embeddings, recurrent layers (LSTM, GRU), graph convolution, activations, and data-parallel
training (DataParallel).-
Uses of DataParallel in ai.beestream.jitnn.nn
Methods in ai.beestream.jitnn.nn that return DataParallelModifier and TypeMethodDescriptionDataParallel.broadcastParameters()Copies rank 0's parameters and buffers to every rank.DataParallel.overlapGradients(long bucketBytes) Overlaps the gradient all-reduce with the backward pass, as PyTorch's DistributedDataParallel: the parameters are grouped, in reverse order (the order their gradients become final), into buckets of aboutbucketBytes; a bucket's all-reduce starts as soon as its last gradient is final, while the backward pass of the earlier layers goes on, andDataParallel.averageGradients()only waits for it.