Class DataParallel
java.lang.Object
ai.beestream.jitnn.nn.DataParallel
Data-parallel training with one model replica per rank (see
ProcessGroup): broadcast rank 0's parameters once, then after
each backward pass average the gradients over the ranks before the optimiser step, so every
replica takes the same step, as PyTorch's DistributedDataParallel does.-
Constructor Summary
ConstructorsConstructorDescriptionDataParallel(Module model, int rank, Collective collective) Wraps rankrank's replicamodel; ranks synchronise throughcollective. -
Method Summary
Modifier and TypeMethodDescriptionvoidReplaces each parameter's gradient by its mean over the ranks (parameters without a gradient are skipped on every rank alike), with one collective for all of them.Copies rank 0's parameters and buffers to every rank.module()The wrapped replica.overlapGradients(long bucketBytes) Overlaps the gradient all-reduce with the backward pass, as PyTorch's DistributedDataParallel: the parameters are grouped, in reverse order (the order their gradients become final), into buckets of aboutbucketBytes; a bucket's all-reduce starts as soon as its last gradient is final, while the backward pass of the earlier layers goes on, andaverageGradients()only waits for it.intrank()This replica's rank.intThe number of ranks.
-
Constructor Details
-
DataParallel
Wraps rankrank's replicamodel; ranks synchronise throughcollective.
-
-
Method Details
-
broadcastParameters
Copies rank 0's parameters and buffers to every rank. -
averageGradients
public void averageGradients()Replaces each parameter's gradient by its mean over the ranks (parameters without a gradient are skipped on every rank alike), with one collective for all of them. -
overlapGradients
Overlaps the gradient all-reduce with the backward pass, as PyTorch's DistributedDataParallel: the parameters are grouped, in reverse order (the order their gradients become final), into buckets of aboutbucketBytes; a bucket's all-reduce starts as soon as its last gradient is final, while the backward pass of the earlier layers goes on, andaverageGradients()only waits for it. With NCCL the reductions run on a stream of their own; other collectives reduce ataverageGradients()as before. One backward pass per step. -
module
The wrapped replica. -
rank
public int rank()This replica's rank. -
worldSize
public int worldSize()The number of ranks.
-