Class DataParallel

java.lang.Object
ai.beestream.jitnn.nn.DataParallel

public final class DataParallel extends Object
Data-parallel training with one model replica per rank (see ProcessGroup): broadcast rank 0's parameters once, then after each backward pass average the gradients over the ranks before the optimiser step, so every replica takes the same step, as PyTorch's DistributedDataParallel does.
  • Constructor Summary

    Constructors
    Constructor
    Description
    DataParallel(Module model, int rank, Collective collective)
    Wraps rank rank's replica model; ranks synchronise through collective.
  • Method Summary

    Modifier and Type
    Method
    Description
    void
    Replaces each parameter's gradient by its mean over the ranks (parameters without a gradient are skipped on every rank alike), with one collective for all of them.
    Copies rank 0's parameters and buffers to every rank.
    The wrapped replica.
    overlapGradients(long bucketBytes)
    Overlaps the gradient all-reduce with the backward pass, as PyTorch's DistributedDataParallel: the parameters are grouped, in reverse order (the order their gradients become final), into buckets of about bucketBytes; a bucket's all-reduce starts as soon as its last gradient is final, while the backward pass of the earlier layers goes on, and averageGradients() only waits for it.
    int
    This replica's rank.
    int
    The number of ranks.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Constructor Details

    • DataParallel

      public DataParallel(Module model, int rank, Collective collective)
      Wraps rank rank's replica model; ranks synchronise through collective.
  • Method Details

    • broadcastParameters

      public DataParallel broadcastParameters()
      Copies rank 0's parameters and buffers to every rank.
    • averageGradients

      public void averageGradients()
      Replaces each parameter's gradient by its mean over the ranks (parameters without a gradient are skipped on every rank alike), with one collective for all of them.
    • overlapGradients

      public DataParallel overlapGradients(long bucketBytes)
      Overlaps the gradient all-reduce with the backward pass, as PyTorch's DistributedDataParallel: the parameters are grouped, in reverse order (the order their gradients become final), into buckets of about bucketBytes; a bucket's all-reduce starts as soon as its last gradient is final, while the backward pass of the earlier layers goes on, and averageGradients() only waits for it. With NCCL the reductions run on a stream of their own; other collectives reduce at averageGradients() as before. One backward pass per step.
    • module

      public Module module()
      The wrapped replica.
    • rank

      public int rank()
      This replica's rank.
    • worldSize

      public int worldSize()
      The number of ranks.