Class SyncBatchNorm2d

java.lang.Object
ai.beestream.jitnn.nn.Module
ai.beestream.jitnn.nn.SyncBatchNorm2d
All Implemented Interfaces:
AutoCloseable

public final class SyncBatchNorm2d extends Module
Batch normalisation whose training statistics are those of the batch over all data-parallel ranks, as PyTorch's SyncBatchNorm: each rank sums its x and x^2 per channel, one all-reduce gives the global mean and variance, and the backward pass all-reduces their gradients the same way. Ranks training on shares of a batch then match one device training on the whole batch. Evaluation uses the running statistics, which are updated alike on every rank. Needs an NcclCollective (its all-reduce is differentiable); without a collective it is plain batch normalisation. NCHW input, PyTorch defaults (momentum 0.1, eps 1e-5).
  • Constructor Details

    • SyncBatchNorm2d

      public SyncBatchNorm2d(int channels, Collective collective, int rank)
      Synchronised batch normalisation for rank rank, with PyTorch's defaults (momentum 0.1, eps 1e-5).
    • SyncBatchNorm2d

      public SyncBatchNorm2d(int channels, Collective collective, int rank, float momentum, float eps)
      Synchronised batch normalisation for rank rank of an NcclCollective; with a null collective (or one rank) it is plain batch normalisation.
  • Method Details

    • runningMean

      public Tensor runningMean()
      The running mean [channels], updated in training mode.
    • runningVar

      public Tensor runningVar()
      The running variance [channels], updated in training mode.
    • forward

      public Tensor forward(Tensor x)
      Description copied from class: Module
      Applies the module to x.
      Specified by:
      forward in class Module