Class SyncBatchNorm2d
java.lang.Object
ai.beestream.jitnn.nn.Module
ai.beestream.jitnn.nn.SyncBatchNorm2d
- All Implemented Interfaces:
AutoCloseable
Batch normalisation whose training statistics are those of the batch over all data-parallel
ranks, as PyTorch's SyncBatchNorm: each rank sums its x and x^2 per channel, one all-reduce
gives the global mean and variance, and the backward pass all-reduces their gradients the same
way. Ranks training on shares of a batch then match one device training on the whole batch.
Evaluation uses the running statistics, which are updated alike on every rank. Needs an
NcclCollective (its all-reduce is differentiable); without a collective it is plain
batch normalisation. NCHW input, PyTorch defaults (momentum 0.1, eps 1e-5).-
Constructor Summary
ConstructorsConstructorDescriptionSyncBatchNorm2d(int channels, Collective collective, int rank) Synchronised batch normalisation for rankrank, with PyTorch's defaults (momentum 0.1, eps 1e-5).SyncBatchNorm2d(int channels, Collective collective, int rank, float momentum, float eps) Synchronised batch normalisation for rankrankof anNcclCollective; with a null collective (or one rank) it is plain batch normalisation. -
Method Summary
Modifier and TypeMethodDescriptionApplies the module tox.The running mean [channels], updated in training mode.The running variance [channels], updated in training mode.Methods inherited from class Module
buffer, buffers, child, close, isClosed, isTraining, loadStateDict, parameter, parameterCount, parameters, stateDict, train, zeroGrad
-
Constructor Details
-
SyncBatchNorm2d
Synchronised batch normalisation for rankrank, with PyTorch's defaults (momentum 0.1, eps 1e-5). -
SyncBatchNorm2d
Synchronised batch normalisation for rankrankof anNcclCollective; with a null collective (or one rank) it is plain batch normalisation.
-
-
Method Details
-
runningMean
The running mean [channels], updated in training mode. -
runningVar
The running variance [channels], updated in training mode. -
forward
-