Package ai.beestream.jitnn.nn.optim
package ai.beestream.jitnn.nn.optim
Optimisers with PyTorch's semantics (SGD, Adam and AdamW, AdaGrad, RMSProp, AdaDelta, AdaMax,
NAdam, LAMB, LARS), learning-rate schedules (
LRScheduler) and dynamic
loss scaling for FP16 training (GradScaler).-
ClassesClassDescriptionAdaDelta (Zeiler), PyTorch's semantics: decay rho, eps, learning rate (default 1), L2 weight decay.AdaGrad (Duchi et al.), PyTorch's semantics: learning-rate decay, L2 weight decay, initial accumulator.Adam (Kingma and Ba) with L2 weight decay, or AdamW (decoupled decay) via
Adam.adamW(List, float, float); withamsgrad, AMSGrad's running maximum of the second moment (Reddi et al.).AdaMax (Kingma and Ba): Adam with an infinity norm, PyTorch's semantics, L2 weight decay.Dynamic loss scaling for FP16 training, as PyTorch'storch.amp.GradScaler: the loss is multiplied by a scale before backward;GradScaler.unscale(Optimizer)makes the optimiser divide the gradients by it (inside the update, together with any clipping) and look for overflows on the device;GradScaler.step(Optimizer)runs the optimiser, which skips the update on the device when an overflow was found;GradScaler.update()(after the step has run, e.g. afterloss.item()) halves the scale on an overflow, or doubles it aftergrowthIntervalclean steps.LAMB (You et al., 2020): Adam's bias-corrected update plus decoupled weight decay, scaled per parameter by the trust ratio |p| / |update|.LARS (You et al., 2017) with momentum: a per-parameter learning rate eta |p| / (|g| + weightDecay |p|), applied to the decayed gradient and accumulated in a momentum buffer.Sets an optimiser's learning rate per epoch; callLRScheduler.step()once at the end of each epoch, as with PyTorch's schedulers.NAdam (Dozat), PyTorch's semantics: momentum decay psi (default 0.004), L2 weight decay.Updates parameters in place from their gradients.RMSProp (Hinton), PyTorch's semantics: smoothing alpha, optional momentum and centring, L2 weight decay.Stochastic gradient descent with optional (Nesterov) momentum and L2 weight decay, as in PyTorch.