Package ai.beestream.jitnn.nn.optim


package ai.beestream.jitnn.nn.optim
Optimisers with PyTorch's semantics (SGD, Adam and AdamW, AdaGrad, RMSProp, AdaDelta, AdaMax, NAdam, LAMB, LARS), learning-rate schedules (LRScheduler) and dynamic loss scaling for FP16 training (GradScaler).
  • Classes
    Class
    Description
    AdaDelta (Zeiler), PyTorch's semantics: decay rho, eps, learning rate (default 1), L2 weight decay.
    AdaGrad (Duchi et al.), PyTorch's semantics: learning-rate decay, L2 weight decay, initial accumulator.
    Adam (Kingma and Ba) with L2 weight decay, or AdamW (decoupled decay) via Adam.adamW(List, float, float); with amsgrad, AMSGrad's running maximum of the second moment (Reddi et al.).
    AdaMax (Kingma and Ba): Adam with an infinity norm, PyTorch's semantics, L2 weight decay.
    Dynamic loss scaling for FP16 training, as PyTorch's torch.amp.GradScaler: the loss is multiplied by a scale before backward; GradScaler.unscale(Optimizer) makes the optimiser divide the gradients by it (inside the update, together with any clipping) and look for overflows on the device; GradScaler.step(Optimizer) runs the optimiser, which skips the update on the device when an overflow was found; GradScaler.update() (after the step has run, e.g. after loss.item()) halves the scale on an overflow, or doubles it after growthInterval clean steps.
    LAMB (You et al., 2020): Adam's bias-corrected update plus decoupled weight decay, scaled per parameter by the trust ratio |p| / |update|.
    LARS (You et al., 2017) with momentum: a per-parameter learning rate eta |p| / (|g| + weightDecay |p|), applied to the decayed gradient and accumulated in a momentum buffer.
    Sets an optimiser's learning rate per epoch; call LRScheduler.step() once at the end of each epoch, as with PyTorch's schedulers.
    NAdam (Dozat), PyTorch's semantics: momentum decay psi (default 0.004), L2 weight decay.
    Updates parameters in place from their gradients.
    RMSProp (Hinton), PyTorch's semantics: smoothing alpha, optional momentum and centring, L2 weight decay.
    Stochastic gradient descent with optional (Nesterov) momentum and L2 weight decay, as in PyTorch.