Uses of Class
ai.beestream.jitnn.nn.optim.Optimizer
Packages that use Optimizer
Package
Description
Optimisers with PyTorch's semantics (SGD, Adam and AdamW, AdaGrad, RMSProp, AdaDelta, AdaMax,
NAdam, LAMB, LARS), learning-rate schedules (
LRScheduler) and dynamic
loss scaling for FP16 training (GradScaler).-
Uses of Optimizer in ai.beestream.jitnn.nn.optim
Subclasses of Optimizer in ai.beestream.jitnn.nn.optimModifier and TypeClassDescriptionfinal classAdaDelta (Zeiler), PyTorch's semantics: decay rho, eps, learning rate (default 1), L2 weight decay.final classAdaGrad (Duchi et al.), PyTorch's semantics: learning-rate decay, L2 weight decay, initial accumulator.final classAdam (Kingma and Ba) with L2 weight decay, or AdamW (decoupled decay) viaAdam.adamW(List, float, float); withamsgrad, AMSGrad's running maximum of the second moment (Reddi et al.).final classAdaMax (Kingma and Ba): Adam with an infinity norm, PyTorch's semantics, L2 weight decay.final classLAMB (You et al., 2020): Adam's bias-corrected update plus decoupled weight decay, scaled per parameter by the trust ratio |p| / |update|.final classLARS (You et al., 2017) with momentum: a per-parameter learning rate eta |p| / (|g| + weightDecay |p|), applied to the decayed gradient and accumulated in a momentum buffer.final classNAdam (Dozat), PyTorch's semantics: momentum decay psi (default 0.004), L2 weight decay.final classRMSProp (Hinton), PyTorch's semantics: smoothing alpha, optional momentum and centring, L2 weight decay.final classStochastic gradient descent with optional (Nesterov) momentum and L2 weight decay, as in PyTorch.Fields in ai.beestream.jitnn.nn.optim declared as OptimizerModifier and TypeFieldDescriptionprotected final OptimizerLRScheduler.optimizerThe optimiser whose rate this schedule sets.Methods in ai.beestream.jitnn.nn.optim with parameters of type OptimizerModifier and TypeMethodDescriptionstatic LRSchedulerCosine annealing from base tominLrovertMaxepochs (PyTorch's closed form).static LRSchedulerlr = base * gamma^m where m counts the milestones passed.voidUnscales (unless done already) and records the optimiser step, skipped on the device on an overflow.static LRSchedulerlr = base * gamma^(epoch / stepSize).voidMakesoptdivide its gradients by the scale at the next step (and clip after unscaling).static LRSchedulerLRScheduler.warmupCosine(Optimizer optimizer, int warmup, int total, float minRatio) Linear warm-up overwarmupsteps, then a half-cosine decay tominRatiotimes the base rate attotalsteps (Hugging Face's get_cosine_schedule_with_warmup when minRatio is 0).static LRSchedulerLRScheduler.warmupLinear(Optimizer optimizer, int warmup, int total) Linear warm-up from 0 to the base rate overwarmupsteps, then linear decay to 0 attotalsteps (Hugging Face's get_linear_schedule_with_warmup).Constructors in ai.beestream.jitnn.nn.optim with parameters of type OptimizerModifierConstructorDescriptionprotectedLRScheduler(Optimizer optimizer) A schedule foroptimizer, starting from its current learning rate.