Class Optimizer

java.lang.Object
ai.beestream.jitnn.nn.optim.Optimizer
Direct Known Subclasses:
AdaDelta, AdaGrad, Adam, AdaMax, Lamb, Lars, NAdam, RMSProp, SGD

public abstract class Optimizer extends Object
Updates parameters in place from their gradients.
  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected float
    The learning rate.
    protected final List<Tensor>
    The parameters this optimiser updates.
    protected int
    Steps taken (the bias corrections of the adaptive methods use it).
  • Constructor Summary

    Constructors
    Modifier
    Constructor
    Description
    protected
    Optimizer(List<Tensor> params, float lr)
    An optimiser over params with learning rate lr.
  • Method Summary

    Modifier and Type
    Method
    Description
    clipGradNorm(float maxNorm)
    Clips the gradients of this optimiser's parameters to a global L2 norm of at most maxNorm at the next step(), as PyTorch's clip_grad_norm_, and returns the norm before clipping (after loss unscaling).
    float
    The current learning rate.
    protected void
    Restores what saveScalars(StateDict) saved.
    void
    Restores stateDict() into this optimiser, in place; type and state shapes must match.
    protected void
    Scalar state beyond the step count and learning rate, for checkpoints.
    void
    setLearningRate(float lr)
    Changes the learning rate; takes effect at the next step without recompiling.
    protected Map<String,Tensor[]>
    Per-parameter state by slot name, each array indexed like the parameters (null where none).
    The optimiser's state for a checkpoint: its type, learning rate, step count and per-parameter state (state.<slot>.<parameter index>), which refers to the live tensors.
    abstract void
    Records one update of every parameter that has a gradient.
    protected Tensor
    The statistics the next update uses (computed now when a pre-scale is pending), or null; clears them.
    void
    Clears the gradients of every parameter.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Field Details

    • params

      protected final List<Tensor> params
      The parameters this optimiser updates.
    • lr

      protected float lr
      The learning rate.
    • t

      protected int t
      Steps taken (the bias corrections of the adaptive methods use it).
  • Constructor Details

    • Optimizer

      protected Optimizer(List<Tensor> params, float lr)
      An optimiser over params with learning rate lr.
  • Method Details

    • step

      public abstract void step()
      Records one update of every parameter that has a gradient.
    • clipGradNorm

      public Tensor clipGradNorm(float maxNorm)
      Clips the gradients of this optimiser's parameters to a global L2 norm of at most maxNorm at the next step(), as PyTorch's clip_grad_norm_, and returns the norm before clipping (after loss unscaling). One small reduction per gradient; the gradients are scaled inside the update itself. Reading the norm is a sync point.
    • takeStats

      protected Tensor takeStats()
      The statistics the next update uses (computed now when a pre-scale is pending), or null; clears them.
    • zeroGrad

      public void zeroGrad()
      Clears the gradients of every parameter.
    • learningRate

      public float learningRate()
      The current learning rate.
    • setLearningRate

      public void setLearningRate(float lr)
      Changes the learning rate; takes effect at the next step without recompiling.
    • state

      protected Map<String,Tensor[]> state()
      Per-parameter state by slot name, each array indexed like the parameters (null where none).
    • saveScalars

      protected void saveScalars(StateDict d)
      Scalar state beyond the step count and learning rate, for checkpoints.
    • loadScalars

      protected void loadScalars(StateDict d)
      Restores what saveScalars(StateDict) saved.
    • stateDict

      public StateDict stateDict()
      The optimiser's state for a checkpoint: its type, learning rate, step count and per-parameter state (state.<slot>.<parameter index>), which refers to the live tensors. The hyper-parameters are not included: build the optimiser with the same arguments before loading.
    • loadStateDict

      public void loadStateDict(StateDict d)
      Restores stateDict() into this optimiser, in place; type and state shapes must match.