Class GradScaler

java.lang.Object
ai.beestream.jitnn.nn.optim.GradScaler

public final class GradScaler extends Object
Dynamic loss scaling for FP16 training, as PyTorch's torch.amp.GradScaler: the loss is multiplied by a scale before backward; unscale(Optimizer) makes the optimiser divide the gradients by it (inside the update, together with any clipping) and look for overflows on the device; step(Optimizer) runs the optimiser, which skips the update on the device when an overflow was found; update() (after the step has run, e.g. after loss.item()) halves the scale on an overflow, or doubles it after growthInterval clean steps. The scale reaches the device as a fresh host input each step, so changing it never recompiles.
Tensor loss = ...;
scaler.scale(loss).backward();
scaler.unscale(opt);              // optional: before clipping
opt.clipGradNorm(1.0f);
scaler.step(opt);
float value = loss.item();
scaler.update();
  • Constructor Summary

    Constructors
    Constructor
    Description
    A scaler with PyTorch's defaults: initial scale 65536, growth 2 after 2000 clean steps, backoff 0.5.
    GradScaler(float initScale, float growthFactor, float backoffFactor, int growthInterval)
    A scaler starting at initScale that multiplies the scale by growthFactor after growthInterval steps without overflow and by backoffFactor after an overflow.
  • Method Summary

    Modifier and Type
    Method
    Description
    void
    Restores the scale and step count saved by stateDict().
    float
    The current loss scale.
    scale(Tensor loss)
    loss * scale.
    The scale and the count of clean steps since it last changed, for checkpoints.
    void
    Unscales (unless done already) and records the optimiser step, skipped on the device on an overflow.
    void
    Makes opt divide its gradients by the scale at the next step (and clip after unscaling).
    boolean
    Adjusts the scale from this step's overflow flag (reading it waits for the step) and returns whether the step's update was skipped.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Constructor Details

    • GradScaler

      public GradScaler()
      A scaler with PyTorch's defaults: initial scale 65536, growth 2 after 2000 clean steps, backoff 0.5.
    • GradScaler

      public GradScaler(float initScale, float growthFactor, float backoffFactor, int growthInterval)
      A scaler starting at initScale that multiplies the scale by growthFactor after growthInterval steps without overflow and by backoffFactor after an overflow.
  • Method Details

    • scale

      public float scale()
      The current loss scale.
    • scale

      public Tensor scale(Tensor loss)
      loss * scale.
    • unscale

      public void unscale(Optimizer opt)
      Makes opt divide its gradients by the scale at the next step (and clip after unscaling).
    • step

      public void step(Optimizer opt)
      Unscales (unless done already) and records the optimiser step, skipped on the device on an overflow.
    • update

      public boolean update()
      Adjusts the scale from this step's overflow flag (reading it waits for the step) and returns whether the step's update was skipped.
    • stateDict

      public StateDict stateDict()
      The scale and the count of clean steps since it last changed, for checkpoints.
    • loadStateDict

      public void loadStateDict(StateDict d)
      Restores the scale and step count saved by stateDict().