Class LRScheduler

java.lang.Object
ai.beestream.jitnn.nn.optim.LRScheduler

public abstract class LRScheduler extends Object
Sets an optimiser's learning rate per epoch; call step() once at the end of each epoch, as with PyTorch's schedulers. The new rate takes effect without recompiling. The warm-up schedules count optimiser steps instead: call step() after every optimizer.step().
  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected final float
    The optimiser's learning rate when the schedule was created.
    protected int
    Epochs (or, for warm-up schedules, steps) counted so far.
    protected final Optimizer
    The optimiser whose rate this schedule sets.
  • Constructor Summary

    Constructors
    Modifier
    Constructor
    Description
    protected
    A schedule for optimizer, starting from its current learning rate.
  • Method Summary

    Modifier and Type
    Method
    Description
    cosine(Optimizer optimizer, int tMax, float minLr)
    Cosine annealing from base to minLr over tMax epochs (PyTorch's closed form).
    float
    The optimiser's current learning rate.
    void
    Restores the count saved by stateDict().
    multiStep(Optimizer optimizer, int[] milestones, float gamma)
    lr = base * gamma^m where m counts the milestones passed.
    protected abstract float
    rate(int epoch)
    The learning rate for epoch (0-based).
    The epoch (or step) count, for checkpoints; the rate itself is restored with the optimiser.
    void
    Advances the count and sets the optimiser's rate for it.
    step(Optimizer optimizer, int stepSize, float gamma)
    lr = base * gamma^(epoch / stepSize).
    warmupCosine(Optimizer optimizer, int warmup, int total, float minRatio)
    Linear warm-up over warmup steps, then a half-cosine decay to minRatio times the base rate at total steps (Hugging Face's get_cosine_schedule_with_warmup when minRatio is 0).
    warmupLinear(Optimizer optimizer, int warmup, int total)
    Linear warm-up from 0 to the base rate over warmup steps, then linear decay to 0 at total steps (Hugging Face's get_linear_schedule_with_warmup).

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Field Details

    • optimizer

      protected final Optimizer optimizer
      The optimiser whose rate this schedule sets.
    • baseLr

      protected final float baseLr
      The optimiser's learning rate when the schedule was created.
    • epoch

      protected int epoch
      Epochs (or, for warm-up schedules, steps) counted so far.
  • Constructor Details

    • LRScheduler

      protected LRScheduler(Optimizer optimizer)
      A schedule for optimizer, starting from its current learning rate.
  • Method Details

    • rate

      protected abstract float rate(int epoch)
      The learning rate for epoch (0-based).
    • step

      public void step()
      Advances the count and sets the optimiser's rate for it.
    • stateDict

      public StateDict stateDict()
      The epoch (or step) count, for checkpoints; the rate itself is restored with the optimiser.
    • loadStateDict

      public void loadStateDict(StateDict d)
      Restores the count saved by stateDict().
    • currentRate

      public float currentRate()
      The optimiser's current learning rate.
    • step

      public static LRScheduler step(Optimizer optimizer, int stepSize, float gamma)
      lr = base * gamma^(epoch / stepSize).
    • multiStep

      public static LRScheduler multiStep(Optimizer optimizer, int[] milestones, float gamma)
      lr = base * gamma^m where m counts the milestones passed.
    • cosine

      public static LRScheduler cosine(Optimizer optimizer, int tMax, float minLr)
      Cosine annealing from base to minLr over tMax epochs (PyTorch's closed form).
    • warmupLinear

      public static LRScheduler warmupLinear(Optimizer optimizer, int warmup, int total)
      Linear warm-up from 0 to the base rate over warmup steps, then linear decay to 0 at total steps (Hugging Face's get_linear_schedule_with_warmup). Sets the step-0 rate now.
    • warmupCosine

      public static LRScheduler warmupCosine(Optimizer optimizer, int warmup, int total, float minRatio)
      Linear warm-up over warmup steps, then a half-cosine decay to minRatio times the base rate at total steps (Hugging Face's get_cosine_schedule_with_warmup when minRatio is 0). Sets the step-0 rate now.