Class GradScaler
java.lang.Object
ai.beestream.jitnn.nn.optim.GradScaler
Dynamic loss scaling for FP16 training, as PyTorch's
torch.amp.GradScaler: the loss is
multiplied by a scale before backward; unscale(Optimizer) makes the optimiser divide the
gradients by it (inside the update, together with any clipping) and look for overflows on the
device; step(Optimizer) runs the optimiser, which skips the update on the device when an overflow
was found; update() (after the step has run, e.g. after loss.item()) halves the
scale on an overflow, or doubles it after growthInterval clean steps. The scale reaches
the device as a fresh host input each step, so changing it never recompiles.
Tensor loss = ...;
scaler.scale(loss).backward();
scaler.unscale(opt); // optional: before clipping
opt.clipGradNorm(1.0f);
scaler.step(opt);
float value = loss.item();
scaler.update();
-
Constructor Summary
ConstructorsConstructorDescriptionA scaler with PyTorch's defaults: initial scale 65536, growth 2 after 2000 clean steps, backoff 0.5.GradScaler(float initScale, float growthFactor, float backoffFactor, int growthInterval) A scaler starting atinitScalethat multiplies the scale bygrowthFactoraftergrowthIntervalsteps without overflow and bybackoffFactorafter an overflow. -
Method Summary
Modifier and TypeMethodDescriptionvoidRestores the scale and step count saved bystateDict().floatscale()The current loss scale.loss * scale.The scale and the count of clean steps since it last changed, for checkpoints.voidUnscales (unless done already) and records the optimiser step, skipped on the device on an overflow.voidMakesoptdivide its gradients by the scale at the next step (and clip after unscaling).booleanupdate()Adjusts the scale from this step's overflow flag (reading it waits for the step) and returns whether the step's update was skipped.
-
Constructor Details
-
GradScaler
public GradScaler()A scaler with PyTorch's defaults: initial scale 65536, growth 2 after 2000 clean steps, backoff 0.5. -
GradScaler
public GradScaler(float initScale, float growthFactor, float backoffFactor, int growthInterval) A scaler starting atinitScalethat multiplies the scale bygrowthFactoraftergrowthIntervalsteps without overflow and bybackoffFactorafter an overflow.
-
-
Method Details
-
scale
public float scale()The current loss scale. -
scale
-
unscale
Makesoptdivide its gradients by the scale at the next step (and clip after unscaling). -
step
Unscales (unless done already) and records the optimiser step, skipped on the device on an overflow. -
update
public boolean update()Adjusts the scale from this step's overflow flag (reading it waits for the step) and returns whether the step's update was skipped. -
stateDict
The scale and the count of clean steps since it last changed, for checkpoints. -
loadStateDict
Restores the scale and step count saved bystateDict().
-