Class Autocast
java.lang.Object
ai.beestream.jitnn.Autocast
Mixed precision, per thread: inside
enable(DType), F.linear(Tensor, Tensor, Tensor), Tensor.matmul(Tensor)
and the products of F.scaledDotProductAttention(Tensor, Tensor, Tensor, Tensor, boolean, float) cast their operands to FP16 or BF16
and run the half-precision GEMMs, accumulating in FP32. Their results stay FP32 (PyTorch's
autocast rounds them to the half type), and every other operator runs in FP32. Gradients of
the products are computed in the half type and flow back to the FP32 parameters through the casts.
For inference, weights are cast once per weight version (weight(Tensor)), and on CUDA a
Linear whose shape the tensor-core kernel covers reads its FP32 input directly, rounding it to
FP16 as the tiles are staged, so no separate cast runs.-
Method Summary
Modifier and TypeMethodDescriptionstatic DTypecurrent()The active autocast type, or null.static Grad.ScopeEnables autocast todtype(F16 or BF16) until the returned scope is closed.static Grad.ScopeHalf-precision activations, as a model converted to FP16: inside the scope, linear layers, element-wise operators, LayerNorm and RMSNorm and attention take and return FP16 tensors (computing in FP32 inside), so activations stay FP16 between them, with half the memory traffic.static booleanTrue insideenableHalf(): FP16-capable operators return FP16.static booleankeepHalf()True when FP16 activations should stay FP16 through operators that compute in FP32 (insideenableHalf(), outside the backward pass): their results are cast back to FP16, so an FP32 fallback does not turn the rest of the forward pass FP32.
-
Method Details
-
enable
Enables autocast todtype(F16 or BF16) until the returned scope is closed. -
enableHalf
Half-precision activations, as a model converted to FP16: inside the scope, linear layers, element-wise operators, LayerNorm and RMSNorm and attention take and return FP16 tensors (computing in FP32 inside), so activations stay FP16 between them, with half the memory traffic. Operators without an FP16 form take FP32 copies of FP16 inputs and return FP32. In inference weights are cast to FP16 once per weight version, as withenable(DType).In training the weights stay FP32 (the master copy the optimiser updates) and are cast to FP16 inside the graph; the backward pass runs the same way (autograd switches it on when the graph holds FP16 tensors): parameters get gradients of their own type (FP32), and activations' gradients stay as the operators compute them (mostly FP32). FP16 training needs loss scaling (
GradScaler). -
keepHalf
public static boolean keepHalf()True when FP16 activations should stay FP16 through operators that compute in FP32 (insideenableHalf(), outside the backward pass): their results are cast back to FP16, so an FP32 fallback does not turn the rest of the forward pass FP32. -
halfActivations
public static boolean halfActivations()True insideenableHalf(): FP16-capable operators return FP16. -
current
The active autocast type, or null.
-