Class Autocast

java.lang.Object
ai.beestream.jitnn.Autocast

public final class Autocast extends Object
Mixed precision, per thread: inside enable(DType), F.linear(Tensor, Tensor, Tensor), Tensor.matmul(Tensor) and the products of F.scaledDotProductAttention(Tensor, Tensor, Tensor, Tensor, boolean, float) cast their operands to FP16 or BF16 and run the half-precision GEMMs, accumulating in FP32. Their results stay FP32 (PyTorch's autocast rounds them to the half type), and every other operator runs in FP32. Gradients of the products are computed in the half type and flow back to the FP32 parameters through the casts. For inference, weights are cast once per weight version (weight(Tensor)), and on CUDA a Linear whose shape the tensor-core kernel covers reads its FP32 input directly, rounding it to FP16 as the tiles are staged, so no separate cast runs.
  • Method Summary

    Modifier and Type
    Method
    Description
    static DType
    The active autocast type, or null.
    static Grad.Scope
    enable(DType dtype)
    Enables autocast to dtype (F16 or BF16) until the returned scope is closed.
    static Grad.Scope
    Half-precision activations, as a model converted to FP16: inside the scope, linear layers, element-wise operators, LayerNorm and RMSNorm and attention take and return FP16 tensors (computing in FP32 inside), so activations stay FP16 between them, with half the memory traffic.
    static boolean
    True inside enableHalf(): FP16-capable operators return FP16.
    static boolean
    True when FP16 activations should stay FP16 through operators that compute in FP32 (inside enableHalf(), outside the backward pass): their results are cast back to FP16, so an FP32 fallback does not turn the rest of the forward pass FP32.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Method Details

    • enable

      public static Grad.Scope enable(DType dtype)
      Enables autocast to dtype (F16 or BF16) until the returned scope is closed.
    • enableHalf

      public static Grad.Scope enableHalf()
      Half-precision activations, as a model converted to FP16: inside the scope, linear layers, element-wise operators, LayerNorm and RMSNorm and attention take and return FP16 tensors (computing in FP32 inside), so activations stay FP16 between them, with half the memory traffic. Operators without an FP16 form take FP32 copies of FP16 inputs and return FP32. In inference weights are cast to FP16 once per weight version, as with enable(DType).

      In training the weights stay FP32 (the master copy the optimiser updates) and are cast to FP16 inside the graph; the backward pass runs the same way (autograd switches it on when the graph holds FP16 tensors): parameters get gradients of their own type (FP32), and activations' gradients stay as the operators compute them (mostly FP32). FP16 training needs loss scaling (GradScaler).

    • keepHalf

      public static boolean keepHalf()
      True when FP16 activations should stay FP16 through operators that compute in FP32 (inside enableHalf(), outside the backward pass): their results are cast back to FP16, so an FP32 fallback does not turn the rest of the forward pass FP32.
    • halfActivations

      public static boolean halfActivations()
      True inside enableHalf(): FP16-capable operators return FP16.
    • current

      public static DType current()
      The active autocast type, or null.