Class Bucketing

java.lang.Object
ai.beestream.jitnn.nn.Bucketing

public final class Bucketing extends Object
Runs models on inputs padded to Buckets sizes, so requests of changing batch and sequence length reuse a few compiled plans, and returns results for the real rows and positions only. Inputs are padded and results cut on the host; the values that depend on the real sizes (masks, loss weights) are tensors, not constants, so they never change a plan.

Padded rows must not change the real rows: this holds for inference (BatchNorm uses its running statistics) and for training with crossEntropy(Tensor, Tensor) over models without BatchNorm in training mode, whose batch statistics would include the padded rows.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final float
    The additive mask value for padded positions.
  • Method Summary

    Modifier and Type
    Method
    Description
    static Tensor
    causal(Buckets batch, Buckets seq, Tensor idx, int padToken, UnaryOperator<Tensor> model)
    A causal token model (GPT-2, Llama) over INT32 tokens [B, T]: rows padded to a batch size, positions padded at the end to a seq size with padToken, which the real positions never attend to; the result [B, T, ...] for the real rows and positions.
    static Tensor
    crossEntropy(Tensor logits, Tensor labels)
    Mean cross-entropy of the first N rows of logits [B, C] (B a padded batch) against INT32 labels [N]: each real row weighs 1/N and each padded row 0, through a weight tensor, so every N in the bucket runs the same plan.
    static Tensor
    masked(Buckets batch, Buckets seq, Tensor idx, Tensor mask, int padToken, BinaryOperator<Tensor> model)
    An encoder with an additive attention mask (BERT): model takes tokens [B, T] and a mask [B, 1, 1, T] or [B, 1, T, T] (mask null for none).
    static Tensor
    rows(Buckets batch, Tensor x, UnaryOperator<Tensor> model)
    model over x [N, ...] with N padded to a batch size by zero rows; the result's first N rows.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Field Details

    • MASKED

      public static final float MASKED
      The additive mask value for padded positions.
      See Also:
  • Method Details

    • rows

      public static Tensor rows(Buckets batch, Tensor x, UnaryOperator<Tensor> model)
      model over x [N, ...] with N padded to a batch size by zero rows; the result's first N rows.
    • causal

      public static Tensor causal(Buckets batch, Buckets seq, Tensor idx, int padToken, UnaryOperator<Tensor> model)
      A causal token model (GPT-2, Llama) over INT32 tokens [B, T]: rows padded to a batch size, positions padded at the end to a seq size with padToken, which the real positions never attend to; the result [B, T, ...] for the real rows and positions.
    • masked

      public static Tensor masked(Buckets batch, Buckets seq, Tensor idx, Tensor mask, int padToken, BinaryOperator<Tensor> model)
      An encoder with an additive attention mask (BERT): model takes tokens [B, T] and a mask [B, 1, 1, T] or [B, 1, T, T] (mask null for none). Padded positions get MASKED, so the real positions do not attend to them; padded rows attend to their padding tokens. The result [B, T, ...] for the real rows and positions.
    • crossEntropy

      public static Tensor crossEntropy(Tensor logits, Tensor labels)
      Mean cross-entropy of the first N rows of logits [B, C] (B a padded batch) against INT32 labels [N]: each real row weighs 1/N and each padded row 0, through a weight tensor, so every N in the bucket runs the same plan. Equal to F.crossEntropy(Tensor, Tensor) over the real rows, gradients included.