Class Bucketing
java.lang.Object
ai.beestream.jitnn.nn.Bucketing
Runs models on inputs padded to
Buckets sizes, so requests of changing batch and
sequence length reuse a few compiled plans, and returns results for the real rows and positions
only. Inputs are padded and results cut on the host; the values that depend on the real sizes
(masks, loss weights) are tensors, not constants, so they never change a plan.
Padded rows must not change the real rows: this holds for inference (BatchNorm uses its
running statistics) and for training with crossEntropy(Tensor, Tensor) over models without
BatchNorm in training mode, whose batch statistics would include the padded rows.
-
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final floatThe additive mask value for padded positions. -
Method Summary
Modifier and TypeMethodDescriptionstatic TensorA causal token model (GPT-2, Llama) over INT32 tokens [B, T]: rows padded to abatchsize, positions padded at the end to aseqsize withpadToken, which the real positions never attend to; the result [B, T, ...] for the real rows and positions.static TensorcrossEntropy(Tensor logits, Tensor labels) Mean cross-entropy of the first N rows oflogits[B, C] (B a padded batch) against INT32labels[N]: each real row weighs 1/N and each padded row 0, through a weight tensor, so every N in the bucket runs the same plan.static Tensormasked(Buckets batch, Buckets seq, Tensor idx, Tensor mask, int padToken, BinaryOperator<Tensor> model) An encoder with an additive attention mask (BERT):modeltakes tokens [B, T] and a mask [B, 1, 1, T] or [B, 1, T, T] (masknull for none).static Tensorrows(Buckets batch, Tensor x, UnaryOperator<Tensor> model) modeloverx[N, ...] with N padded to abatchsize by zero rows; the result's first N rows.
-
Field Details
-
MASKED
public static final float MASKEDThe additive mask value for padded positions.- See Also:
-
-
Method Details
-
rows
modeloverx[N, ...] with N padded to abatchsize by zero rows; the result's first N rows. -
causal
public static Tensor causal(Buckets batch, Buckets seq, Tensor idx, int padToken, UnaryOperator<Tensor> model) A causal token model (GPT-2, Llama) over INT32 tokens [B, T]: rows padded to abatchsize, positions padded at the end to aseqsize withpadToken, which the real positions never attend to; the result [B, T, ...] for the real rows and positions. -
masked
public static Tensor masked(Buckets batch, Buckets seq, Tensor idx, Tensor mask, int padToken, BinaryOperator<Tensor> model) An encoder with an additive attention mask (BERT):modeltakes tokens [B, T] and a mask [B, 1, 1, T] or [B, 1, T, T] (masknull for none). Padded positions getMASKED, so the real positions do not attend to them; padded rows attend to their padding tokens. The result [B, T, ...] for the real rows and positions. -
crossEntropy
Mean cross-entropy of the first N rows oflogits[B, C] (B a padded batch) against INT32labels[N]: each real row weighs 1/N and each padded row 0, through a weight tensor, so every N in the bucket runs the same plan. Equal toF.crossEntropy(Tensor, Tensor)over the real rows, gradients included.
-