Class Buckets

java.lang.Object
ai.beestream.jitnn.Buckets

public final class Buckets extends Object
A fixed set of sizes that a changing dimension (batch, sequence length) is rounded up to, so requests of many shapes run a few compiled plans instead of compiling one per shape, as TensorRT's optimization profiles or vLLM's CUDA-graph batch sizes do.

Padding and cutting happen on the host, before and after the model runs: a pad or a slice on the device would put the real size into that operator's plan key and compile again. ai.beestream.jitnn.nn.Bucketing builds on this for whole models (masks for padded sequence positions, losses over the real rows only).

  • Method Summary

    Modifier and Type
    Method
    Description
    static Tensor
    cut(Tensor t, int axis, int length)
    The first length entries of dimension axis of t, copied on the host (a device result is read back first).
    int
    max()
    The largest size.
    static Buckets
    multiplesOf(int step, int max)
    The multiples of step up to and including max (rounded up to a multiple).
    static Buckets
    of(int... sizes)
    These sizes.
    static Tensor
    pad(Tensor t, int axis, int size, float value)
    t with dimension axis grown to size, the new elements set to value; built on the host (a device result is read back first).
    static Buckets
    powersOfTwo(int min, int max)
    The powers of two from min up to and including max (both rounded up to a power of two).
    int
    size(int n)
    The smallest size that holds n; throws if n is larger than every size.
    int[]
    The sizes, ascending.
     

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
  • Method Details

    • of

      public static Buckets of(int... sizes)
      These sizes.
    • powersOfTwo

      public static Buckets powersOfTwo(int min, int max)
      The powers of two from min up to and including max (both rounded up to a power of two).
    • multiplesOf

      public static Buckets multiplesOf(int step, int max)
      The multiples of step up to and including max (rounded up to a multiple).
    • sizes

      public int[] sizes()
      The sizes, ascending.
    • max

      public int max()
      The largest size.
    • size

      public int size(int n)
      The smallest size that holds n; throws if n is larger than every size.
    • pad

      public static Tensor pad(Tensor t, int axis, int size, float value)
      t with dimension axis grown to size, the new elements set to value; built on the host (a device result is read back first). Returns t itself when the dimension is already size. INT32 tensors stay INT32.
    • cut

      public static Tensor cut(Tensor t, int axis, int length)
      The first length entries of dimension axis of t, copied on the host (a device result is read back first). Returns t itself when nothing is cut.
    • toString

      public String toString()
      Overrides:
      toString in class Object