Class Buckets
java.lang.Object
ai.beestream.jitnn.Buckets
A fixed set of sizes that a changing dimension (batch, sequence length) is rounded up to, so
requests of many shapes run a few compiled plans instead of compiling one per shape, as
TensorRT's optimization profiles or vLLM's CUDA-graph batch sizes do.
Padding and cutting happen on the host, before and after the model runs: a pad or a slice
on the device would put the real size into that operator's plan key and compile again.
ai.beestream.jitnn.nn.Bucketing builds on this for whole models (masks for padded
sequence positions, losses over the real rows only).
-
Method Summary
Modifier and TypeMethodDescriptionstatic TensorThe firstlengthentries of dimensionaxisoft, copied on the host (a device result is read back first).intmax()The largest size.static BucketsmultiplesOf(int step, int max) The multiples ofstepup to and includingmax(rounded up to a multiple).static Bucketsof(int... sizes) These sizes.static Tensortwith dimensionaxisgrown tosize, the new elements set tovalue; built on the host (a device result is read back first).static BucketspowersOfTwo(int min, int max) The powers of two fromminup to and includingmax(both rounded up to a power of two).intsize(int n) The smallest size that holdsn; throws ifnis larger than every size.int[]sizes()The sizes, ascending.toString()
-
Method Details
-
of
These sizes. -
powersOfTwo
The powers of two fromminup to and includingmax(both rounded up to a power of two). -
multiplesOf
The multiples ofstepup to and includingmax(rounded up to a multiple). -
sizes
public int[] sizes()The sizes, ascending. -
max
public int max()The largest size. -
size
public int size(int n) The smallest size that holdsn; throws ifnis larger than every size. -
pad
-
cut
-
toString
-