Class DataLoader

java.lang.Object
ai.beestream.jitnn.data.DataLoader
All Implemented Interfaces:
AutoCloseable

public final class DataLoader extends Object implements AutoCloseable
Batches of a Dataset for training, as PyTorch's DataLoader: shuffled per epoch from a seed, optional per-sample transforms (augmentation), the last short batch kept or dropped, and batches assembled ahead by worker threads while the device computes. Batches come out in order and do not depend on the number of workers: shuffling and transforms draw from random streams derived from the seed, the epoch and the sample.
try (DataLoader loader = DataLoader.builder(train, 128).shuffle(42).transform(Transforms.randomCrop(4)).workers(4).build()) {
    for (int epoch = 0; epoch < epochs; epoch++) {
        for (DataLoader.Batch b : loader.epoch(epoch)) {
            Tensor loss = F.crossEntropy(model.forward(b.x()), b.y());
            ...
        }
    }
}
  • Method Details

    • builder

      public static DataLoader.Builder builder(Dataset data, int batchSize)
      A builder for a loader of data in batches of batchSize.
    • batches

      public int batches()
      Batches per epoch.
    • epoch

      public Iterable<DataLoader.Batch> epoch(int epoch)
      The batches of epoch epoch (its shuffle and augmentation draws).
    • close

      public void close()
      Specified by:
      close in interface AutoCloseable