Class TextCorpus

java.lang.Object
ai.beestream.jitnn.data.TextCorpus

public final class TextCorpus extends Object
A byte-level text corpus for language-model gates: every byte is a token (vocabulary 256). The first 90% trains and the rest evaluates. Training windows are laid out in order without shuffling, so another framework can reproduce the exact batches (scripts/pytorch/datasets.py).
  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
    The vocabulary size: one token per byte value.
  • Method Summary

    Modifier and Type
    Method
    Description
    void
    evalWindow(int w, int seq, int[] inputs, int[] targets, int at)
    Evaluation window w: inputs and targets [seq].
    int
    evalWindows(int seq)
    Number of non-overlapping evaluation windows of seq tokens.
    int
    The number of tokens.
    static TextCorpus
    load(Path file)
    The bytes of file as a corpus.
    int
    token(int i)
    Token i.
    void
    trainBatch(int b, int rows, int seq, int[] inputs, int[] targets)
    Fills inputs and next-token targets [rows, seq] for training batch b.
    int
    trainStart(int b, int rows, int seq, int r)
    Start of training row r of batch b: consecutive windows, wrapping around the training part.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Field Details

    • VOCAB

      public static final int VOCAB
      The vocabulary size: one token per byte value.
      See Also:
  • Method Details

    • load

      public static TextCorpus load(Path file) throws IOException
      The bytes of file as a corpus.
      Throws:
      IOException
    • length

      public int length()
      The number of tokens.
    • trainStart

      public int trainStart(int b, int rows, int seq, int r)
      Start of training row r of batch b: consecutive windows, wrapping around the training part.
    • trainBatch

      public void trainBatch(int b, int rows, int seq, int[] inputs, int[] targets)
      Fills inputs and next-token targets [rows, seq] for training batch b.
    • evalWindows

      public int evalWindows(int seq)
      Number of non-overlapping evaluation windows of seq tokens.
    • evalWindow

      public void evalWindow(int w, int seq, int[] inputs, int[] targets, int at)
      Evaluation window w: inputs and targets [seq].
    • token

      public int token(int i)
      Token i.