Class TextCorpus
java.lang.Object
ai.beestream.jitnn.data.TextCorpus
A byte-level text corpus for language-model gates: every byte is a token (vocabulary 256).
The first 90% trains and the rest evaluates. Training windows are laid out in order without
shuffling, so another framework can reproduce the exact batches (scripts/pytorch/datasets.py).
-
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final intThe vocabulary size: one token per byte value. -
Method Summary
Modifier and TypeMethodDescriptionvoidevalWindow(int w, int seq, int[] inputs, int[] targets, int at) Evaluation windoww: inputs and targets [seq].intevalWindows(int seq) Number of non-overlapping evaluation windows ofseqtokens.intlength()The number of tokens.static TextCorpusThe bytes offileas a corpus.inttoken(int i) Tokeni.voidtrainBatch(int b, int rows, int seq, int[] inputs, int[] targets) Fills inputs and next-token targets [rows, seq] for training batchb.inttrainStart(int b, int rows, int seq, int r) Start of training rowrof batchb: consecutive windows, wrapping around the training part.
-
Field Details
-
VOCAB
public static final int VOCABThe vocabulary size: one token per byte value.- See Also:
-
-
Method Details
-
load
The bytes offileas a corpus.- Throws:
IOException
-
length
public int length()The number of tokens. -
trainStart
public int trainStart(int b, int rows, int seq, int r) Start of training rowrof batchb: consecutive windows, wrapping around the training part. -
trainBatch
public void trainBatch(int b, int rows, int seq, int[] inputs, int[] targets) Fills inputs and next-token targets [rows, seq] for training batchb. -
evalWindows
public int evalWindows(int seq) Number of non-overlapping evaluation windows ofseqtokens. -
evalWindow
public void evalWindow(int w, int seq, int[] inputs, int[] targets, int at) Evaluation windoww: inputs and targets [seq]. -
token
public int token(int i) Tokeni.
-