Class Tokenizer
java.lang.Object
ai.beestream.jitnn.nn.text.Tokenizer
A Hugging Face tokenizer read from its
tokenizer.json, for the two families jitNN's models
use: WordPiece with the BERT normaliser and pre-tokeniser (BERT and its relatives), and
byte-level BPE (GPT-2, RoBERTa). Added tokens such as [MASK] or <|endoftext|> are
matched in the raw text first; the post-processor (template, BERT or RoBERTa) adds the special
tokens and type ids. Other tokenizer types are rejected when loading.
Tokenizer tok = Tokenizer.load(Path.of("bert-base-uncased"));
Tokenizer.Batch batch = tok.encodeBatch(List.of("Hello world!", "A longer sentence."), 128);
Tensor hidden = bert.encode(batch.ids(), batch.typeIds(), batch.additiveMask());
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionstatic final recordEncodings padded to the longest one:[batch, length]ids and type ids, and an attention mask of 1 for tokens and 0 for padding.static final recordThe ids, type ids and tokens of one text (or pair), special tokens included. -
Method Summary
Modifier and TypeMethodDescriptiondecode(int[] ids, boolean skipSpecial) The text ofids; withskipSpecial, special added tokens are left out.Encodes one text, with the post-processor's special tokens.Encodes a pair of texts (sentence A, sentence B).Encodesa(andbwhen not null) into at mostmaxLengthids, special tokens included, cutting the longer sequence first as Hugging Face's longest-first truncation.encodeBatch(List<String> texts, int maxLength) intThe id oftoken, or -1.static TokenizerLoadstokenizer.json, or the one inside directorypath.intpadId()The id used for padding: the file's padding id, else[PAD]or<pad>, else GPT-2's<|endoftext|>, else 0.token(int id) The token with idid, or null.int[]The ids oftextwithout special tokens.intThe number of tokens in the vocabulary.
-
Method Details
-
load
Loadstokenizer.json, or the one inside directorypath.- Throws:
IOException
-
encode
Encodes one text, with the post-processor's special tokens. -
encode
Encodes a pair of texts (sentence A, sentence B). -
encode
Encodesa(andbwhen not null) into at mostmaxLengthids, special tokens included, cutting the longer sequence first as Hugging Face's longest-first truncation. -
encodeBatch
-
tokenize
The ids oftextwithout special tokens. -
decode
The text ofids; withskipSpecial, special added tokens are left out. -
vocabSize
public int vocabSize()The number of tokens in the vocabulary. -
id
The id oftoken, or -1. -
token
The token with idid, or null. -
padId
public int padId()The id used for padding: the file's padding id, else[PAD]or<pad>, else GPT-2's<|endoftext|>, else 0. Padded positions are masked out byTokenizer.Batch.attention.
-