Class Tokenizer

java.lang.Object
ai.beestream.jitnn.nn.text.Tokenizer

public final class Tokenizer extends Object
A Hugging Face tokenizer read from its tokenizer.json, for the two families jitNN's models use: WordPiece with the BERT normaliser and pre-tokeniser (BERT and its relatives), and byte-level BPE (GPT-2, RoBERTa). Added tokens such as [MASK] or <|endoftext|> are matched in the raw text first; the post-processor (template, BERT or RoBERTa) adds the special tokens and type ids. Other tokenizer types are rejected when loading.
Tokenizer tok = Tokenizer.load(Path.of("bert-base-uncased"));
Tokenizer.Batch batch = tok.encodeBatch(List.of("Hello world!", "A longer sentence."), 128);
Tensor hidden = bert.encode(batch.ids(), batch.typeIds(), batch.additiveMask());
  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static final record 
    Encodings padded to the longest one: [batch, length] ids and type ids, and an attention mask of 1 for tokens and 0 for padding.
    static final record 
    The ids, type ids and tokens of one text (or pair), special tokens included.
  • Method Summary

    Modifier and Type
    Method
    Description
    decode(int[] ids, boolean skipSpecial)
    The text of ids; with skipSpecial, special added tokens are left out.
    encode(String text)
    Encodes one text, with the post-processor's special tokens.
    Encodes a pair of texts (sentence A, sentence B).
    encode(String a, String b, int maxLength)
    Encodes a (and b when not null) into at most maxLength ids, special tokens included, cutting the longer sequence first as Hugging Face's longest-first truncation.
    encodeBatch(List<String> texts, int maxLength)
    Encodes texts, each cut to maxLength, padded to the longest with padId().
    int
    id(String token)
    The id of token, or -1.
    static Tokenizer
    load(Path path)
    Loads tokenizer.json, or the one inside directory path.
    int
    The id used for padding: the file's padding id, else [PAD] or <pad>, else GPT-2's <|endoftext|>, else 0.
    token(int id)
    The token with id id, or null.
    int[]
    The ids of text without special tokens.
    int
    The number of tokens in the vocabulary.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Method Details

    • load

      public static Tokenizer load(Path path) throws IOException
      Loads tokenizer.json, or the one inside directory path.
      Throws:
      IOException
    • encode

      public Tokenizer.Encoding encode(String text)
      Encodes one text, with the post-processor's special tokens.
    • encode

      public Tokenizer.Encoding encode(String a, String b)
      Encodes a pair of texts (sentence A, sentence B).
    • encode

      public Tokenizer.Encoding encode(String a, String b, int maxLength)
      Encodes a (and b when not null) into at most maxLength ids, special tokens included, cutting the longer sequence first as Hugging Face's longest-first truncation.
    • encodeBatch

      public Tokenizer.Batch encodeBatch(List<String> texts, int maxLength)
      Encodes texts, each cut to maxLength, padded to the longest with padId().
    • tokenize

      public int[] tokenize(String text)
      The ids of text without special tokens.
    • decode

      public String decode(int[] ids, boolean skipSpecial)
      The text of ids; with skipSpecial, special added tokens are left out.
    • vocabSize

      public int vocabSize()
      The number of tokens in the vocabulary.
    • id

      public int id(String token)
      The id of token, or -1.
    • token

      public String token(int id)
      The token with id id, or null.
    • padId

      public int padId()
      The id used for padding: the file's padding id, else [PAD] or <pad>, else GPT-2's <|endoftext|>, else 0. Padded positions are masked out by Tokenizer.Batch.attention.