Uses of Package
ai.beestream.jitnn.nn.text

Packages that use ai.beestream.jitnn.nn.text
Package
Description
Vocabularies and training examples for the embedding models in ai.beestream.jitnn.nn.models.
  • Class
    Description
    A Hugging Face tokenizer read from its tokenizer.json, for the two families jitNN's models use: WordPiece with the BERT normaliser and pre-tokeniser (BERT and its relatives), and byte-level BPE (GPT-2, RoBERTa).
    Encodings padded to the longest one: [batch, length] ids and type ids, and an attention mask of 1 for tokens and 0 for padding.
    The ids, type ids and tokens of one text (or pair), special tokens included.
    Word ids by descending frequency, the words seen at least minCount times.