Package ai.beestream.jitnn.nn.text


package ai.beestream.jitnn.nn.text
Vocabularies and training examples for the embedding models in ai.beestream.jitnn.nn.models.
  • Class
    Description
    Training examples for the embedding models, drawn on the host from sentences of word ids: skip-gram pairs, CBOW windows, negative samples from the unigram distribution raised to 3/4, and GloVe's distance-weighted co-occurrence counts.
    A Hugging Face tokenizer read from its tokenizer.json, for the two families jitNN's models use: WordPiece with the BERT normaliser and pre-tokeniser (BERT and its relatives), and byte-level BPE (GPT-2, RoBERTa).
    Encodings padded to the longest one: [batch, length] ids and type ids, and an attention mask of 1 for tokens and 0 for padding.
    The ids, type ids and tokens of one text (or pair), special tokens included.
    Word ids by descending frequency, the words seen at least minCount times.