Package ai.beestream.jitnn.nn.text
package ai.beestream.jitnn.nn.text
Vocabularies and training examples for the embedding models in
ai.beestream.jitnn.nn.models.-
ClassDescriptionTraining examples for the embedding models, drawn on the host from sentences of word ids: skip-gram pairs, CBOW windows, negative samples from the unigram distribution raised to 3/4, and GloVe's distance-weighted co-occurrence counts.A Hugging Face tokenizer read from its
tokenizer.json, for the two families jitNN's models use: WordPiece with the BERT normaliser and pre-tokeniser (BERT and its relatives), and byte-level BPE (GPT-2, RoBERTa).Encodings padded to the longest one:[batch, length]ids and type ids, and an attention mask of 1 for tokens and 0 for padding.The ids, type ids and tokens of one text (or pair), special tokens included.Word ids by descending frequency, the words seen at leastminCounttimes.