Class Bert
java.lang.Object
ai.beestream.jitnn.nn.Module
ai.beestream.jitnn.nn.models.Bert
- All Implemented Interfaces:
AutoCloseable
A BERT-style encoder for masked language modelling: token and learned position embeddings with
LayerNorm, post-LayerNorm blocks of bidirectional attention (with an additive padding mask)
and a GELU MLP of 4 x dim, and an MLM head (dense, GELU, LayerNorm, then the token embedding
as the decoder with its own bias). LayerNorm eps is 1e-12 as in BERT. Seeds are handed out in
construction order (scripts/pytorch/transformers.py builds the same model). The full form adds
BERT's token-type (segment) embeddings and pooler;
fromHuggingFace(Path) builds it.-
Constructor Summary
ConstructorsConstructorDescriptionBert(int vocab, int context, int dim, int heads, int layers, int hidden, int typeVocab, boolean pooler, float eps, long seed) As above with LayerNorm epsiloneps(RoBERTa uses 1e-5).Bert(int vocab, int context, int dim, int heads, int layers, int hidden, int typeVocab, boolean pooler, long seed) A BERT with an MLP ofhiddenunits,typeVocabtoken types (0 for none) and, optionally, the pooler (dense and tanh on the first token).Bert(int vocab, int context, int dim, int heads, int layers, long seed) An encoder overvocabtokens and up tocontextpositions, withlayersblocks ofdimfeatures inheadsheads, initialised fromseed. -
Method Summary
Modifier and TypeMethodDescriptioncheckpointing(boolean on) Withon, training keeps no activations inside the blocks: each block runs again in the backward pass (Grad.checkpoint(Function, Tensor...)), for less memory and about a third more compute.Hidden states [B, T, dim];maskis additive, broadcast to [B, heads, T, T] (e.g.Hidden states [B, T, dim] with INT32 token types [B, T] (null for all 0; ignored by a model without token-type embeddings) and an additivemask(null for none).Hidden states [B, T, dim] for INT32 tokens [B, T], without a padding mask.static BertfromHuggingFace(Path dir) A BERT-family encoder from a Hugging Face model directory: config.json and safetensors weights, bymodel_type:bert(bert-base-uncasedand other BERT checkpoints, asBertModel,BertForMaskedLMorBertForPreTraining);roberta(roberta-baseand the like): positions counted from the padding index + 1, skipping padding, as RoBERTa does; LayerNorm epsilon 1e-5;distilbert: no token types, no pooler. The MLM head is loaded when the checkpoint has it; otherwise it keeps its initial values.booleanTrue when the model has the pooler (a Hugging Face checkpoint without one, such as roberta-base, gives none).MLM logits [M, vocab] at the INT32 flat positionsrows[M] (indices into B * T): the hidden rows are gathered first, so the head runs on the masked positions only.The pooler output [B, dim]: tanh(dense(hidden state of the first token)); needs the pooler.Methods inherited from class Module
buffer, buffers, child, close, isClosed, isTraining, loadStateDict, parameter, parameterCount, parameters, stateDict, train, zeroGrad
-
Constructor Details
-
Bert
public Bert(int vocab, int context, int dim, int heads, int layers, long seed) An encoder overvocabtokens and up tocontextpositions, withlayersblocks ofdimfeatures inheadsheads, initialised fromseed. -
Bert
public Bert(int vocab, int context, int dim, int heads, int layers, int hidden, int typeVocab, boolean pooler, long seed) A BERT with an MLP ofhiddenunits,typeVocabtoken types (0 for none) and, optionally, the pooler (dense and tanh on the first token). The extra parameters are registered after the others, so the short constructor's parameters and seeds are unchanged. -
Bert
public Bert(int vocab, int context, int dim, int heads, int layers, int hidden, int typeVocab, boolean pooler, float eps, long seed) As above with LayerNorm epsiloneps(RoBERTa uses 1e-5).
-
-
Method Details
-
checkpointing
Withon, training keeps no activations inside the blocks: each block runs again in the backward pass (Grad.checkpoint(Function, Tensor...)), for less memory and about a third more compute. -
fromHuggingFace
A BERT-family encoder from a Hugging Face model directory: config.json and safetensors weights, bymodel_type:bert(bert-base-uncasedand other BERT checkpoints, asBertModel,BertForMaskedLMorBertForPreTraining);roberta(roberta-baseand the like): positions counted from the padding index + 1, skipping padding, as RoBERTa does; LayerNorm epsilon 1e-5;distilbert: no token types, no pooler.
HfCheckpoint.- Throws:
IOException
-
forward
-
encode
-
encode
-
mlmLogits
-
hasPooler
public boolean hasPooler()True when the model has the pooler (a Hugging Face checkpoint without one, such as roberta-base, gives none). -
pooled
-