Class Bert

java.lang.Object
ai.beestream.jitnn.nn.Module
ai.beestream.jitnn.nn.models.Bert
All Implemented Interfaces:
AutoCloseable

public final class Bert extends Module
A BERT-style encoder for masked language modelling: token and learned position embeddings with LayerNorm, post-LayerNorm blocks of bidirectional attention (with an additive padding mask) and a GELU MLP of 4 x dim, and an MLM head (dense, GELU, LayerNorm, then the token embedding as the decoder with its own bias). LayerNorm eps is 1e-12 as in BERT. Seeds are handed out in construction order (scripts/pytorch/transformers.py builds the same model). The full form adds BERT's token-type (segment) embeddings and pooler; fromHuggingFace(Path) builds it.
  • Constructor Summary

    Constructors
    Constructor
    Description
    Bert(int vocab, int context, int dim, int heads, int layers, int hidden, int typeVocab, boolean pooler, float eps, long seed)
    As above with LayerNorm epsilon eps (RoBERTa uses 1e-5).
    Bert(int vocab, int context, int dim, int heads, int layers, int hidden, int typeVocab, boolean pooler, long seed)
    A BERT with an MLP of hidden units, typeVocab token types (0 for none) and, optionally, the pooler (dense and tanh on the first token).
    Bert(int vocab, int context, int dim, int heads, int layers, long seed)
    An encoder over vocab tokens and up to context positions, with layers blocks of dim features in heads heads, initialised from seed.
  • Method Summary

    Modifier and Type
    Method
    Description
    checkpointing(boolean on)
    With on, training keeps no activations inside the blocks: each block runs again in the backward pass (Grad.checkpoint(Function, Tensor...)), for less memory and about a third more compute.
    encode(Tensor idx, Tensor mask)
    Hidden states [B, T, dim]; mask is additive, broadcast to [B, heads, T, T] (e.g.
    encode(Tensor idx, Tensor tokenTypes, Tensor mask)
    Hidden states [B, T, dim] with INT32 token types [B, T] (null for all 0; ignored by a model without token-type embeddings) and an additive mask (null for none).
    Hidden states [B, T, dim] for INT32 tokens [B, T], without a padding mask.
    static Bert
    A BERT-family encoder from a Hugging Face model directory: config.json and safetensors weights, by model_type: bert (bert-base-uncased and other BERT checkpoints, as BertModel, BertForMaskedLM or BertForPreTraining); roberta (roberta-base and the like): positions counted from the padding index + 1, skipping padding, as RoBERTa does; LayerNorm epsilon 1e-5; distilbert: no token types, no pooler. The MLM head is loaded when the checkpoint has it; otherwise it keeps its initial values.
    boolean
    True when the model has the pooler (a Hugging Face checkpoint without one, such as roberta-base, gives none).
    mlmLogits(Tensor idx, Tensor mask, Tensor rows)
    MLM logits [M, vocab] at the INT32 flat positions rows [M] (indices into B * T): the hidden rows are gathered first, so the head runs on the masked positions only.
    pooled(Tensor hidden)
    The pooler output [B, dim]: tanh(dense(hidden state of the first token)); needs the pooler.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Constructor Details

    • Bert

      public Bert(int vocab, int context, int dim, int heads, int layers, long seed)
      An encoder over vocab tokens and up to context positions, with layers blocks of dim features in heads heads, initialised from seed.
    • Bert

      public Bert(int vocab, int context, int dim, int heads, int layers, int hidden, int typeVocab, boolean pooler, long seed)
      A BERT with an MLP of hidden units, typeVocab token types (0 for none) and, optionally, the pooler (dense and tanh on the first token). The extra parameters are registered after the others, so the short constructor's parameters and seeds are unchanged.
    • Bert

      public Bert(int vocab, int context, int dim, int heads, int layers, int hidden, int typeVocab, boolean pooler, float eps, long seed)
      As above with LayerNorm epsilon eps (RoBERTa uses 1e-5).
  • Method Details

    • checkpointing

      public Bert checkpointing(boolean on)
      With on, training keeps no activations inside the blocks: each block runs again in the backward pass (Grad.checkpoint(Function, Tensor...)), for less memory and about a third more compute.
    • fromHuggingFace

      public static Bert fromHuggingFace(Path dir) throws IOException
      A BERT-family encoder from a Hugging Face model directory: config.json and safetensors weights, by model_type:
      • bert (bert-base-uncased and other BERT checkpoints, as BertModel, BertForMaskedLM or BertForPreTraining);
      • roberta (roberta-base and the like): positions counted from the padding index + 1, skipping padding, as RoBERTa does; LayerNorm epsilon 1e-5;
      • distilbert: no token types, no pooler.
      The MLM head is loaded when the checkpoint has it; otherwise it keeps its initial values. Task heads (classification and the like) are left to the caller, who can read them with HfCheckpoint.
      Throws:
      IOException
    • forward

      public Tensor forward(Tensor idx)
      Hidden states [B, T, dim] for INT32 tokens [B, T], without a padding mask.
      Specified by:
      forward in class Module
    • encode

      public Tensor encode(Tensor idx, Tensor mask)
      Hidden states [B, T, dim]; mask is additive, broadcast to [B, heads, T, T] (e.g. [B, 1, 1, T]).
    • encode

      public Tensor encode(Tensor idx, Tensor tokenTypes, Tensor mask)
      Hidden states [B, T, dim] with INT32 token types [B, T] (null for all 0; ignored by a model without token-type embeddings) and an additive mask (null for none).
    • mlmLogits

      public Tensor mlmLogits(Tensor idx, Tensor mask, Tensor rows)
      MLM logits [M, vocab] at the INT32 flat positions rows [M] (indices into B * T): the hidden rows are gathered first, so the head runs on the masked positions only.
    • hasPooler

      public boolean hasPooler()
      True when the model has the pooler (a Hugging Face checkpoint without one, such as roberta-base, gives none).
    • pooled

      public Tensor pooled(Tensor hidden)
      The pooler output [B, dim]: tanh(dense(hidden state of the first token)); needs the pooler.