Class GPT2

java.lang.Object
ai.beestream.jitnn.nn.Module
ai.beestream.jitnn.nn.models.GPT2
All Implemented Interfaces:
AutoCloseable

public final class GPT2 extends Module
A GPT-2 style decoder: token and learned position embeddings, pre-LayerNorm blocks of causal multi-head attention and a GELU (tanh form) MLP of 4 x dim, a final LayerNorm, and logits from the token embedding (tied weights). Embeddings start uniform with standard deviation 0.02 and linear layers as Linear, with seeds handed out in construction order (scripts/pytorch/transformers.py builds the same model).
  • Constructor Details

    • GPT2

      public GPT2(int vocab, int context, int dim, int heads, int layers, long seed)
      A decoder over vocab tokens and up to context positions, with layers blocks of dim features in heads heads, initialised from seed.
  • Method Details

    • checkpointing

      public GPT2 checkpointing(boolean on)
      With on, training keeps no activations inside the blocks: each block runs again in the backward pass (Grad.checkpoint(Function, Tensor...)), for less memory and about a third more compute.
    • fromHuggingFace

      public static GPT2 fromHuggingFace(Path dir) throws IOException
      GPT-2 from a Hugging Face model directory (gpt2, gpt2-medium, ... or a fine-tuned copy): config.json and safetensors weights, as GPT2Model or GPT2LMHeadModel save them. The language-model head is the tied token embedding, as in GPT-2.
      Throws:
      IOException
    • forward

      public Tensor forward(Tensor idx)
      Logits [B, T, vocab] for INT32 tokens [B, T].
      Specified by:
      forward in class Module