Class GPT2
java.lang.Object
ai.beestream.jitnn.nn.Module
ai.beestream.jitnn.nn.models.GPT2
- All Implemented Interfaces:
AutoCloseable
A GPT-2 style decoder: token and learned position embeddings, pre-LayerNorm blocks of causal
multi-head attention and a GELU (tanh form) MLP of 4 x dim, a final LayerNorm, and logits from
the token embedding (tied weights). Embeddings start uniform with standard deviation 0.02 and
linear layers as
Linear, with seeds handed out in construction order
(scripts/pytorch/transformers.py builds the same model).-
Constructor Summary
ConstructorsConstructorDescriptionGPT2(int vocab, int context, int dim, int heads, int layers, long seed) A decoder overvocabtokens and up tocontextpositions, withlayersblocks ofdimfeatures inheadsheads, initialised fromseed. -
Method Summary
Modifier and TypeMethodDescriptioncheckpointing(boolean on) Withon, training keeps no activations inside the blocks: each block runs again in the backward pass (Grad.checkpoint(Function, Tensor...)), for less memory and about a third more compute.Logits [B, T, vocab] for INT32 tokens [B, T].static GPT2fromHuggingFace(Path dir) GPT-2 from a Hugging Face model directory (gpt2,gpt2-medium, ... or a fine-tuned copy):config.jsonand safetensors weights, asGPT2ModelorGPT2LMHeadModelsave them.Methods inherited from class Module
buffer, buffers, child, close, isClosed, isTraining, loadStateDict, parameter, parameterCount, parameters, stateDict, train, zeroGrad
-
Constructor Details
-
GPT2
public GPT2(int vocab, int context, int dim, int heads, int layers, long seed) A decoder overvocabtokens and up tocontextpositions, withlayersblocks ofdimfeatures inheadsheads, initialised fromseed.
-
-
Method Details
-
checkpointing
Withon, training keeps no activations inside the blocks: each block runs again in the backward pass (Grad.checkpoint(Function, Tensor...)), for less memory and about a third more compute. -
fromHuggingFace
GPT-2 from a Hugging Face model directory (gpt2,gpt2-medium, ... or a fine-tuned copy):config.jsonand safetensors weights, asGPT2ModelorGPT2LMHeadModelsave them. The language-model head is the tied token embedding, as in GPT-2.- Throws:
IOException
-
forward
-