Class Llama

java.lang.Object
ai.beestream.jitnn.nn.Module
ai.beestream.jitnn.nn.models.Llama
All Implemented Interfaces:
AutoCloseable

public final class Llama extends Module
A Llama-style decoder: token embedding, pre-RMSNorm blocks of causal attention with rotary position embedding (rotate-half form) and a SwiGLU feed-forward, a final RMSNorm and an untied output projection; no biases. Seeds are handed out in construction order (scripts/pytorch/transformers.py builds the same model). With fewer key/value heads than query heads the attention is grouped-query (GQA), as Llama 2 70B and Llama 3; fromHuggingFace(Path) builds a model from a Hugging Face Llama checkpoint.
  • Constructor Summary

    Constructors
    Constructor
    Description
    Llama(int vocab, int dim, int heads, int layers, int hidden, float ropeBase, long seed)
    A decoder over vocab tokens with layers blocks of dim features in heads heads, a feed-forward of hidden features and rotary base ropeBase, initialised from seed.
    Llama(int vocab, int dim, int heads, int kvHeads, int layers, int hidden, float ropeBase, float eps, boolean tied, boolean qkvBias, long seed)
    A Llama with kvHeads key/value heads (grouped-query attention when fewer than heads), RMSNorm epsilon eps, the output projection tied to the token embedding when tied, and biases on the q, k and v projections when qkvBias.
  • Method Summary

    Modifier and Type
    Method
    Description
    checkpointing(boolean on)
    With on, training keeps no activations inside the blocks: each block runs again in the backward pass (Grad.checkpoint(Function, Tensor...)), for less memory and about a third more compute.
    Logits [B, T, vocab] for INT32 tokens [B, T].
    static Llama
    A Llama from a Hugging Face checkpoint directory (config.json and safetensors, as HfCheckpoint): model_type llama (or mistral without a sliding window, qwen2 with its q, k, v biases), grouped-query attention, tied or separate output weights, and linear or Llama 3.1 RoPE scaling.
    Rescales the RoPE frequencies of every block (Llama 3.1's rope_scaling); null for none.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Constructor Details

    • Llama

      public Llama(int vocab, int dim, int heads, int layers, int hidden, float ropeBase, long seed)
      A decoder over vocab tokens with layers blocks of dim features in heads heads, a feed-forward of hidden features and rotary base ropeBase, initialised from seed.
    • Llama

      public Llama(int vocab, int dim, int heads, int kvHeads, int layers, int hidden, float ropeBase, float eps, boolean tied, boolean qkvBias, long seed)
      A Llama with kvHeads key/value heads (grouped-query attention when fewer than heads), RMSNorm epsilon eps, the output projection tied to the token embedding when tied, and biases on the q, k and v projections when qkvBias.
  • Method Details

    • checkpointing

      public Llama checkpointing(boolean on)
      With on, training keeps no activations inside the blocks: each block runs again in the backward pass (Grad.checkpoint(Function, Tensor...)), for less memory and about a third more compute.
    • ropeScaling

      public Llama ropeScaling(RopeScaling scaling)
      Rescales the RoPE frequencies of every block (Llama 3.1's rope_scaling); null for none.
    • fromHuggingFace

      public static Llama fromHuggingFace(Path dir) throws IOException
      A Llama from a Hugging Face checkpoint directory (config.json and safetensors, as HfCheckpoint): model_type llama (or mistral without a sliding window, qwen2 with its q, k, v biases), grouped-query attention, tied or separate output weights, and linear or Llama 3.1 RoPE scaling. A head dimension other than hidden_size / heads is not supported.
      Throws:
      IOException
    • forward

      public Tensor forward(Tensor idx)
      Logits [B, T, vocab] for INT32 tokens [B, T].
      Specified by:
      forward in class Module