Class MultiheadAttention

java.lang.Object
ai.beestream.jitnn.nn.Module
ai.beestream.jitnn.nn.MultiheadAttention
All Implemented Interfaces:
AutoCloseable

public final class MultiheadAttention extends Module
Multi-head self-attention on [B, T, D] inputs, like PyTorch's nn.MultiheadAttention with batch_first: one fused q/k/v projection [3D, D], heads heads of D / heads features, and an output projection. Optional causal masking and an additive mask broadcast to [B, heads, T, T] (for example [B, 1, 1, T] for padding).
  • Constructor Details

    • MultiheadAttention

      public MultiheadAttention(int dim, int heads, boolean bias, long seed)
      Attention over dim features in heads heads (dim divisible by it), with or without projection biases, initialised from seed.
  • Method Details

    • qkv

      public Linear qkv()
      The fused q/k/v input projection [3 dim, dim].
    • outProjection

      public Linear outProjection()
      The output projection [dim, dim].
    • forward

      public Tensor forward(Tensor x)
      Description copied from class: Module
      Applies the module to x.
      Specified by:
      forward in class Module
    • forward

      public Tensor forward(Tensor x, Tensor mask, boolean causal)
      Attention over x [B, T, D] with an optional additive mask (null for none) and optional causal masking.