Class MultiheadAttention
java.lang.Object
ai.beestream.jitnn.nn.Module
ai.beestream.jitnn.nn.MultiheadAttention
- All Implemented Interfaces:
AutoCloseable
Multi-head self-attention on [B, T, D] inputs, like PyTorch's
nn.MultiheadAttention
with batch_first: one fused q/k/v projection [3D, D], heads heads of D / heads
features, and an output projection. Optional causal masking and an additive mask broadcast to
[B, heads, T, T] (for example [B, 1, 1, T] for padding).-
Constructor Summary
ConstructorsConstructorDescriptionMultiheadAttention(int dim, int heads, boolean bias, long seed) Attention overdimfeatures inheadsheads (dimdivisible by it), with or without projection biases, initialised fromseed. -
Method Summary
Modifier and TypeMethodDescriptionApplies the module tox.Attention over x [B, T, D] with an optional additivemask(null for none) and optional causal masking.The output projection [dim, dim].qkv()The fused q/k/v input projection [3 dim, dim].Methods inherited from class Module
buffer, buffers, child, close, isClosed, isTraining, loadStateDict, parameter, parameterCount, parameters, stateDict, train, zeroGrad
-
Constructor Details
-
MultiheadAttention
public MultiheadAttention(int dim, int heads, boolean bias, long seed) Attention overdimfeatures inheadsheads (dimdivisible by it), with or without projection biases, initialised fromseed.
-
-
Method Details
-
qkv
The fused q/k/v input projection [3 dim, dim]. -
outProjection
The output projection [dim, dim]. -
forward
-
forward
-