Package ai.beestream.jitnn.nn
package ai.beestream.jitnn.nn
Neural-network modules after PyTorch's
torch.nn: Module and
Sequential, linear and convolution layers, pooling, normalisation,
attention, embeddings, recurrent layers (LSTM, GRU), graph convolution, activations, and data-parallel
training (DataParallel).-
ClassDescriptionAdaptive average pooling to fixed output sizes (1 to 3 spatial axes), PyTorch's nn.AdaptiveAvgPool1d/2d/3d.Adaptive max pooling to fixed output sizes (1 to 3 spatial axes), PyTorch's nn.AdaptiveMaxPool1d/2d/3d.Residual connection: relu(main(x) + shortcut(x)) or main(x) + shortcut(x).Average pooling over 1-D windows, PyTorch's nn.AvgPool1d.Average pooling; padding counts in the divisor, as PyTorch's default.Average pooling over 3-D windows, PyTorch's nn.AvgPool3d.Batch normalisation over the channels of NCHW (or NC) input, PyTorch defaults: weight 1, bias 0, running mean 0 and variance 1, momentum 0.1, eps 1e-5.Runs models on inputs padded to
Bucketssizes, so requests of changing batch and sequence length reuse a few compiled plans, and returns results for the real rows and positions only.CELU withalpha(PyTorch's default 1).1-D convolution over [N, C, ...] with a cubic kernel, optionally grouped, PyTorch's initialisation: weight and bias uniform in [-1/sqrt(fan_in), 1/sqrt(fan_in)), fan_in = in * kernel^1.2-D convolution over NCHW with a square kernel, PyTorch'snn.Conv2dsemantics and initialisation: weight and bias uniform in [-1/sqrt(fan_in), 1/sqrt(fan_in)), fan_in = in/groups * k * k.3-D convolution over [N, C, ...] with a cubic kernel, optionally grouped, PyTorch's initialisation: weight and bias uniform in [-1/sqrt(fan_in), 1/sqrt(fan_in)), fan_in = in * kernel^3.Transposed 2-D convolution over NCHW with a square kernel, PyTorch'snn.ConvTranspose2dsemantics and initialisation: weight [in, out/groups, k, k] and bias uniform in [-1/sqrt(fan_in), 1/sqrt(fan_in)), fan_in = out/groups * k * k.x^3.Data-parallel training with one model replica per rank (seeProcessGroup): broadcast rank 0's parameters once, then after each backward pass average the gradients over the ranks before the optimiser step, so every replica takes the same step, as PyTorch's DistributedDataParallel does.Inverted dropout in training mode, identity in evaluation mode.ELU with slopealphafor negative inputs (PyTorch's default 1).A lookup table ofcountvectors of sizedim, applied to INT32 indices.EmbeddingBag: bags of INT32 indices (one flat list plus bag offsets) to the sum, mean or max of their embedding rows, PyTorch's nn.EmbeddingBag with standard normal initial weights.[N, d1, d2, ...] to [N, d1 * d2 * ...], a view.[N, C, H, W] to [N, C], the mean over each feature map (adaptive average pooling to 1x1, flattened).Graph convolution (Kipf and Welling, 2017), PyTorch Geometric's GCNConv:  (x W^T) + b, with  the normalised adjacencySparseMatrix.gcn(int, int[], int[])of the graph, given with each call since one layer serves many graphs. x [nodes, in] gives [nodes, out].GroupNorm over [N, C, ...] with a per-channel affine (weight 1, bias 0 at the start), PyTorch's nn.GroupNorm.A stack of GRU layers over [T, B, I] sequences (time first), optionally bidirectional, PyTorch'snn.GRUequations, gate order, parameter order and initialisation: every weight and bias uniform in [-1/sqrt(H), 1/sqrt(H)).Output sequence of the last layer [T, B, directions H] and the final hidden states, [layers directions, B, H] each.HardSigmoid: clamp(x / 6 + 1/2, 0, 1).HardSwish: x hardSigmoid(x).HardTanh: x clamped to [min, max].InstanceNorm over [N, C, ...]: each sample and channel normalised over its positions, PyTorch's nn.InstanceNorm1d/2d/3d without running statistics; affine optional (PyTorch's default: none).LayerNorm over the last axis, with weight 1 and bias 0 initially, like PyTorch'snn.LayerNorm.LeakyReLU with a fixed negative slope (PyTorch's default 0.01).y = x W^T + b with W of shape [out, in].Local response normalisation across channels, PyTorch's nn.LocalResponseNorm.A stack of LSTM layers over [T, B, I] sequences (time first), optionally bidirectional, PyTorch'snn.LSTMequations, gate order, parameter order and initialisation: every weight and bias uniform in [-1/sqrt(H), 1/sqrt(H)).Output sequence of the last layer [T, B, directions H] and the final hidden states and cell states, [layers directions, B, H] each.Max pooling over 1-D windows, PyTorch's nn.MaxPool1d.Max pooling over square windows of NCHW input, PyTorch's nn.MaxPool2d.Max pooling over 3-D windows, PyTorch's nn.MaxPool3d.Mish: x tanh(softplus(x)).A differentiable building block with parameters and child modules.Multi-head self-attention on [B, T, D] inputs, like PyTorch'snn.MultiheadAttentionwith batch_first: one fused q/k/v projection [3D, D],headsheads of D / heads features, and an output projection.Padding of the spatial axes: constant, reflect or replicate (PyTorch's nn.ConstantPadNd, ReflectionPadNd, ReplicationPadNd).PReLU with a learnable slope, one shared or one per channel, initialised toinit(PyTorch: 0.25).ReLU: max(x, 0).ReLU6: min(max(x, 0), 6).RMSNorm over the last axis, with weight 1 initially, like PyTorch'snn.RMSNorm.RReLU: random slopes in [lower, upper) while training (a fresh seed per call, so a replayed step draws new slopes without recompiling), the mean slope in evaluation.SELU with PyTorch's constants.Applies its modules in order.The logistic sigmoid 1 / (1 + exp(-x)).Softplus: log(1 + exp(x)).Softsign: x / (1 + |x|).Batch normalisation whose training statistics are those of the batch over all data-parallel ranks, as PyTorch's SyncBatchNorm: each rank sums its x and x^2 per channel, one all-reduce gives the global mean and variance, and the backward pass all-reduces their gradients the same way.The hyperbolic tangent.Keras's ThresholdedReLU: x where x > theta, else 0.Nearest or bilinear upsampling of NCHW by a scale factor, PyTorch'snn.Upsample.