Package ai.beestream.jitnn.dist
package ai.beestream.jitnn.dist
Several devices: collectives (all-reduce, broadcast, reduce-scatter, all-gather) among ranks,
staged through the host (
HostCollective) or run by NCCL on NVIDIA GPUs
(NcclCollective), and ProcessGroup, which
runs one rank per device.-
ClassDescriptionCollective communication among the ranks of a group, each rank one thread and usually one device (after torch.distributed).How a reduction combines the ranks' tensors, element-wise.Collectives staged through host memory: each rank copies its tensor to the host, the reduction runs on the host in rank order (so every rank gets bit-identical results, run after run), and each rank copies the result back.Collectives on the GPUs, through NCCL (TornadoVM's tornado-nccl): one rank per CUDA device, in one process.Runs one rank per device in this process: rank firstRank + r is a thread whose plans run on devices[r] (see
Engine.onDevice(String, Supplier)), and all ranks share oneCollective.