Package ai.beestream.jitnn.dist


package ai.beestream.jitnn.dist
Several devices: collectives (all-reduce, broadcast, reduce-scatter, all-gather) among ranks, staged through the host (HostCollective) or run by NCCL on NVIDIA GPUs (NcclCollective), and ProcessGroup, which runs one rank per device.
  • Class
    Description
    Collective communication among the ranks of a group, each rank one thread and usually one device (after torch.distributed).
    How a reduction combines the ranks' tensors, element-wise.
    Collectives staged through host memory: each rank copies its tensor to the host, the reduction runs on the host in rank order (so every rank gets bit-identical results, run after run), and each rank copies the result back.
    Collectives on the GPUs, through NCCL (TornadoVM's tornado-nccl): one rank per CUDA device, in one process.
    Runs one rank per device in this process: rank firstRank + r is a thread whose plans run on devices[r] (see Engine.onDevice(String, Supplier)), and all ranks share one Collective.