Class NcclCollective

java.lang.Object
ai.beestream.jitnn.dist.NcclCollective
All Implemented Interfaces:
Collective, AutoCloseable

public final class NcclCollective extends Object implements Collective, AutoCloseable
Collectives on the GPUs, through NCCL (TornadoVM's tornado-nccl): one rank per CUDA device, in one process. Unlike HostCollective the data never goes through the host, and the collectives do not block: they are recorded into each rank's trace like any operator and run inside the rank's plan (and its CUDA graph), so a training step's backward pass, gradient all-reduce and optimiser step are one plan per rank, and the ranks' plans run at the same time.

Results are new tensors (allReduced(int, Tensor[], Collective.ReduceOp)) or persistent tensors written in place (allReduce(int, Tensor, Collective.ReduceOp), broadcast(int, Tensor, int), the outputs of reduceScatter(int, Tensor, Tensor, Collective.ReduceOp) and allGather(int, Tensor, Tensor)): parameters, buffers and optimiser state. Gradients averaged with allReduced(int, Tensor[], Collective.ReduceOp) are packed into one buffer and reduced with one NCCL call.

try (NcclCollective nccl = new NcclCollective("1:0", "1:1")) {
    ProcessGroup.run(new String[] { "1:0", "1:1" }, nccl, (rank, c) -> train(rank, c));
}

Close it after the plans that use it are released (Engine.reset()), as NCCL waits for CUDA graphs that still hold the communicator. FP32 tensors only.

  • Constructor Details

    • NcclCollective

      public NcclCollective(String... devices)
      A communicator over devices ("driver:device" as for tornado --devices), rank r on devices[r]: distinct CUDA GPUs.
    • NcclCollective

      public NcclCollective(String uniqueId, int worldSize, int firstRank, String... devices)
      This process's share of a communicator over several processes (on one machine or many): ranks firstRank, firstRank + 1, ... on devices, out of worldSize. Every process passes the same uniqueId, made once by uniqueId() and handed to the others by the application (a file, an environment variable, the command line). It returns once every rank of every process has joined; the process that made the id must stay alive until then.
  • Method Details

    • uniqueId

      public static String uniqueId()
      A new NCCL unique id (base64) for NcclCollective(String, int, int, String...); this process then hosts NCCL's rendezvous.
    • firstRank

      public int firstRank()
      Description copied from interface: Collective
      The global rank of this process's first rank (0 in a single process).
      Specified by:
      firstRank in interface Collective
    • localRanks

      public int localRanks()
      Description copied from interface: Collective
      The number of ranks in this process.
      Specified by:
      localRanks in interface Collective
    • worldSize

      public int worldSize()
      Description copied from interface: Collective
      The number of ranks, over every process.
      Specified by:
      worldSize in interface Collective
    • allReduce

      public void allReduce(int rank, Tensor tensor, Collective.ReduceOp op)
      Persistent tensor replaced, on every rank, by the reduction over the ranks.
      Specified by:
      allReduce in interface Collective
    • broadcast

      public void broadcast(int rank, Tensor tensor, int root)
      Persistent tensor replaced, on every rank, by rank root's.
      Specified by:
      broadcast in interface Collective
    • allReduced

      public Tensor[] allReduced(int rank, Tensor[] tensors, Collective.ReduceOp op)
      The reduction over the ranks of each tensor, as new tensors: one NCCL call for all of them. A SUM or AVG of a single tensor is differentiable (its gradient is the same all-reduce of the output's gradient), as synchronised batch normalisation needs.
      Specified by:
      allReduced in interface Collective
    • startAllReduce

      public void startAllReduce(int rank, Tensor[] tensors, Tensor buffer, Collective.ReduceOp op, int tag)
      Packs tensors into persistent buffer and starts its all-reduce on NCCL's side stream (see Collective.startAllReduce(int, Tensor[], Tensor, Collective.ReduceOp, int)).
      Specified by:
      startAllReduce in interface Collective
    • finishAllReduce

      public Tensor[] finishAllReduce(int rank, Tensor[] tensors, Tensor buffer, Collective.ReduceOp op, int tag)
      Waits for the all-reduce started with tag and unpacks it into new tensors.
      Specified by:
      finishAllReduce in interface Collective
    • reduceScatter

      public void reduceScatter(int rank, Tensor input, Tensor output, Collective.ReduceOp op)
      Persistent output [n, ...] receives the reduction of every rank's slice r of input [world * n, ...].
      Specified by:
      reduceScatter in interface Collective
    • allGather

      public void allGather(int rank, Tensor input, Tensor output)
      Persistent output [world * n, ...] receives every rank's input [n, ...], in rank order.
      Specified by:
      allGather in interface Collective
    • barrier

      public void barrier(int rank)
      Runs everything recorded so far, and returns once every rank has.
      Specified by:
      barrier in interface Collective
    • abort

      public void abort()
      Releases ranks waiting in a collective after another rank failed; the communicator is unusable after it.
      Specified by:
      abort in interface Collective
    • close

      public void close()
      Specified by:
      close in interface AutoCloseable