Class NcclCollective
java.lang.Object
ai.beestream.jitnn.dist.NcclCollective
- All Implemented Interfaces:
Collective, AutoCloseable
Collectives on the GPUs, through NCCL (TornadoVM's tornado-nccl): one rank per CUDA device, in
one process. Unlike
HostCollective the data never goes through the host, and the
collectives do not block: they are recorded into each rank's trace like any operator and run
inside the rank's plan (and its CUDA graph), so a training step's backward pass, gradient
all-reduce and optimiser step are one plan per rank, and the ranks' plans run at the same time.
Results are new tensors (allReduced(int, Tensor[], Collective.ReduceOp)) or persistent tensors written in place
(allReduce(int, Tensor, Collective.ReduceOp), broadcast(int, Tensor, int), the outputs of reduceScatter(int, Tensor, Tensor, Collective.ReduceOp) and
allGather(int, Tensor, Tensor)): parameters, buffers and optimiser state. Gradients averaged with
allReduced(int, Tensor[], Collective.ReduceOp) are packed into one buffer and reduced with one NCCL call.
try (NcclCollective nccl = new NcclCollective("1:0", "1:1")) {
ProcessGroup.run(new String[] { "1:0", "1:1" }, nccl, (rank, c) -> train(rank, c));
}
Close it after the plans that use it are released (Engine.reset()), as NCCL waits for
CUDA graphs that still hold the communicator. FP32 tensors only.
-
Nested Class Summary
Nested classes/interfaces inherited from interface Collective
Collective.ReduceOp -
Constructor Summary
ConstructorsConstructorDescriptionNcclCollective(String... devices) A communicator overdevices("driver:device" as fortornado --devices), rank r on devices[r]: distinct CUDA GPUs.NcclCollective(String uniqueId, int worldSize, int firstRank, String... devices) This process's share of a communicator over several processes (on one machine or many): ranks firstRank, firstRank + 1, ... ondevices, out ofworldSize. -
Method Summary
Modifier and TypeMethodDescriptionvoidabort()Releases ranks waiting in a collective after another rank failed; the communicator is unusable after it.voidPersistentoutput[world * n, ...] receives every rank's input [n, ...], in rank order.voidallReduce(int rank, Tensor tensor, Collective.ReduceOp op) Persistenttensorreplaced, on every rank, by the reduction over the ranks.Tensor[]allReduced(int rank, Tensor[] tensors, Collective.ReduceOp op) The reduction over the ranks of each tensor, as new tensors: one NCCL call for all of them.voidbarrier(int rank) Runs everything recorded so far, and returns once every rank has.voidPersistenttensorreplaced, on every rank, by rankroot's.voidclose()Tensor[]finishAllReduce(int rank, Tensor[] tensors, Tensor buffer, Collective.ReduceOp op, int tag) Waits for the all-reduce started withtagand unpacks it into new tensors.intThe global rank of this process's first rank (0 in a single process).intThe number of ranks in this process.voidreduceScatter(int rank, Tensor input, Tensor output, Collective.ReduceOp op) Persistentoutput[n, ...] receives the reduction of every rank's slice r of input [world * n, ...].voidstartAllReduce(int rank, Tensor[] tensors, Tensor buffer, Collective.ReduceOp op, int tag) Packstensorsinto persistentbufferand starts its all-reduce on NCCL's side stream (seeCollective.startAllReduce(int, Tensor[], Tensor, Collective.ReduceOp, int)).static StringuniqueId()A new NCCL unique id (base64) forNcclCollective(String, int, int, String...); this process then hosts NCCL's rendezvous.intThe number of ranks, over every process.
-
Constructor Details
-
NcclCollective
A communicator overdevices("driver:device" as fortornado --devices), rank r on devices[r]: distinct CUDA GPUs. -
NcclCollective
This process's share of a communicator over several processes (on one machine or many): ranks firstRank, firstRank + 1, ... ondevices, out ofworldSize. Every process passes the sameuniqueId, made once byuniqueId()and handed to the others by the application (a file, an environment variable, the command line). It returns once every rank of every process has joined; the process that made the id must stay alive until then.
-
-
Method Details
-
uniqueId
A new NCCL unique id (base64) forNcclCollective(String, int, int, String...); this process then hosts NCCL's rendezvous. -
firstRank
public int firstRank()Description copied from interface:CollectiveThe global rank of this process's first rank (0 in a single process).- Specified by:
firstRankin interfaceCollective
-
localRanks
public int localRanks()Description copied from interface:CollectiveThe number of ranks in this process.- Specified by:
localRanksin interfaceCollective
-
worldSize
public int worldSize()Description copied from interface:CollectiveThe number of ranks, over every process.- Specified by:
worldSizein interfaceCollective
-
allReduce
Persistenttensorreplaced, on every rank, by the reduction over the ranks.- Specified by:
allReducein interfaceCollective
-
broadcast
Persistenttensorreplaced, on every rank, by rankroot's.- Specified by:
broadcastin interfaceCollective
-
allReduced
The reduction over the ranks of each tensor, as new tensors: one NCCL call for all of them. A SUM or AVG of a single tensor is differentiable (its gradient is the same all-reduce of the output's gradient), as synchronised batch normalisation needs.- Specified by:
allReducedin interfaceCollective
-
startAllReduce
public void startAllReduce(int rank, Tensor[] tensors, Tensor buffer, Collective.ReduceOp op, int tag) Packstensorsinto persistentbufferand starts its all-reduce on NCCL's side stream (seeCollective.startAllReduce(int, Tensor[], Tensor, Collective.ReduceOp, int)).- Specified by:
startAllReducein interfaceCollective
-
finishAllReduce
public Tensor[] finishAllReduce(int rank, Tensor[] tensors, Tensor buffer, Collective.ReduceOp op, int tag) Waits for the all-reduce started withtagand unpacks it into new tensors.- Specified by:
finishAllReducein interfaceCollective
-
reduceScatter
Persistentoutput[n, ...] receives the reduction of every rank's slice r of input [world * n, ...].- Specified by:
reduceScatterin interfaceCollective
-
allGather
Persistentoutput[world * n, ...] receives every rank's input [n, ...], in rank order.- Specified by:
allGatherin interfaceCollective
-
barrier
public void barrier(int rank) Runs everything recorded so far, and returns once every rank has.- Specified by:
barrierin interfaceCollective
-
abort
public void abort()Releases ranks waiting in a collective after another rank failed; the communicator is unusable after it.- Specified by:
abortin interfaceCollective
-
close
public void close()- Specified by:
closein interfaceAutoCloseable
-