Interface Collective
- All Known Implementing Classes:
HostCollective, NcclCollective
public interface Collective
Collective communication among the ranks of a group, each rank one thread and usually one
device (after torch.distributed). Every rank calls the same collectives in the same order with
tensors of the same shape; the calls block until all ranks have arrived.
-
Nested Class Summary
Nested ClassesModifier and TypeInterfaceDescriptionstatic enumHow a reduction combines the ranks' tensors, element-wise. -
Method Summary
Modifier and TypeMethodDescriptiondefault voidabort()Releases the ranks waiting in a collective after another rank failed.voidinput [n, ...] on every rank; output [worldSize * n, ...] receives every rank's input, in rank order.voidallReduce(int rank, Tensor tensor, Collective.ReduceOp op) Every rank's tensor replaced, in place, by the reduction over all ranks.default Tensor[]allReduced(int rank, Tensor[] tensors, Collective.ReduceOp op) The reduction over all ranks of each tensor.voidbarrier(int rank) Blocks until every rank has called it.voidEvery rank's tensor replaced, in place, by rankroot's.default Tensor[]finishAllReduce(int rank, Tensor[] tensors, Tensor buffer, Collective.ReduceOp op, int tag) The results ofstartAllReduce(int, Tensor[], Tensor, Collective.ReduceOp, int)(same tensors, buffer and tag), asallReduced(int, Tensor[], Collective.ReduceOp)returns them.default intThe global rank of this process's first rank (0 in a single process).default intThe number of ranks in this process.voidreduceScatter(int rank, Tensor input, Tensor output, Collective.ReduceOp op) input [worldSize * n, ...] on every rank; output [n, ...] receives on rank r the reduction of every rank's r-th slice.default voidstartAllReduce(int rank, Tensor[] tensors, Tensor buffer, Collective.ReduceOp op, int tag) Starts the reduction over all ranks oftensors, packed into the persistentbuffer(exactly their total size), so that it runs while the operators recorded after it do: NCCL runs it on a stream of its own.intThe number of ranks, over every process.
-
Method Details
-
worldSize
int worldSize()The number of ranks, over every process. -
firstRank
default int firstRank()The global rank of this process's first rank (0 in a single process). -
localRanks
default int localRanks()The number of ranks in this process. -
allReduce
Every rank's tensor replaced, in place, by the reduction over all ranks. -
broadcast
Every rank's tensor replaced, in place, by rankroot's. -
reduceScatter
input [worldSize * n, ...] on every rank; output [n, ...] receives on rank r the reduction of every rank's r-th slice. -
allGather
-
barrier
void barrier(int rank) Blocks until every rank has called it. -
allReduced
The reduction over all ranks of each tensor. A host collective reduces the tensors in place and returns them; a device collective (NcclCollective) records the reduction and returns new tensors, without waiting. -
startAllReduce
default void startAllReduce(int rank, Tensor[] tensors, Tensor buffer, Collective.ReduceOp op, int tag) Starts the reduction over all ranks oftensors, packed into the persistentbuffer(exactly their total size), so that it runs while the operators recorded after it do: NCCL runs it on a stream of its own.finishAllReduce(int, Tensor[], Tensor, Collective.ReduceOp, int)with the sametagreturns the results. Tags tell apart reductions in flight at the same time. A collective that cannot overlap does nothing here and reduces infinishAllReduce(int, Tensor[], Tensor, Collective.ReduceOp, int). -
finishAllReduce
default Tensor[] finishAllReduce(int rank, Tensor[] tensors, Tensor buffer, Collective.ReduceOp op, int tag) The results ofstartAllReduce(int, Tensor[], Tensor, Collective.ReduceOp, int)(same tensors, buffer and tag), asallReduced(int, Tensor[], Collective.ReduceOp)returns them. -
abort
default void abort()Releases the ranks waiting in a collective after another rank failed.
-