Interface Collective

All Known Implementing Classes:
HostCollective, NcclCollective

public interface Collective
Collective communication among the ranks of a group, each rank one thread and usually one device (after torch.distributed). Every rank calls the same collectives in the same order with tensors of the same shape; the calls block until all ranks have arrived.
  • Method Details

    • worldSize

      int worldSize()
      The number of ranks, over every process.
    • firstRank

      default int firstRank()
      The global rank of this process's first rank (0 in a single process).
    • localRanks

      default int localRanks()
      The number of ranks in this process.
    • allReduce

      void allReduce(int rank, Tensor tensor, Collective.ReduceOp op)
      Every rank's tensor replaced, in place, by the reduction over all ranks.
    • broadcast

      void broadcast(int rank, Tensor tensor, int root)
      Every rank's tensor replaced, in place, by rank root's.
    • reduceScatter

      void reduceScatter(int rank, Tensor input, Tensor output, Collective.ReduceOp op)
      input [worldSize * n, ...] on every rank; output [n, ...] receives on rank r the reduction of every rank's r-th slice.
    • allGather

      void allGather(int rank, Tensor input, Tensor output)
      input [n, ...] on every rank; output [worldSize * n, ...] receives every rank's input, in rank order.
    • barrier

      void barrier(int rank)
      Blocks until every rank has called it.
    • allReduced

      default Tensor[] allReduced(int rank, Tensor[] tensors, Collective.ReduceOp op)
      The reduction over all ranks of each tensor. A host collective reduces the tensors in place and returns them; a device collective (NcclCollective) records the reduction and returns new tensors, without waiting.
    • startAllReduce

      default void startAllReduce(int rank, Tensor[] tensors, Tensor buffer, Collective.ReduceOp op, int tag)
      Starts the reduction over all ranks of tensors, packed into the persistent buffer (exactly their total size), so that it runs while the operators recorded after it do: NCCL runs it on a stream of its own. finishAllReduce(int, Tensor[], Tensor, Collective.ReduceOp, int) with the same tag returns the results. Tags tell apart reductions in flight at the same time. A collective that cannot overlap does nothing here and reduces in finishAllReduce(int, Tensor[], Tensor, Collective.ReduceOp, int).
    • finishAllReduce

      default Tensor[] finishAllReduce(int rank, Tensor[] tensors, Tensor buffer, Collective.ReduceOp op, int tag)
    • abort

      default void abort()
      Releases the ranks waiting in a collective after another rank failed.