Uses of Enum Class
ai.beestream.jitnn.dist.Collective.ReduceOp
Packages that use Collective.ReduceOp
Package
Description
Several devices: collectives (all-reduce, broadcast, reduce-scatter, all-gather) among ranks,
staged through the host (
HostCollective) or run by NCCL on NVIDIA GPUs
(NcclCollective), and ProcessGroup, which
runs one rank per device.-
Uses of Collective.ReduceOp in ai.beestream.jitnn.dist
Subclasses with type arguments of type Collective.ReduceOp in ai.beestream.jitnn.distModifier and TypeClassDescriptionstatic enumHow a reduction combines the ranks' tensors, element-wise.Methods in ai.beestream.jitnn.dist that return Collective.ReduceOpModifier and TypeMethodDescriptionstatic Collective.ReduceOpReturns the enum constant of this class with the specified name.static Collective.ReduceOp[]Collective.ReduceOp.values()Returns an array containing the constants of this enum class, in the order they are declared.Methods in ai.beestream.jitnn.dist with parameters of type Collective.ReduceOpModifier and TypeMethodDescriptionvoidCollective.allReduce(int rank, Tensor tensor, Collective.ReduceOp op) Every rank's tensor replaced, in place, by the reduction over all ranks.voidHostCollective.allReduce(int rank, Tensor tensor, Collective.ReduceOp op) voidNcclCollective.allReduce(int rank, Tensor tensor, Collective.ReduceOp op) Persistenttensorreplaced, on every rank, by the reduction over the ranks.default Tensor[]Collective.allReduced(int rank, Tensor[] tensors, Collective.ReduceOp op) The reduction over all ranks of each tensor.Tensor[]NcclCollective.allReduced(int rank, Tensor[] tensors, Collective.ReduceOp op) The reduction over the ranks of each tensor, as new tensors: one NCCL call for all of them.default Tensor[]Collective.finishAllReduce(int rank, Tensor[] tensors, Tensor buffer, Collective.ReduceOp op, int tag) The results ofCollective.startAllReduce(int, Tensor[], Tensor, Collective.ReduceOp, int)(same tensors, buffer and tag), asCollective.allReduced(int, Tensor[], Collective.ReduceOp)returns them.Tensor[]NcclCollective.finishAllReduce(int rank, Tensor[] tensors, Tensor buffer, Collective.ReduceOp op, int tag) Waits for the all-reduce started withtagand unpacks it into new tensors.voidCollective.reduceScatter(int rank, Tensor input, Tensor output, Collective.ReduceOp op) input [worldSize * n, ...] on every rank; output [n, ...] receives on rank r the reduction of every rank's r-th slice.voidHostCollective.reduceScatter(int rank, Tensor input, Tensor output, Collective.ReduceOp op) voidNcclCollective.reduceScatter(int rank, Tensor input, Tensor output, Collective.ReduceOp op) Persistentoutput[n, ...] receives the reduction of every rank's slice r of input [world * n, ...].default voidCollective.startAllReduce(int rank, Tensor[] tensors, Tensor buffer, Collective.ReduceOp op, int tag) Starts the reduction over all ranks oftensors, packed into the persistentbuffer(exactly their total size), so that it runs while the operators recorded after it do: NCCL runs it on a stream of its own.voidNcclCollective.startAllReduce(int rank, Tensor[] tensors, Tensor buffer, Collective.ReduceOp op, int tag) Packstensorsinto persistentbufferand starts its all-reduce on NCCL's side stream (seeCollective.startAllReduce(int, Tensor[], Tensor, Collective.ReduceOp, int)).