Class HostCollective
java.lang.Object
ai.beestream.jitnn.dist.HostCollective
- All Implemented Interfaces:
Collective
Collectives staged through host memory: each rank copies its tensor to the host, the reduction
runs on the host in rank order (so every rank gets bit-identical results, run after run), and
each rank copies the result back. It works with any mix of devices and backends (CUDA, OpenCL,
Metal, even different backends in one group); the price is a device-host round trip per call.
-
Nested Class Summary
Nested classes/interfaces inherited from interface Collective
Collective.ReduceOp -
Constructor Summary
ConstructorsConstructorDescriptionHostCollective(int worldSize) Collectives amongworldSizeranks of this process. -
Method Summary
Modifier and TypeMethodDescriptionvoidabort()Releases every rank waiting in a collective with an error (called when a rank fails).voidinput [n, ...] on every rank; output [worldSize * n, ...] receives every rank's input, in rank order.voidallReduce(int rank, Tensor tensor, Collective.ReduceOp op) Every rank's tensor replaced, in place, by the reduction over all ranks.voidbarrier(int rank) Blocks until every rank has called it.voidEvery rank's tensor replaced, in place, by rankroot's.voidreduceScatter(int rank, Tensor input, Tensor output, Collective.ReduceOp op) input [worldSize * n, ...] on every rank; output [n, ...] receives on rank r the reduction of every rank's r-th slice.intThe number of ranks, over every process.Methods inherited from class Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitMethods inherited from interface Collective
allReduced, finishAllReduce, firstRank, localRanks, startAllReduce
-
Constructor Details
-
HostCollective
public HostCollective(int worldSize) Collectives amongworldSizeranks of this process.
-
-
Method Details
-
worldSize
public int worldSize()Description copied from interface:CollectiveThe number of ranks, over every process.- Specified by:
worldSizein interfaceCollective
-
allReduce
Description copied from interface:CollectiveEvery rank's tensor replaced, in place, by the reduction over all ranks.- Specified by:
allReducein interfaceCollective
-
broadcast
Description copied from interface:CollectiveEvery rank's tensor replaced, in place, by rankroot's.- Specified by:
broadcastin interfaceCollective
-
reduceScatter
Description copied from interface:Collectiveinput [worldSize * n, ...] on every rank; output [n, ...] receives on rank r the reduction of every rank's r-th slice.- Specified by:
reduceScatterin interfaceCollective
-
allGather
Description copied from interface:Collectiveinput [n, ...] on every rank; output [worldSize * n, ...] receives every rank's input, in rank order.- Specified by:
allGatherin interfaceCollective
-
abort
public void abort()Releases every rank waiting in a collective with an error (called when a rank fails).- Specified by:
abortin interfaceCollective
-
barrier
public void barrier(int rank) Description copied from interface:CollectiveBlocks until every rank has called it.- Specified by:
barrierin interfaceCollective
-