Class JitnnRuntime
java.lang.Object
ai.beestream.jitnn.JitnnRuntime
jitNN's runtime: where operators run, sync points, determinism, profiling and statistics.
Operators on Backend.TORNADO and Backend.CPU are recorded lazily and run at a
sync point: reading a tensor (Tensor.item(), Tensor.toFloatArray()) or
sync(Tensor...). Compiled plans are cached by the trace's structure, so a training step or an
inference call with the same shapes replays its plan. Options are set with JitnnConfig.
The backend and the device are per thread; everything else is per JVM.
-
Method Summary
Modifier and TypeMethodDescriptionstatic Backendbackend()This thread's backend.static StringThe OpenCL device on the host CPU (<driver>:<device>) where the CPU backend runs the operators it has no kernel for, or null when there is none.static StringThe CPU backend's operators by time since the last call (as printed at exit withjitnn.cpu.stats), then starts counting again; empty when nothing ran.cpuStats()The CPU backend's counters so far, per operator name:[calls, nanoseconds, calls that ran the reference implementation, calls that ran on the host's OpenCL device]; the times only withjitnn.cpu.stats=true.static booleanWhether deterministic results were asked for (setDeterministic(boolean),jitnn.deterministic).Withjitnn.profile=true: device time per operator task ("op#k", the k-th task of the operator) as[launches, nanoseconds], accumulated since the last call, which resets it.static longDevice memory for intermediates (activations, gradients and other temporaries, after buffer reuse) of the last plan this thread ran, in bytes; 0 on the reference backend.static long[]Withjitnn.profile=true:[device kernel ns, TornadoVM total ns]of this thread's last plan execution, from TornadoVM's profiler.static booleanonCuda()True when this thread's plans run through TornadoVM on a CUDA device.static <T> TRunsbodywith this thread's plans on devicespec("driver:device", asjitnn.device), syncs, then restores the previous device.static intpending()Number of operators recorded and not yet run on this thread.static long[]Plans compiled, replayed and reused for released storages on this JVM so far, and the replays that skipped the rewrite:[compiles, replays, reuses, remembered].static voidprintKernels(PrintStream out) Prints every plan compiled from now on toout(null stops): the plan's operators with their shapes and task counts, then the CUDA, OpenCL or Metal source of each kernel.static voidreservePlans(int plans) Makes room for at leastplansmore compiled plans in the plan cache (per concurrency slot), beyondjitnn.plan.cache: for callers that cycle through many shapes, as shape buckets do, so their plans are not evicted and compiled again.static voidreset()Closes every cached compiled plan, first saving persistent storages (weights) to the host; the next run of each trace compiles again.static voidsetAutotune(String mode) Sets run-time autotuning:"on"measures new shapes,"use"only applies saved decisions,"off".static voidsetAutotuneDirectory(Path dir) Sets where autotuning decisions are saved; decisions loaded so far are dropped.static voidsetBackend(Backend backend) Switches this thread's backend, first running anything recorded on the old one.static voidsetDeterministic(boolean on) Run-to-run reproducible results, as PyTorch'suse_deterministic_algorithms: jitNN's own kernels are deterministic already (no atomics; split reductions sum in a fixed order); this also stops autotuning from measuring and asks cuDNN for deterministic algorithms only.static voidStarts recording a timeline of plan executions forstopTrace(Path).static voidStops recording and writes the timeline as Chrome trace events (JSON), for Perfetto or chrome://tracing: plan executions, kernels, and the spans opened withtraceRange(String).static voidRuns everything recorded on this thread, fetchingwantedto the host.static long[]Cumulative nanoseconds spent in jitNN's executor outside TornadoVM ([0]) and inside TornadoVM's plan execution ([1]).static AutoCloseabletraceRange(String name) A named span on the timeline (one training step, say) while recording; nothing otherwise.static long[]Totals over every plan execution so far:[executions, kernel launches, library calls, bytes], the bytes being what the operators read and write at least (a lower bound on device-memory traffic that fusion lowers).static long[]Host-device traffic of this JVM's plans so far:[fetches to the host, elements fetched, re-uploads of persistent storages written by another plan, elements re-uploaded, input copies, elements copied].static <T> TRunsbodyonbackend, syncs, and restores the previous backend.
-
Method Details
-
backend
This thread's backend. -
setBackend
Switches this thread's backend, first running anything recorded on the old one. -
with
-
onDevice
Runsbodywith this thread's plans on devicespec("driver:device", asjitnn.device), syncs, then restores the previous device. Tensors belong to the device they were computed on: do not share them between threads on different devices (move data through the host, asHostCollectivedoes). -
onCuda
public static boolean onCuda()True when this thread's plans run through TornadoVM on a CUDA device. -
sync
Runs everything recorded on this thread, fetchingwantedto the host. -
pending
public static int pending()Number of operators recorded and not yet run on this thread. -
reset
public static void reset()Closes every cached compiled plan, first saving persistent storages (weights) to the host; the next run of each trace compiles again. -
reservePlans
public static void reservePlans(int plans) Makes room for at leastplansmore compiled plans in the plan cache (per concurrency slot), beyondjitnn.plan.cache: for callers that cycle through many shapes, as shape buckets do, so their plans are not evicted and compiled again. -
setDeterministic
public static void setDeterministic(boolean on) Run-to-run reproducible results, as PyTorch'suse_deterministic_algorithms: jitNN's own kernels are deterministic already (no atomics; split reductions sum in a fixed order); this also stops autotuning from measuring and asks cuDNN for deterministic algorithms only. Same device, same library versions, same inputs: the same bits. -
deterministic
public static boolean deterministic()Whether deterministic results were asked for (setDeterministic(boolean),jitnn.deterministic). -
setAutotune
Sets run-time autotuning:"on"measures new shapes,"use"only applies saved decisions,"off". -
setAutotuneDirectory
Sets where autotuning decisions are saved; decisions loaded so far are dropped. -
printKernels
Prints every plan compiled from now on toout(null stops): the plan's operators with their shapes and task counts, then the CUDA, OpenCL or Metal source of each kernel. Plans already compiled are not printed again; callreset()first to recompile them. -
planStats
public static long[] planStats()Plans compiled, replayed and reused for released storages on this JVM so far, and the replays that skipped the rewrite:[compiles, replays, reuses, remembered]. -
transferStats
public static long[] transferStats()Host-device traffic of this JVM's plans so far:[fetches to the host, elements fetched, re-uploads of persistent storages written by another plan, elements re-uploaded, input copies, elements copied]. -
trafficStats
public static long[] trafficStats()Totals over every plan execution so far:[executions, kernel launches, library calls, bytes], the bytes being what the operators read and write at least (a lower bound on device-memory traffic that fusion lowers). Differences between two calls give a step's counts. -
timeSplit
public static long[] timeSplit()Cumulative nanoseconds spent in jitNN's executor outside TornadoVM ([0]) and inside TornadoVM's plan execution ([1]). -
lastIntermediateBytes
public static long lastIntermediateBytes()Device memory for intermediates (activations, gradients and other temporaries, after buffer reuse) of the last plan this thread ran, in bytes; 0 on the reference backend. -
lastProfile
public static long[] lastProfile()Withjitnn.profile=true:[device kernel ns, TornadoVM total ns]of this thread's last plan execution, from TornadoVM's profiler. Zeros otherwise. -
kernelBreakdown
-
startTrace
public static void startTrace()Starts recording a timeline of plan executions forstopTrace(Path). Needsjitnn.profile=trueand, on CUDA,jitnn.cudaGraph=false. -
stopTrace
Stops recording and writes the timeline as Chrome trace events (JSON), for Perfetto or chrome://tracing: plan executions, kernels, and the spans opened withtraceRange(String).- Throws:
IOException
-
traceRange
A named span on the timeline (one training step, say) while recording; nothing otherwise. Close it (try-with-resources) where the span ends. -
cpuStats
-
cpuReport
The CPU backend's operators by time since the last call (as printed at exit withjitnn.cpu.stats), then starts counting again; empty when nothing ran. -
cpuOpenClDevice
The OpenCL device on the host CPU (<driver>:<device>) where the CPU backend runs the operators it has no kernel for, or null when there is none.
-