Class JitnnRuntime

java.lang.Object
ai.beestream.jitnn.JitnnRuntime

public final class JitnnRuntime extends Object
jitNN's runtime: where operators run, sync points, determinism, profiling and statistics.

Operators on Backend.TORNADO and Backend.CPU are recorded lazily and run at a sync point: reading a tensor (Tensor.item(), Tensor.toFloatArray()) or sync(Tensor...). Compiled plans are cached by the trace's structure, so a training step or an inference call with the same shapes replays its plan. Options are set with JitnnConfig.

The backend and the device are per thread; everything else is per JVM.

  • Method Summary

    Modifier and Type
    Method
    Description
    static Backend
    This thread's backend.
    static String
    The OpenCL device on the host CPU (<driver>:<device>) where the CPU backend runs the operators it has no kernel for, or null when there is none.
    static String
    The CPU backend's operators by time since the last call (as printed at exit with jitnn.cpu.stats), then starts counting again; empty when nothing ran.
    static Map<String,long[]>
    The CPU backend's counters so far, per operator name: [calls, nanoseconds, calls that ran the reference implementation, calls that ran on the host's OpenCL device]; the times only with jitnn.cpu.stats=true.
    static boolean
    Whether deterministic results were asked for (setDeterministic(boolean), jitnn.deterministic).
    static Map<String,long[]>
    With jitnn.profile=true: device time per operator task ("op#k", the k-th task of the operator) as [launches, nanoseconds], accumulated since the last call, which resets it.
    static long
    Device memory for intermediates (activations, gradients and other temporaries, after buffer reuse) of the last plan this thread ran, in bytes; 0 on the reference backend.
    static long[]
    With jitnn.profile=true: [device kernel ns, TornadoVM total ns] of this thread's last plan execution, from TornadoVM's profiler.
    static boolean
    True when this thread's plans run through TornadoVM on a CUDA device.
    static <T> T
    onDevice(String spec, Supplier<T> body)
    Runs body with this thread's plans on device spec ("driver:device", as jitnn.device), syncs, then restores the previous device.
    static int
    Number of operators recorded and not yet run on this thread.
    static long[]
    Plans compiled, replayed and reused for released storages on this JVM so far, and the replays that skipped the rewrite: [compiles, replays, reuses, remembered].
    static void
    Prints every plan compiled from now on to out (null stops): the plan's operators with their shapes and task counts, then the CUDA, OpenCL or Metal source of each kernel.
    static void
    reservePlans(int plans)
    Makes room for at least plans more compiled plans in the plan cache (per concurrency slot), beyond jitnn.plan.cache: for callers that cycle through many shapes, as shape buckets do, so their plans are not evicted and compiled again.
    static void
    Closes every cached compiled plan, first saving persistent storages (weights) to the host; the next run of each trace compiles again.
    static void
    Sets run-time autotuning: "on" measures new shapes, "use" only applies saved decisions, "off".
    static void
    Sets where autotuning decisions are saved; decisions loaded so far are dropped.
    static void
    Switches this thread's backend, first running anything recorded on the old one.
    static void
    setDeterministic(boolean on)
    Run-to-run reproducible results, as PyTorch's use_deterministic_algorithms: jitNN's own kernels are deterministic already (no atomics; split reductions sum in a fixed order); this also stops autotuning from measuring and asks cuDNN for deterministic algorithms only.
    static void
    Starts recording a timeline of plan executions for stopTrace(Path).
    static void
    Stops recording and writes the timeline as Chrome trace events (JSON), for Perfetto or chrome://tracing: plan executions, kernels, and the spans opened with traceRange(String).
    static void
    sync(Tensor... wanted)
    Runs everything recorded on this thread, fetching wanted to the host.
    static long[]
    Cumulative nanoseconds spent in jitNN's executor outside TornadoVM ([0]) and inside TornadoVM's plan execution ([1]).
    A named span on the timeline (one training step, say) while recording; nothing otherwise.
    static long[]
    Totals over every plan execution so far: [executions, kernel launches, library calls, bytes], the bytes being what the operators read and write at least (a lower bound on device-memory traffic that fusion lowers).
    static long[]
    Host-device traffic of this JVM's plans so far: [fetches to the host, elements fetched, re-uploads of persistent storages written by another plan, elements re-uploaded, input copies, elements copied].
    static <T> T
    with(Backend backend, Supplier<T> body)
    Runs body on backend, syncs, and restores the previous backend.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Method Details

    • backend

      public static Backend backend()
      This thread's backend.
    • setBackend

      public static void setBackend(Backend backend)
      Switches this thread's backend, first running anything recorded on the old one.
    • with

      public static <T> T with(Backend backend, Supplier<T> body)
      Runs body on backend, syncs, and restores the previous backend.
    • onDevice

      public static <T> T onDevice(String spec, Supplier<T> body)
      Runs body with this thread's plans on device spec ("driver:device", as jitnn.device), syncs, then restores the previous device. Tensors belong to the device they were computed on: do not share them between threads on different devices (move data through the host, as HostCollective does).
    • onCuda

      public static boolean onCuda()
      True when this thread's plans run through TornadoVM on a CUDA device.
    • sync

      public static void sync(Tensor... wanted)
      Runs everything recorded on this thread, fetching wanted to the host.
    • pending

      public static int pending()
      Number of operators recorded and not yet run on this thread.
    • reset

      public static void reset()
      Closes every cached compiled plan, first saving persistent storages (weights) to the host; the next run of each trace compiles again.
    • reservePlans

      public static void reservePlans(int plans)
      Makes room for at least plans more compiled plans in the plan cache (per concurrency slot), beyond jitnn.plan.cache: for callers that cycle through many shapes, as shape buckets do, so their plans are not evicted and compiled again.
    • setDeterministic

      public static void setDeterministic(boolean on)
      Run-to-run reproducible results, as PyTorch's use_deterministic_algorithms: jitNN's own kernels are deterministic already (no atomics; split reductions sum in a fixed order); this also stops autotuning from measuring and asks cuDNN for deterministic algorithms only. Same device, same library versions, same inputs: the same bits.
    • deterministic

      public static boolean deterministic()
      Whether deterministic results were asked for (setDeterministic(boolean), jitnn.deterministic).
    • setAutotune

      public static void setAutotune(String mode)
      Sets run-time autotuning: "on" measures new shapes, "use" only applies saved decisions, "off".
    • setAutotuneDirectory

      public static void setAutotuneDirectory(Path dir)
      Sets where autotuning decisions are saved; decisions loaded so far are dropped.
    • printKernels

      public static void printKernels(PrintStream out)
      Prints every plan compiled from now on to out (null stops): the plan's operators with their shapes and task counts, then the CUDA, OpenCL or Metal source of each kernel. Plans already compiled are not printed again; call reset() first to recompile them.
    • planStats

      public static long[] planStats()
      Plans compiled, replayed and reused for released storages on this JVM so far, and the replays that skipped the rewrite: [compiles, replays, reuses, remembered].
    • transferStats

      public static long[] transferStats()
      Host-device traffic of this JVM's plans so far: [fetches to the host, elements fetched, re-uploads of persistent storages written by another plan, elements re-uploaded, input copies, elements copied].
    • trafficStats

      public static long[] trafficStats()
      Totals over every plan execution so far: [executions, kernel launches, library calls, bytes], the bytes being what the operators read and write at least (a lower bound on device-memory traffic that fusion lowers). Differences between two calls give a step's counts.
    • timeSplit

      public static long[] timeSplit()
      Cumulative nanoseconds spent in jitNN's executor outside TornadoVM ([0]) and inside TornadoVM's plan execution ([1]).
    • lastIntermediateBytes

      public static long lastIntermediateBytes()
      Device memory for intermediates (activations, gradients and other temporaries, after buffer reuse) of the last plan this thread ran, in bytes; 0 on the reference backend.
    • lastProfile

      public static long[] lastProfile()
      With jitnn.profile=true: [device kernel ns, TornadoVM total ns] of this thread's last plan execution, from TornadoVM's profiler. Zeros otherwise.
    • kernelBreakdown

      public static Map<String,long[]> kernelBreakdown()
      With jitnn.profile=true: device time per operator task ("op#k", the k-th task of the operator) as [launches, nanoseconds], accumulated since the last call, which resets it.
    • startTrace

      public static void startTrace()
      Starts recording a timeline of plan executions for stopTrace(Path). Needs jitnn.profile=true and, on CUDA, jitnn.cudaGraph=false.
    • stopTrace

      public static void stopTrace(Path file) throws IOException
      Stops recording and writes the timeline as Chrome trace events (JSON), for Perfetto or chrome://tracing: plan executions, kernels, and the spans opened with traceRange(String).
      Throws:
      IOException
    • traceRange

      public static AutoCloseable traceRange(String name)
      A named span on the timeline (one training step, say) while recording; nothing otherwise. Close it (try-with-resources) where the span ends.
    • cpuStats

      public static Map<String,long[]> cpuStats()
      The CPU backend's counters so far, per operator name: [calls, nanoseconds, calls that ran the reference implementation, calls that ran on the host's OpenCL device]; the times only with jitnn.cpu.stats=true.
    • cpuReport

      public static String cpuReport()
      The CPU backend's operators by time since the last call (as printed at exit with jitnn.cpu.stats), then starts counting again; empty when nothing ran.
    • cpuOpenClDevice

      public static String cpuOpenClDevice()
      The OpenCL device on the host CPU (<driver>:<device>) where the CPU backend runs the operators it has no kernel for, or null when there is none.