Class OnnxSession

java.lang.Object
ai.beestream.jitnn.onnx.OnnxSession
All Implemented Interfaces:
AutoCloseable

public final class OnnxSession extends Object implements AutoCloseable
Runs an ONNX model with jitNN: each node becomes jitNN operations, so a call records one trace that jitNN compiles into a TornadoVM plan on the first run and replays afterwards (same input shapes). Weights are loaded once into persistent tensors that stay on the device.
OnnxSession s = OnnxSession.load(Path.of("resnet50.onnx"));
Tensor logits = s.run(Map.of("x", images)).get("y");
Floating-point weights of any precision are held in FP32; with load(Path, boolean) in half precision activations stay FP16 between operators (Autocast.enableHalf), linear weights are stored in FP16, and outputs are handed out in FP32. -Djitnn.onnx.halfActivations=false keeps FP32 activations with FP16 products only (jitNN's autocast). Integer tensors become INT32 on the device. Booleans are FP32 0 / 1.
  • Method Details

    • load

      public static OnnxSession load(Path file) throws IOException
      Loads an ONNX file and prepares it to run in FP32; external data files next to it are memory-mapped.
      Throws:
      IOException
    • load

      public static OnnxSession load(Path file, boolean halfPrecision) throws IOException
      With halfPrecision, activations stay FP16 between operators (products accumulate in FP32) and the linear layers' weights are stored in FP16; outputs are FP32.
      Throws:
      IOException
    • of

      public static OnnxSession of(OnnxModel model, boolean halfPrecision)
      A session over a model built in memory (see OnnxModel.of(OnnxModel.Graph, Map)); halfPrecision as in load(Path, boolean).
    • model

      public OnnxModel model()
      The model this session runs.
    • inputNames

      public List<String> inputNames()
      The names of the graph's inputs, in graph order.
    • outputNames

      public List<String> outputNames()
      The names of the graph's outputs, in graph order.
    • close

      public void close()
      Releases the model's weights: plans compiled for this session can then run another session of the same model (loading it again) without compiling. The session cannot run afterwards.
      Specified by:
      close in interface AutoCloseable
    • run

      public Map<String,Tensor> run(Map<String,Tensor> inputs)
      Runs the graph on inputs (by graph input name) and returns every graph output, as lazy jitNN tensors (read them with toFloatArray, or keep computing with them).
    • withBuckets

      public OnnxSession withBuckets(int... sizes)
      Pads the model's symbolic dimensions to the smallest of sizes that holds them (a size beyond the largest is run as it is), so a server compiles one plan per bucket instead of one per request shape. Only dimensions padding cannot change the results of are padded:
      • a dimension that is the first axis of every input it appears in (the batch: rows are computed independently), and
      • a dimension of an input whose name contains mask (a sequence: padded positions are masked out, as zeros in the mask).
      Inputs are padded with zeros and outputs cut back to the request's sizes. Returns this session.
    • paddedDimensions

      public Set<String> paddedDimensions()
      The symbolic dimensions withBuckets(int...) pads (for tests and logging).