Class OnnxSession
java.lang.Object
ai.beestream.jitnn.onnx.OnnxSession
- All Implemented Interfaces:
AutoCloseable
Runs an ONNX model with jitNN: each node becomes jitNN operations, so a call records one trace
that jitNN compiles into a TornadoVM plan on the first run and replays afterwards (same input
shapes). Weights are loaded once into persistent tensors that stay on the device.
OnnxSession s = OnnxSession.load(Path.of("resnet50.onnx"));
Tensor logits = s.run(Map.of("x", images)).get("y");
Floating-point weights of any precision are held in FP32; with load(Path, boolean) in
half precision activations stay FP16 between operators (Autocast.enableHalf), linear
weights are stored in FP16, and outputs are handed out in FP32. -Djitnn.onnx.halfActivations=false
keeps FP32 activations with FP16 products only (jitNN's autocast). Integer tensors become INT32
on the device. Booleans are FP32 0 / 1.-
Method Summary
Modifier and TypeMethodDescriptionvoidclose()Releases the model's weights: plans compiled for this session can then run another session of the same model (loading it again) without compiling.The names of the graph's inputs, in graph order.static OnnxSessionLoads an ONNX file and prepares it to run in FP32; external data files next to it are memory-mapped.static OnnxSessionWithhalfPrecision, activations stay FP16 between operators (products accumulate in FP32) and the linear layers' weights are stored in FP16; outputs are FP32.model()The model this session runs.static OnnxSessionA session over a model built in memory (seeOnnxModel.of(OnnxModel.Graph, Map));halfPrecisionas inload(Path, boolean).The names of the graph's outputs, in graph order.The symbolic dimensionswithBuckets(int...)pads (for tests and logging).Runs the graph oninputs(by graph input name) and returns every graph output, as lazy jitNN tensors (read them with toFloatArray, or keep computing with them).withBuckets(int... sizes) Pads the model's symbolic dimensions to the smallest ofsizesthat holds them (a size beyond the largest is run as it is), so a server compiles one plan per bucket instead of one per request shape.
-
Method Details
-
load
Loads an ONNX file and prepares it to run in FP32; external data files next to it are memory-mapped.- Throws:
IOException
-
load
WithhalfPrecision, activations stay FP16 between operators (products accumulate in FP32) and the linear layers' weights are stored in FP16; outputs are FP32.- Throws:
IOException
-
of
A session over a model built in memory (seeOnnxModel.of(OnnxModel.Graph, Map));halfPrecisionas inload(Path, boolean). -
model
The model this session runs. -
inputNames
-
outputNames
-
close
public void close()Releases the model's weights: plans compiled for this session can then run another session of the same model (loading it again) without compiling. The session cannot run afterwards.- Specified by:
closein interfaceAutoCloseable
-
run
-
withBuckets
Pads the model's symbolic dimensions to the smallest ofsizesthat holds them (a size beyond the largest is run as it is), so a server compiles one plan per bucket instead of one per request shape. Only dimensions padding cannot change the results of are padded:- a dimension that is the first axis of every input it appears in (the batch: rows are computed independently), and
- a dimension of an input whose name contains
mask(a sequence: padded positions are masked out, as zeros in the mask).
-
paddedDimensions
The symbolic dimensionswithBuckets(int...)pads (for tests and logging).
-