Class GgufFile
java.lang.Object
ai.beestream.jitnn.gguf.GgufFile
A GGUF file (llama.cpp's model format, versions 2 and 3): its metadata and its tensors, which load
as persistent jitNN tensors. The block-quantised weights (Q8_0, Q4_0, Q4_1, Q4_K, Q5_K, Q6_K, Q5_0, Q5_1, Q2_K, Q3_K) stay
in their GGUF blocks and run through
Gguf; F32, F16, BF16 and I32
tensors load as those types.
Tensors are memory-mapped rather than read: each one is mapped copy-on-write together with the few bytes before it, which become the TornadoVM array header, so loading a model costs no copy and no Java heap, and the file itself is never changed. Where that is not possible (a file that cannot be opened for writing, or data not 8-byte aligned) the tensor is copied instead.
GgufFile f = GgufFile.open(Path.of("model.gguf"));
int layers = f.integer("llama.block_count");
Tensor wq = f.tensor("blk.0.attn_q.weight"); // Q4_K, shape [out, in]
Tensor y = Gguf.linear(x, wq, null);
Metadata values are Java objects: the 8-, 16- and 32-bit signed and unsigned integers except
uint32 are Integer, uint32 and the 64-bit integers are Long (a uint64 above
Long.MAX_VALUE reads as negative), then Float, Double, Boolean and
String. Arrays are int[], long[], float[], double[],
boolean[] and String[] by the same rule, and Object[] for arrays of arrays.
-
Method Summary
Modifier and TypeMethodDescriptionintThe data alignment in bytes (general.alignment, 32 by default).general.architecture, for examplellama, ornullwhen absent.booleanA boolean value; throws when the key is absent or holds another type.longByte offset of the tensor data section.float[]An array of float32 values; throws when the key is absent or holds another type.floatfloatValue(String key) A float32 or float64 value (an integer value is accepted too).floatfloatValue(String key, float fallback) floatValue(String), orfallbackwhen the key is absent.The value ofkey, ornullwhen absent.booleanWhether the metadata haskey.booleanWhether the file lists a tensor namedname.The listing of tensorname: its type, shape and offset; throws when there is none.intAn integer value of any width that fits in anint.intinteger(String), orfallbackwhen the key is absent.int[]An array of 8-, 16- or 32-bit integers (other than uint32).keys()The metadata keys, in file order.long[]An array of uint32 or 64-bit integers.longAn integer value of any width.metadata()All metadata, in file order.static GgufFileReads the metadata and the tensor list ofpath; tensors are mapped when loaded.path()The file this was read from.The raw bytes of tensorname, mapped read-only and released when unreachable.A string value; throws when the key is absent or holds another type.String[]An array of strings (a tokenizer's vocabulary, say); throws when the key is absent or holds another type.Tensornameas a persistent jitNN tensor, memory-mapped where possible (see the class comment).Tensorname;copyreads it into memory instead of mapping it.The tensors listed in the file, by name, in file order.tensors()Every tensor in the file (all must be of a type jitNN loads), by name, in file order.toString()intversion()The GGUF format version (2 or 3).
-
Method Details
-
open
Reads the metadata and the tensor list ofpath; tensors are mapped when loaded.- Throws:
IOException
-
path
The file this was read from. -
version
public int version()The GGUF format version (2 or 3). -
alignment
public int alignment()The data alignment in bytes (general.alignment, 32 by default). -
dataOffset
public long dataOffset()Byte offset of the tensor data section. -
metadata
-
keys
-
has
Whether the metadata haskey. -
get
-
architecture
general.architecture, for examplellama, ornullwhen absent. -
string
-
integer
An integer value of any width that fits in anint. -
integer
integer(String), orfallbackwhen the key is absent. -
longValue
An integer value of any width. -
floatValue
A float32 or float64 value (an integer value is accepted too). -
floatValue
floatValue(String), orfallbackwhen the key is absent. -
bool
A boolean value; throws when the key is absent or holds another type. -
strings
-
floats
An array of float32 values; throws when the key is absent or holds another type. -
ints
An array of 8-, 16- or 32-bit integers (other than uint32). -
longs
An array of uint32 or 64-bit integers. -
tensorInfos
The tensors listed in the file, by name, in file order. -
hasTensor
Whether the file lists a tensor namedname. -
info
The listing of tensorname: its type, shape and offset; throws when there is none. -
tensor
Tensornameas a persistent jitNN tensor, memory-mapped where possible (see the class comment).- Throws:
IOException
-
tensor
Tensorname;copyreads it into memory instead of mapping it.- Throws:
IOException
-
tensors
Every tensor in the file (all must be of a type jitNN loads), by name, in file order.- Throws:
IOException
-
raw
The raw bytes of tensorname, mapped read-only and released when unreachable.- Throws:
IOException
-
toString
-