Class GgufFile

java.lang.Object
ai.beestream.jitnn.gguf.GgufFile

public final class GgufFile extends Object
A GGUF file (llama.cpp's model format, versions 2 and 3): its metadata and its tensors, which load as persistent jitNN tensors. The block-quantised weights (Q8_0, Q4_0, Q4_1, Q4_K, Q5_K, Q6_K, Q5_0, Q5_1, Q2_K, Q3_K) stay in their GGUF blocks and run through Gguf; F32, F16, BF16 and I32 tensors load as those types.

Tensors are memory-mapped rather than read: each one is mapped copy-on-write together with the few bytes before it, which become the TornadoVM array header, so loading a model costs no copy and no Java heap, and the file itself is never changed. Where that is not possible (a file that cannot be opened for writing, or data not 8-byte aligned) the tensor is copied instead.

GgufFile f = GgufFile.open(Path.of("model.gguf"));
int layers = f.integer("llama.block_count");
Tensor wq = f.tensor("blk.0.attn_q.weight");   // Q4_K, shape [out, in]
Tensor y = Gguf.linear(x, wq, null);

Metadata values are Java objects: the 8-, 16- and 32-bit signed and unsigned integers except uint32 are Integer, uint32 and the 64-bit integers are Long (a uint64 above Long.MAX_VALUE reads as negative), then Float, Double, Boolean and String. Arrays are int[], long[], float[], double[], boolean[] and String[] by the same rule, and Object[] for arrays of arrays.

  • Method Details

    • open

      public static GgufFile open(Path path) throws IOException
      Reads the metadata and the tensor list of path; tensors are mapped when loaded.
      Throws:
      IOException
    • path

      public Path path()
      The file this was read from.
    • version

      public int version()
      The GGUF format version (2 or 3).
    • alignment

      public int alignment()
      The data alignment in bytes (general.alignment, 32 by default).
    • dataOffset

      public long dataOffset()
      Byte offset of the tensor data section.
    • metadata

      public Map<String,Object> metadata()
      All metadata, in file order.
    • keys

      public Set<String> keys()
      The metadata keys, in file order.
    • has

      public boolean has(String key)
      Whether the metadata has key.
    • get

      public Object get(String key)
      The value of key, or null when absent.
    • architecture

      public String architecture()
      general.architecture, for example llama, or null when absent.
    • string

      public String string(String key)
      A string value; throws when the key is absent or holds another type.
    • integer

      public int integer(String key)
      An integer value of any width that fits in an int.
    • integer

      public int integer(String key, int fallback)
      integer(String), or fallback when the key is absent.
    • longValue

      public long longValue(String key)
      An integer value of any width.
    • floatValue

      public float floatValue(String key)
      A float32 or float64 value (an integer value is accepted too).
    • floatValue

      public float floatValue(String key, float fallback)
      floatValue(String), or fallback when the key is absent.
    • bool

      public boolean bool(String key)
      A boolean value; throws when the key is absent or holds another type.
    • strings

      public String[] strings(String key)
      An array of strings (a tokenizer's vocabulary, say); throws when the key is absent or holds another type.
    • floats

      public float[] floats(String key)
      An array of float32 values; throws when the key is absent or holds another type.
    • ints

      public int[] ints(String key)
      An array of 8-, 16- or 32-bit integers (other than uint32).
    • longs

      public long[] longs(String key)
      An array of uint32 or 64-bit integers.
    • tensorInfos

      public Map<String, GgufTensorInfo> tensorInfos()
      The tensors listed in the file, by name, in file order.
    • hasTensor

      public boolean hasTensor(String name)
      Whether the file lists a tensor named name.
    • info

      public GgufTensorInfo info(String name)
      The listing of tensor name: its type, shape and offset; throws when there is none.
    • tensor

      public Tensor tensor(String name) throws IOException
      Tensor name as a persistent jitNN tensor, memory-mapped where possible (see the class comment).
      Throws:
      IOException
    • tensor

      public Tensor tensor(String name, boolean copy) throws IOException
      Tensor name; copy reads it into memory instead of mapping it.
      Throws:
      IOException
    • tensors

      public Map<String,Tensor> tensors() throws IOException
      Every tensor in the file (all must be of a type jitNN loads), by name, in file order.
      Throws:
      IOException
    • raw

      public MemorySegment raw(String name) throws IOException
      The raw bytes of tensor name, mapped read-only and released when unreachable.
      Throws:
      IOException
    • toString

      public String toString()
      Overrides:
      toString in class Object