Package ai.beestream.jitnn.io


package ai.beestream.jitnn.io
Checkpoints: named tensors and values (StateDict) saved and loaded in Hugging Face's safetensors format (SafeTensors).
  • Class
    Description
    A small JSON reader for the files jitNN loads (safetensors headers, Hugging Face configs and tokenizers): objects become LinkedHashMaps in file order, arrays Lists, integers Long, other numbers Double.
    Reads and writes the safetensors format (Hugging Face): an 8-byte little-endian header length, a JSON header that maps each tensor name to its dtype, shape and byte range, optional __metadata__ string pairs, then the raw little-endian data. jitNN's checkpoints use it, so they open in Python with safetensors.torch.load_file.
    One tensor of a file: its type (null when jitNN has no such type, e.g.
    A file's header: the tensors, in file order, and the metadata.
    The data to write for one tensor: its type, shape, and a segment of exactly its bytes.
    Named tensors and string values: the state of a model, an optimiser or a whole training run, as PyTorch's state_dict.