Record Class Tokenizer.Batch

java.lang.Object
java.lang.Record
ai.beestream.jitnn.nn.text.Tokenizer.Batch
Enclosing class:
Tokenizer

public static record Tokenizer.Batch(int[] idArray, int[] typeIdArray, int[] attention, int batch, int length) extends Record
Encodings padded to the longest one: [batch, length] ids and type ids, and an attention mask of 1 for tokens and 0 for padding.
  • Constructor Summary

    Constructors
    Constructor
    Description
    Batch(int[] idArray, int[] typeIdArray, int[] attention, int batch, int length)
    Creates an instance of a Batch record class.
  • Method Summary

    Modifier and Type
    Method
    Description
    An additive attention mask [batch, 1, 1, length]: 0 for tokens, -10000 for padding.
    int[]
    Returns the value of the attention record component.
    int
    Returns the value of the batch record component.
    final boolean
    Indicates whether some other object is "equal to" this one.
    final int
    Returns a hash code value for this object.
    int[]
    Returns the value of the idArray record component.
    ids()
    The token ids as an INT32 tensor [batch, length].
    int
    Returns the value of the length record component.
    final String
    Returns a string representation of this record class.
    int[]
    Returns the value of the typeIdArray record component.
    The token type (segment) ids as an INT32 tensor [batch, length].

    Methods inherited from class Object

    clone, finalize, getClass, notify, notifyAll, wait, wait, wait
  • Constructor Details

    • Batch

      public Batch(int[] idArray, int[] typeIdArray, int[] attention, int batch, int length)
      Creates an instance of a Batch record class.
      Parameters:
      idArray - the value for the idArray record component
      typeIdArray - the value for the typeIdArray record component
      attention - the value for the attention record component
      batch - the value for the batch record component
      length - the value for the length record component
  • Method Details

    • ids

      public Tensor ids()
      The token ids as an INT32 tensor [batch, length].
    • typeIds

      public Tensor typeIds()
      The token type (segment) ids as an INT32 tensor [batch, length].
    • additiveMask

      public Tensor additiveMask()
      An additive attention mask [batch, 1, 1, length]: 0 for tokens, -10000 for padding.
    • toString

      public final String toString()
      Returns a string representation of this record class. The representation contains the name of the class, followed by the name and value of each of the record components.
      Specified by:
      toString in class Record
      Returns:
      a string representation of this object
    • hashCode

      public final int hashCode()
      Returns a hash code value for this object. The value is derived from the hash code of each of the record components.
      Specified by:
      hashCode in class Record
      Returns:
      a hash code value for this object
    • equals

      public final boolean equals(Object o)
      Indicates whether some other object is "equal to" this one. The objects are equal if the other object is of the same class and if all the record components are equal. Reference components are compared with Objects::equals(Object,Object); primitive components are compared with the compare method from their corresponding wrapper classes.
      Specified by:
      equals in class Record
      Parameters:
      o - the object with which to compare
      Returns:
      true if this object is the same as the o argument; false otherwise.
    • idArray

      public int[] idArray()
      Returns the value of the idArray record component.
      Returns:
      the value of the idArray record component
    • typeIdArray

      public int[] typeIdArray()
      Returns the value of the typeIdArray record component.
      Returns:
      the value of the typeIdArray record component
    • attention

      public int[] attention()
      Returns the value of the attention record component.
      Returns:
      the value of the attention record component
    • batch

      public int batch()
      Returns the value of the batch record component.
      Returns:
      the value of the batch record component
    • length

      public int length()
      Returns the value of the length record component.
      Returns:
      the value of the length record component