Class Corpus

java.lang.Object
ai.beestream.jitnn.nn.text.Corpus

public final class Corpus extends Object
Training examples for the embedding models, drawn on the host from sentences of word ids: skip-gram pairs, CBOW windows, negative samples from the unigram distribution raised to 3/4, and GloVe's distance-weighted co-occurrence counts. Everything is driven by a seed.
  • Constructor Summary

    Constructors
    Constructor
    Description
    Corpus(List<int[]> sentences, long[] counts, long seed)
    Examples from sentences of word ids, negatives drawn by the word counts counts (indexed by id), randomness from seed.
  • Method Summary

    Modifier and Type
    Method
    Description
    int[][]
    cbowSamples(int window)
    CBOW samples, only positions with a full window on both sides, shuffled: [sentence, centre, context (2 window per sample, flattened)].
    cooccurrence(int window)
    Symmetric co-occurrence counts within window, each weighted 1 / distance: [rows, cols, value].
    static float
    cosine(float[] vectors, int dim, int a, int b)
    Cosine similarity of rows a and b of vectors [n, dim].
    static int
    nearest(float[] vectors, int dim, int of)
    The row of vectors [n, dim] most similar to row of (by cosine), excluding itself.
    int[]
    negatives(int n)
    n words drawn from the unigram distribution to the power 3/4.
    List<int[]>
    The sentences, as word ids.
    int[][]
    skipGramPairs(int window, boolean docs)
    Skip-gram (centre, context) pairs with word2vec's dynamic window (each centre draws a window 1..window), shuffled: [centres, contexts].

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Constructor Details

    • Corpus

      public Corpus(List<int[]> sentences, long[] counts, long seed)
      Examples from sentences of word ids, negatives drawn by the word counts counts (indexed by id), randomness from seed.
  • Method Details

    • sentences

      public List<int[]> sentences()
      The sentences, as word ids.
    • negatives

      public int[] negatives(int n)
      n words drawn from the unigram distribution to the power 3/4.
    • skipGramPairs

      public int[][] skipGramPairs(int window, boolean docs)
      Skip-gram (centre, context) pairs with word2vec's dynamic window (each centre draws a window 1..window), shuffled: [centres, contexts]. With docs the centre is the sentence index instead (PV-DBOW).
    • cbowSamples

      public int[][] cbowSamples(int window)
      CBOW samples, only positions with a full window on both sides, shuffled: [sentence, centre, context (2 window per sample, flattened)].
    • cooccurrence

      public Object[] cooccurrence(int window)
      Symmetric co-occurrence counts within window, each weighted 1 / distance: [rows, cols, value].
    • cosine

      public static float cosine(float[] vectors, int dim, int a, int b)
      Cosine similarity of rows a and b of vectors [n, dim].
    • nearest

      public static int nearest(float[] vectors, int dim, int of)
      The row of vectors [n, dim] most similar to row of (by cosine), excluding itself.