20.1. Pre-Training DataSet

Alpha Version: Work in progress.

The Stanford Natural Language Inference (SNLI) dataset is a large, annotated corpus containing approximately 570,000 English sentence pairs. Although originally designed for training and evaluating Natural Language Inference (NLI) models, its rich text stream makes it a suitable playground for our language modeling experiment.

The sentences in the SNLI corpus are relatively simple. Most are short and feature straightforward grammatical structures rather than long or syntactically complex clauses.

In addition to the sentence pairs, the dataset contains NLI-specific annotations, such as entailment labels and parse trees. During the GPT-1 pre-training phase, however, all annotation data is ignored, and only the raw sentences are utilized. (The label information is preserved and used later in Task 1 during the fine-tuning stage.)

The prepare_datasets.py script generates the following files from the original SNLI dataset¹:

1. snli_pretrain.txt
2. bpe_vocab.json

The dataset generation procedure is described in detail in Section 20.2.3.1.

These files are generated by coordinating several underlying scripts in the text_processing directory. Since these utility scripts perform basic text preprocessing and do not directly interact with the deep learning model architecture, their implementation details are omitted here.

20.1.1. snli_pretrain.txt

data/pretrain/snli_pretrain.txt contains a list of flat sentences extracted from the SNLI corpus. Specifically, it is constructed by collecting both the premise and hypothesis sentences from the original sentence pairs.

A small excerpt is shown below:

$ cat data/pretrain/snli_pretrain.txt
A person on a horse jumps over a broken down airplane.
A person is training his horse for a competition.
A person is at a diner, ordering an omelette.
A person is outdoors, on a horse.
Children smiling and waving at camera
They are smiling at their parents
There are children present
The kids are frowning

... snip ...

Two women are observing something together.
Two women are looking at a flower together.
Two women are standing with their eyes closed.
Two girls are looking at something.
A man in a black leather jacket and a book in his hand speaks in a classroom.
A man is flying a kite.
A man is speaking in a classroom.
A man is teaching science in a classroom.

Although the processed corpus contains a total of 655,600 sentences, the average sentence length is quite compact, standing at only 8.85 words.

The distribution of sentence lengths is summarized below. As shown, 99% of the sentences contain no more than 23 words.

• Sentence-length coverage:
  • 80% of sentences contain 11 words or fewer.
  • 85% contain 13 words or fewer.
  • 90% contain 14 words or fewer.
  • 95% contain 17 words or fewer.
  • 99% contain 23 words or fewer.

20.1.2. bpe_vocab.json

bpe_vocab.json stores the token vocabulary generated by Byte Pair Encoding (BPE). The vocabulary consists of 21,119 entries, encompassing special tokens, subwords, and punctuation symbols.

$ cat bpe_vocab.json
{
  "<pad>": 0,
  "<unk>": 1,
  "<s>": 2,
  "</s>": 3,
  "$": 4,
  "!</w>": 5,
  "\"</w>": 6,
  "#</w>": 7,
  "$</w>": 8,

... snip ...

  "angel</w>": 776,
  "angele</w>": 777,
  "angeles</w>": 778,
  "angelic</w>": 779,
  "angelos</w>": 780,
  "angels</w>": 781,
  "anger</w>": 782,
  "angered</w>": 783,
  "angering</w>": 784,

... snip ...

  "zooming</w>": 21111,
  "zooms</w>": 21112,
  "zuccherberg</w>": 21113,
  "zucchini</w>": 21114,
  "zumba</w>": 21115,
  "zune</w>": 21116,
  "zydeco</w>": 21117,
  "zz</w>": 21118
}