20. Pre-Training on Hironobu SUZUKI @ InterDB

20.1. Pre-Training DataSet

Mon, 01 Jan 0001 00:00:00 +0000

Alpha Version: Work in progress.

The Stanford Natural Language Inference (SNLI) dataset is a large, annotated corpus containing approximately 570,000 English sentence pairs. Although originally designed for training and evaluating Natural Language Inference (NLI) models, its rich text stream makes it a suitable playground for our language modeling experiment.

The sentences in the SNLI corpus are relatively simple. Most are short and feature straightforward grammatical structures rather than long or syntactically complex clauses.

20.2. Implementation

Mon, 01 Jan 0001 00:00:00 +0000

Alpha Version: Work in progress.

This section describes the architectural components specific to GPT-1—namely, its position encoding mechanism and the language modeling loss function. It then walks through how to execute the pre-training scripts.

Section Contents

20.2.1. Embedding and Position Encoding
20.2.2. Loss Function
20.2.3. Running Pre-Training

20.2.1. Embedding and Position Encoding

In the original Transformer architecture, the token embedding matrix is learnable, whereas the positional encoding relies on a fixed, predefined sinusoidal function.