20. Pre-Training

Alpha Version: Work in progress.

The pre-training task in GPT-1 is Next Word Prediction (causal/autoregressive language modeling). In other words, the model predicts the next token, $w_{i}$, given all preceding tokens, $w_{< i}$.

Therefore, the training objective is identical to the RNN-based Last Word Prediction introduced in Section 11.3. Furthermore, the underlying language model is mathematically identical to the one defined by Equations (11.1) and (11.6) in Section 11.1.

For example, suppose the input sequence is:

• “A cat is in the kitchen”

While the Transformer receives the entire sequence as a single input all at once, it utilizes a causal attention mask to simultaneously predict the next token at every position in the sequence using only the previously observed tokens.

For clarity, these parallel predictions can be conceptualized sequentially as follows:

• 1-word context (equivalent to a bi-gram): $\text{“A”} \rightarrow \text{“cat”}$
• 2-word context (equivalent to a tri-gram): $\text{“A cat”} \rightarrow \text{“is”}$
• 3-word context (equivalent to a 4-gram): $\text{“A cat is”} \rightarrow \text{“in”}$
• 4-word context (equivalent to a 5-gram): $\text{“A cat is in”} \rightarrow \text{“the”}$
• 5-word context (equivalent to a 6-gram): $\text{“A cat is in the”} \rightarrow \text{“kitchen”}$

By executing Next Word Prediction at every position across the sequence in parallel, the Transformer efficiently learns the conditional probability $P(w_{i} | w_{< i})$, ultimately modeling the joint probability distribution defined by Equation (11.6).

In this chapter, Section 20.1 introduces the SNLI text dataset used for our pre-training practical implementation. While the original GPT-1 was pre-trained on the massive BookCorpus dataset, we utilize the smaller SNLI corpus here to keep the computational cost manageable for hands-on learning.

We then explain how the pre-training model is implemented. Since most of its components share the same core architecture as the Transformer described in Part 4, Section 20.2 focuses exclusively on the components unique to GPT-1.