Part 5: GPT-1

Alpha Version: Work in progress.

This part introduces GPT-1 (Generative Pre-trained Transformer).

Proposed in 2018, exactly one year after the original Transformer architecture (2017), GPT-1 adopts a two-stage training approach consisting of unsupervised pre-training followed by supervised fine-tuning.

More importantly, it demonstrated that a single pre-trained Transformer could be adapted to a wide variety of downstream NLP tasks. Through simple fine-tuning, the model achieved this without requiring substantial architectural changes.

In the following chapters, Chapter 19 provides an overview of the GPT-1 architecture and its evolution.

Chapter 20 and Chapter 21 then explain the pre-training and fine-tuning stages, respectively. As concrete examples of fine-tuning, we will implement two downstream tasks: Natural Language Inference (NLI) and Question Answering (QA).

Finally, Chapter 22 presents a brief overview of the evolution of large language models (LLMs) after GPT-1.