19. Overview

Alpha Version: Work in progress.

This section provides a brief overview of the evolution of large language models (LLMs) since the landmark Transformer architecture, clarifying the positioning of GPT-1. It also summarizes the core architecture and key features of GPT-1.

19.1. LLM Evolution After Transformer

In just 16 months following the introduction of the original Encoder-Decoder Transformer (June 12, 2017), two prominent variants emerged: the decoder-based GPT-1 (June 11, 2018) and the encoder-based BERT (October 11, 2018).

During the first two years, BERT and its derivatives held a dominant position in the field. Over the subsequent three years, a fierce competition unfolded between the direct Encoder-Decoder descendants of the Transformer (such as T5 and GLM) and decoder-based GPT-style models. However, following the release of GPT-3, decoder-based LLMs became the dominant paradigm in the industry.

19.2. GPT-1 Architecture

The architecture of GPT-1 simplifies the standard Transformer’s decoder block. Each unit consists of a single Masked Multi-Head Attention (MHA) layer followed by a Feed-Forward Network (FFN), with both components integrated via residual connections. See Figure 19-1.

Compared to the original Transformer model, GPT-1 does not utilize sinusoidal positional encoding, but instead employs a learnable position embedding matrix.

Since these underlying core technologies were thoroughly covered in Part 4, the implementation details of the learnable position embedding are focused on in Section 20.2.

19.3. Pre-Training and Fine-Tuning

Despite having a simpler architecture than the original Transformer, GPT-1 outperformed it across a wider range of tasks. This success lies in its two-stage training methodology:

• (Unsupervised) Pre-Training: The model first acquires general linguistic capabilities through next-token prediction.
• (Supervised) Fine-Tuning: The model is then adapted to specific target tasks through supervised training on small, carefully curated datasets.

True to its name, “Generative Pre-trained Transformer”, this two-stage process is the core of GPT’s success. The model first grasps the fundamental structure of language through self-supervised pre-training, then masterfully specializes in targeted tasks via supervised fine-tuning.

By decoupling general language acquisition from task-specific training, GPT-1 achieved state-of-the-art (SOTA) performance across a broad spectrum of NLP tasks, including translation, inference, and question answering.