GPT-1 from Scratch
I built GPT-1 from scratch and trained it to perform two downstream tasks. (To be more precise, I developed this version of GPT-1 in collaboration with Claude, based upon a transformer_tf.py script that I had created from scratch.)
A detailed explanation of the internal architecture of GPT-1 will be published in The Engineer’s Guide To Deep Learning in the near future (within a few years).
In this post, I will only briefly demonstrate how to use it. Please note that this is an alpha version, and many refactoring updates are planned for the future.
-
Pre-training: We pre-train the model using the SNLI dataset:
-
Fine-tuning: Using the pre-trained weights, we fine-tune the model on two specific tasks:
- Task 1: Inference (Natural Language Inference) Given a “Premise” and a “Hypothesis”, the model classifies the relationship into one of three categories: entailment, neutral, or contradiction.
- Task 2: Question and Answer (Physics QA) The model answers simple questions about physical concepts (e.g., motion, heat, sound) suitable for preschool children.
Clone the GitHub repository and navigate to the Part-05 directory.
$ git clone https://github.com/s-hironobu/guide2dl
$ cd guide2dl/Part-05
Install the required packages as needed. For more details, please refer to the repository’s README.
$ python -m venv ~/venv-metal$ source ~/venv-metal/bin/activate
$ python -m pip install -U pip$ pip install tensorflow==2.15.1
$ pip install tensorflow-metal==1.1.0
$ pip install matplotlib==3.9.0
Run prepare_datasets.py in the Part-05 directory. This script will create a ./data directory and download the SNLI dataset.
$ python prepare_datasets.py
To pre-train the model on the SNLI corpus, run the gpt1_pretrain.py script. The training runs for 50 epochs and takes more than 10 hours to complete.
$ python gpt1_pretrain.py 50
Given a “Premise” and a “Hypothesis”, the model predicts whether the hypothesis is an entailment, neutral, or contradiction relative to the premise.
The SNLI is a supervised learning dataset containing a large number of pairs like the ones shown below:
{
"annotator_labels": ["neutral"],
"gold_label": "entailment",
"sentence1": "A person on a horse jumps over a broken down airplane.",
"sentence2": "A person is outdoors, on a horse.",
}
...
{
"annotator_labels": ["contradiction"],
"gold_label": "contradiction",
"sentence1": "A black race car starts up in front of a crowd of people.",
"sentence2": "A man is driving down a lonely road.",
}
...
We use this structure directly. The inputs are split into “Premise” and “Hypothesis” to train the model as a 3-option classification problem (entailment, neutral, contradiction).
Run the following command to start fine-tuning. Training for 2 to 3 epochs is sufficient, which takes a few hours.
$ python gpt1_finetune.py 2 checkpoints/gpt1-pretrain-layers-12-d_model-512-d_ffn-2048-heads-8/
You can launch the interactive mode with the following command:
$ python gpt1_inference.py checkpoints/gpt1-finetune-nli-layers-12-d_model-512-d_ffn-2048-heads-8/
Input any Premise and Hypothesis to see the model’s prediction.
============================================================
GPT-1 NLI Inference — Interactive Mode
Enter 'q' or leave empty to exit
============================================================
Premise > A man is playing guitar on stage.
Hypothesis > A man is playing a musical instrument.
Premise : A man is playing guitar on stage.
Hypothesis : A man is playing a musical instrument.
Prediction : ✅ entailment (confidence: 99.2%)
Scores : entailment=99.2% neutral=0.8% contradiction=0.0%
Token length : 19
Premise > A man is playing guitar on stage.
Hypothesis > The man is sleeping in bed.
Premise : A man is playing guitar on stage.
Hypothesis : The man is sleeping in bed.
Prediction : ❌ contradiction (confidence: 100.0%)
Scores : entailment=0.0% neutral=0.0% contradiction=100.0%
Token length : 18
Premise >
Since the SNLI corpus consists of very simple sentences, the model often struggles to understand complex phrasing. This clearly demonstrates how the quality and nature of the training data directly impact model performance.
You can also prepare a tab-separated values (TSV) file where the Premise and Hypothesis are separated by a tab space, as shown below:
$ cat snli.tsv
A man is playing guitar on stage. A man is playing a musical instrument.
Two dogs are running in the park. There are animals outside.
A woman is cooking dinner in the kitchen. A woman is preparing food.
A man is playing guitar on stage. The man is a famous musician.
Two dogs are running in the park. The dogs are chasing a ball.
A woman is cooking dinner in the kitchen. She is making pasta for her family.
A man is playing guitar on stage. The man is sleeping in bed.
Two dogs are running in the park. The dogs are sitting still indoors.
A woman is cooking dinner in the kitchen. No one is in the kitchen.
By passing this file to the inference script, the model will run predictions on all the pairs and display the results.
$ python gpt1_inference.py checkpoints/gpt1-finetune-nli-layers-12-d_model-512-d_ffn-2048-heads-8/ snli.tsv
... snip ...
[File] Processing snli.tsv ...
[Pair 1]
Premise : A man is playing guitar on stage.
Hypothesis : A man is playing a musical instrument.
Prediction : ✅ entailment (confidence: 99.2%)
Scores : entailment=99.2% neutral=0.8% contradiction=0.0%
Token length : 19
[Pair 2]
Premise : Two dogs are running in the park.
Hypothesis : There are animals outside.
Prediction : ✅ entailment (confidence: 99.9%)
Scores : entailment=99.9% neutral=0.1% contradiction=0.0%
Token length : 16
[Pair 3]
Premise : A woman is cooking dinner in the kitchen.
Hypothesis : A woman is preparing food.
Prediction : ✅ entailment (confidence: 99.0%)
Scores : entailment=99.0% neutral=1.0% contradiction=0.0%
Token length : 18
[Pair 4]
Premise : A man is playing guitar on stage.
Hypothesis : The man is a famous musician.
Prediction : ➖ neutral (confidence: 99.6%)
Scores : entailment=0.4% neutral=99.6% contradiction=0.1%
Token length : 18
[Pair 5]
Premise : Two dogs are running in the park.
Hypothesis : The dogs are chasing a ball.
Prediction : ➖ neutral (confidence: 99.9%)
Scores : entailment=0.0% neutral=99.9% contradiction=0.0%
Token length : 18
[Pair 6]
Premise : A woman is cooking dinner in the kitchen.
Hypothesis : She is making pasta for her family.
Prediction : ➖ neutral (confidence: 99.3%)
Scores : entailment=0.5% neutral=99.3% contradiction=0.2%
Token length : 20
[Pair 7]
Premise : A man is playing guitar on stage.
Hypothesis : The man is sleeping in bed.
Prediction : ❌ contradiction (confidence: 100.0%)
Scores : entailment=0.0% neutral=0.0% contradiction=100.0%
Token length : 18
[Pair 8]
Premise : Two dogs are running in the park.
Hypothesis : The dogs are sitting still indoors.
Prediction : ❌ contradiction (confidence: 100.0%)
Scores : entailment=0.0% neutral=0.0% contradiction=100.0%
Token length : 18
[Pair 9]
Premise : A woman is cooking dinner in the kitchen.
Hypothesis : No one is in the kitchen.
Prediction : ❌ contradiction (confidence: 99.8%)
Scores : entailment=0.1% neutral=0.1% contradiction=99.8%
Token length : 19
Done: 9 pairs processed / 0 lines skipped
The next task is Question and Answer, specifically testing basic physical concepts (motion, heat, sound, etc.) at a preschooler level.
The training data (questions and answers) is automatically generated using the gpt1_physics_qa_generator.py script during fine-tuning.
To fine-tune the model on the physics dataset, run the following command. In this example, we train for 5 epochs:
python gpt1_finetune_physics.py 5 checkpoints/gpt1-pretrain-layers-12-d_model-512-d_ffn-2048-heads-8/
You can launch the interactive dialogue mode with the following command:
python gpt1_inference_physics.py checkpoints/gpt1-finetune-physics-layers-12-d_model-512-d_ffn-2048-heads-8-best/
Input a simple scenario and a question where the answer falls into the predefined Answer Vocabulary (e.g., fast, slow, falls, stays, etc.).
================================================================
GPT-1 Physics Concepts QA Inference - Dialogue Mode (Type 'q' or press Enter to quit)
Answer Vocabulary: fast, slow, falls, stays, slides, grips, floats, sinks, hard, soft, bounces, no_bounce, attracts, no_attract, loud, quiet, bright, dark, hot, cold, rolls, no_roll, breaks, unbreakable
================================================================
Scenario > A cheetah runs very quickly.
Question > Is it fast or slow?
Scenario : A cheetah runs very quickly.
Question : Is it fast or slow?
Prediction : 🏃 fast (confidence: 100.0%)
Scores : fast=100.0% slow=0.0% falls=0.0% stays=0.0% slides=0.0% grips=0.0% floats=0.0% sinks=0.0% hard=0.0% soft=0.0% bounces=0.0% no_bounce=0.0% attracts=0.0% no_attract=0.0% loud=0.0% quiet=0.0% bright=0.0% dark=0.0% hot=0.0% cold=0.0% rolls=0.0% no_roll=0.0% breaks=0.0% unbreakable=0.0%
Token length: 15
Scenario > You walk on the ice.
Question > Do you slide or not?
Scenario : You walk on the ice.
Question : Do you slide or not?
Prediction : 🧊 slides (confidence: 100.0%)
Scores : fast=0.0% slow=0.0% falls=0.0% stays=0.0% slides=100.0% grips=0.0% floats=0.0% sinks=0.0% hard=0.0% soft=0.0% bounces=0.0% no_bounce=0.0% attracts=0.0% no_attract=0.0% loud=0.0% quiet=0.0% bright=0.0% dark=0.0% hot=0.0% cold=0.0% rolls=0.0% no_roll=0.0% breaks=0.0% unbreakable=0.0%
Token length: 15
Scenario > You drop the basketball on the floor.
Question > Does it bounce back up?
Scenario : You drop the basketball on the floor.
Question : Does it bounce back up?
Prediction : ❓ bounces (confidence: 99.9%)
Scores : fast=0.0% slow=0.0% falls=0.0% stays=0.0% slides=0.0% grips=0.0% floats=0.0% sinks=0.0% hard=0.0% soft=0.0% bounces=99.9% no_bounce=0.0% attracts=0.0% no_attract=0.0% loud=0.0% quiet=0.0% bright=0.0% dark=0.0% hot=0.0% cold=0.0% rolls=0.0% no_roll=0.0% breaks=0.0% unbreakable=0.0%
Token length: 17
Scenario > You touch the campfire. Does it feel hot or cold?
Question > Does it feel hot or cold?
Scenario : You touch the campfire.
Question : Does it feel hot or cold?
Prediction : ❓ hot (confidence: 99.9%)
Scores : fast=0.0% slow=0.0% falls=0.0% stays=0.0% slides=0.0% grips=0.0% floats=0.0% sinks=0.0% hard=0.0% soft=0.0% bounces=0.0% no_bounce=0.0% attracts=0.0% no_attract=0.0% loud=0.0% quiet=0.0% bright=0.0% dark=0.0% hot=99.9% cold=0.0% rolls=0.0% no_roll=0.0% breaks=0.0% unbreakable=0.0%
Token length: 22
Prepare a TSV file with Scenarios and Questions separated by a tab space:
$cat physics-qa.tsv
A cheetah runs very quickly. Is it fast or slow?
A snail moves very slowly. Is it fast or slow?
You drop the apple from your hand. What happens to it?
You gently place the book on the shelf. What happens to it?
You walk on the ice. Do you slide or not?
You walk on the carpet. Do you slide or not?
You put the wooden block in the water. Does it float or sink?
You drop the rock into the pool. Does it float or sink?
You touch the diamond. Does it feel hard or soft?
You press your hand on the marshmallow. Does it feel hard or soft?
You drop the basketball on the floor. Does it bounce back up?
You drop the egg on the floor. Does it bounce back up?
You bring a magnet near the iron nail. Does the magnet pull it?
You bring a magnet near the wooden pencil. Does the magnet pull it?
You hear a thunderclap. Is it loud or quiet?
You hear a whisper. Is it loud or quiet?
... snip ...
Run the inference script with the TSV file to process all questions and output the results.
$ python gpt1_inference_physics.py checkpoints/gpt1-finetune-physics-layers-12-d_model-512-d_ffn-2048-heads-8-best/ physics-qa.tsv
... snip ...
[File] Processing physics-qa.tsv ...
[Pair 1]
Scenario : A cheetah runs very quickly.
Question : Is it fast or slow?
Prediction : 🏃 fast (confidence: 100.0%)
Scores : fast=100.0% slow=0.0% falls=0.0% stays=0.0% slides=0.0% grips=0.0% floats=0.0% sinks=0.0% hard=0.0% soft=0.0% bounces=0.0% no_bounce=0.0% attracts=0.0% no_attract=0.0% loud=0.0% quiet=0.0% bright=0.0% dark=0.0% hot=0.0% cold=0.0% rolls=0.0% no_roll=0.0% breaks=0.0% unbreakable=0.0%
Token length: 15
[Pair 2]
Scenario : A snail moves very slowly.
Question : Is it fast or slow?
Prediction : 🐢 slow (confidence: 100.0%)
Scores : fast=0.0% slow=100.0% falls=0.0% stays=0.0% slides=0.0% grips=0.0% floats=0.0% sinks=0.0% hard=0.0% soft=0.0% bounces=0.0% no_bounce=0.0% attracts=0.0% no_attract=0.0% loud=0.0% quiet=0.0% bright=0.0% dark=0.0% hot=0.0% cold=0.0% rolls=0.0% no_roll=0.0% breaks=0.0% unbreakable=0.0%
Token length: 15
[Pair 3]
Scenario : You drop the apple from your hand.
Question : What happens to it?
Prediction : ⬇️ falls (confidence: 100.0%)
Scores : fast=0.0% slow=0.0% falls=100.0% stays=0.0% slides=0.0% grips=0.0% floats=0.0% sinks=0.0% hard=0.0% soft=0.0% bounces=0.0% no_bounce=0.0% attracts=0.0% no_attract=0.0% loud=0.0% quiet=0.0% bright=0.0% dark=0.0% hot=0.0% cold=0.0% rolls=0.0% no_roll=0.0% breaks=0.0% unbreakable=0.0%
Token length: 16
[Pair 4]
Scenario : You gently place the book on the shelf.
Question : What happens to it?
Prediction : 🧍 stays (confidence: 100.0%)
Scores : fast=0.0% slow=0.0% falls=0.0% stays=100.0% slides=0.0% grips=0.0% floats=0.0% sinks=0.0% hard=0.0% soft=0.0% bounces=0.0% no_bounce=0.0% attracts=0.0% no_attract=0.0% loud=0.0% quiet=0.0% bright=0.0% dark=0.0% hot=0.0% cold=0.0% rolls=0.0% no_roll=0.0% breaks=0.0% unbreakable=0.0%
Token length: 17
[Pair 5]
Scenario : You walk on the ice.
Question : Do you slide or not?
Prediction : 🧊 slides (confidence: 100.0%)
Scores : fast=0.0% slow=0.0% falls=0.0% stays=0.0% slides=100.0% grips=0.0% floats=0.0% sinks=0.0% hard=0.0% soft=0.0% bounces=0.0% no_bounce=0.0% attracts=0.0% no_attract=0.0% loud=0.0% quiet=0.0% bright=0.0% dark=0.0% hot=0.0% cold=0.0% rolls=0.0% no_roll=0.0% breaks=0.0% unbreakable=0.0%
Token length: 15
[Pair 6]
Scenario : You walk on the carpet.
Question : Do you slide or not?
Prediction : ❓ grips (confidence: 72.7%)
Scores : fast=0.0% slow=0.0% falls=0.0% stays=0.0% slides=27.3% grips=72.7% floats=0.0% sinks=0.0% hard=0.0% soft=0.0% bounces=0.0% no_bounce=0.0% attracts=0.0% no_attract=0.0% loud=0.0% quiet=0.0% bright=0.0% dark=0.0% hot=0.0% cold=0.0% rolls=0.0% no_roll=0.0% breaks=0.0% unbreakable=0.0%
Token length: 15
... snip ...