21.2. Task 1: Natural Language Inference (NLI)
Alpha Version: Work in progress.
This section covers fine-tuning and inference for the Natural Language Inference (NLI) task.
21.2.1. Dataset
As mentioned at the beginning of this chapter, NLI is a task where the model receives a premise and a hypothesis to determine whether the hypothesis logically follows (entailment), contradicts (contradiction), or is neutral relative to the premise.
Given a “Premise” and a “Hypothesis”, the model predicts whether the hypothesis is an entailment, neutral, or contradiction relative to the premise.
The SNLI is a supervised learning dataset containing a large number of pairs like the ones shown below:
{
"annotator_labels": ["neutral"],
"gold_label": "entailment",
"sentence1": "A person on a horse jumps over a broken down airplane.",
"sentence2": "A person is outdoors, on a horse.",
}
...
{
"annotator_labels": ["contradiction"],
"gold_label": "contradiction",
"sentence1": "A black race car starts up in front of a crowd of people.",
"sentence2": "A man is driving down a lonely road.",
}
...We use this structure directly. The inputs are split into “Premise” and “Hypothesis” to train the model as a 3-option classification problem (entailment, neutral, contradiction).
21.2.2. Fine-Tuning
The gpt1_finetune.py script performs fine-tuning for NLI.
Usage: gpt1_finetune.py trainnig-epoch pretrained-checkpoint-dirRun the following command to start fine-tuning. Make sure to specify the checkpoint directory containing your pre-trained weights:
$ python gpt1_finetune.py 2 checkpoints/gpt1-pretrain-layers-12-d_model-512-d_ffn-2048-heads-8/Training for 2 to 3 epochs is sufficient, which takes a few hours.
21.2.3. Inference
21.2.3.1. Interactive Mode
The gpt1_inference.py script executes inference for NLI.
Usage: gpt1_inference.py pretrained-checkpoint-dir [premise-hypothesis-file.tsv]You can launch interactive mode with the following command:
$ python gpt1_inference.py checkpoints/gpt1-finetune-nli-layers-12-d_model-512-d_ffn-2048-heads-8/Input any Premise and Hypothesis to see the model’s prediction.
============================================================
GPT-1 NLI Inference — Interactive Mode
Enter 'q' or leave empty to exit
============================================================
Premise > A man is playing guitar on stage.
Hypothesis > A man is playing a musical instrument.
Premise : A man is playing guitar on stage.
Hypothesis : A man is playing a musical instrument.
Prediction : ✅ entailment (confidence: 99.2%)
Scores : entailment=99.2% neutral=0.8% contradiction=0.0%
Token length : 19
Premise > A man is playing guitar on stage.
Hypothesis > The man is sleeping in bed.
Premise : A man is playing guitar on stage.
Hypothesis : The man is sleeping in bed.
Prediction : ❌ contradiction (confidence: 100.0%)
Scores : entailment=0.0% neutral=0.0% contradiction=100.0%
Token length : 18
Premise > Since the SNLI corpus consists of very simple sentences, the model often struggles to understand complex phrasing. This clearly demonstrates how the quality and nature of the training data directly impact model performance.
21.2.3.2. File Mode
You can also prepare a tab-separated values (TSV) file where the Premise and Hypothesis are separated by a tab character, as shown below:
$ cat snli.tsv
A man is playing guitar on stage. A man is playing a musical instrument.
Two dogs are running in the park. There are animals outside.
A woman is cooking dinner in the kitchen. A woman is preparing food.
A man is playing guitar on stage. The man is a famous musician.
Two dogs are running in the park. The dogs are chasing a ball.
A woman is cooking dinner in the kitchen. She is making pasta for her family.
A man is playing guitar on stage. The man is sleeping in bed.
Two dogs are running in the park. The dogs are sitting still indoors.
A woman is cooking dinner in the kitchen. No one is in the kitchen.By passing this file to the inference script, the model will run predictions on all the pairs and display the results.
$ python gpt1_inference.py checkpoints/gpt1-finetune-nli-layers-12-d_model-512-d_ffn-2048-heads-8/ snli.tsv
... snip ...
[File] Processing snli.tsv ...
[Pair 1]
Premise : A man is playing guitar on stage.
Hypothesis : A man is playing a musical instrument.
Prediction : ✅ entailment (confidence: 99.2%)
Scores : entailment=99.2% neutral=0.8% contradiction=0.0%
Token length : 19
[Pair 2]
Premise : Two dogs are running in the park.
Hypothesis : There are animals outside.
Prediction : ✅ entailment (confidence: 99.9%)
Scores : entailment=99.9% neutral=0.1% contradiction=0.0%
Token length : 16
[Pair 3]
Premise : A woman is cooking dinner in the kitchen.
Hypothesis : A woman is preparing food.
Prediction : ✅ entailment (confidence: 99.0%)
Scores : entailment=99.0% neutral=1.0% contradiction=0.0%
Token length : 18
[Pair 4]
Premise : A man is playing guitar on stage.
Hypothesis : The man is a famous musician.
Prediction : ➖ neutral (confidence: 99.6%)
Scores : entailment=0.4% neutral=99.6% contradiction=0.1%
Token length : 18
[Pair 5]
Premise : Two dogs are running in the park.
Hypothesis : The dogs are chasing a ball.
Prediction : ➖ neutral (confidence: 99.9%)
Scores : entailment=0.0% neutral=99.9% contradiction=0.0%
Token length : 18
[Pair 6]
Premise : A woman is cooking dinner in the kitchen.
Hypothesis : She is making pasta for her family.
Prediction : ➖ neutral (confidence: 99.3%)
Scores : entailment=0.5% neutral=99.3% contradiction=0.2%
Token length : 20
[Pair 7]
Premise : A man is playing guitar on stage.
Hypothesis : The man is sleeping in bed.
Prediction : ❌ contradiction (confidence: 100.0%)
Scores : entailment=0.0% neutral=0.0% contradiction=100.0%
Token length : 18
[Pair 8]
Premise : Two dogs are running in the park.
Hypothesis : The dogs are sitting still indoors.
Prediction : ❌ contradiction (confidence: 100.0%)
Scores : entailment=0.0% neutral=0.0% contradiction=100.0%
Token length : 18
[Pair 9]
Premise : A woman is cooking dinner in the kitchen.
Hypothesis : No one is in the kitchen.
Prediction : ❌ contradiction (confidence: 99.8%)
Scores : entailment=0.1% neutral=0.1% contradiction=99.8%
Token length : 19
Done: 9 pairs processed / 0 lines skipped