上半部分介紹了如何從BERT模型提取嵌入，下半部分介紹如何針對下游任務進行微調，分為四個任務。
下游任務：
微調方式：
- 分類器層與BERT模型一起更新權重（通常情況且效果更好）
- 僅更新分類器層的權重而不更新BERT模型的權重。BERT模型僅作為特征提取器
1 自然語言推理任務
1.1 任務說明
確定在給定前提下，一個假設是必然的（真）、矛盾的（假）還是未定的（中性）
自然語言推理任務的樣本數據集
1.2 代碼
- snli數據集文檔: https://huggingface.co/datasets/snli
- pipline文檔: https://huggingface.co/docs/transformers/main_classes/pipelines
- 查看所有評估指標:
evaluate.list_evaluation_modules("metric")
from transformers import AutoModelForSequenceClassification, AutoTokenizer, Trainer, TrainingArguments
from transformers import DataCollatorWithPadding
from nlp import load_dataset
import evaluate
import numpy as np
from transformers import pipeline
from transformers.pipelines.pt_utils import KeyDataset, KeyPairDataset
# entailment: 包含, neutral: 中性, contradiction: 矛盾
id2label = {0: "entailment", 1: "neutral", 2: "contradiction"}
label2id = {"entailment": 0, "neutral": 1, "contradiction": 2}
def train():
# snli數據集: https://huggingface.co/datasets/snli
# snli數據集可視化：https://huggingface.co/datasets/viewer/?dataset=snli
dataset = load_dataset('snli', split=['train[:10%]', 'validation[:100]'])
print('dataset: {}'.format(dataset))
train_set = dataset[0].filter(lambda x: x['label'] != -1)
test_set = dataset[1].filter(lambda x: x['label'] != -1)
print('train_set[0]: {}'.format(train_set[0]))
print('test_set[0]: {}'.format(test_set[0]))
print('train_set: {}'.format(train_set))
print('test_set: {}'.format(test_set))
model = AutoModelForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=3, id2label=id2label, label2id=label2id)
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
def preprocess_function(examples):
return tokenizer(examples['premise'], examples['hypothesis'], truncation=True)
train_set = train_set.map(preprocess_function, batched=True)
test_set = test_set.map(preprocess_function, batched=True)
# 按一個batch的最大長度補齊，而非整個數據集的最大長度補齊
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)
# 查看所有評估指標: evaluate.list_evaluation_modules("metric")
accuracy = evaluate.load('accuracy')
def compute_metrics(eval_pred):
predictions, labels = eval_pred
predictions = np.argmax(predictions, axis=1)
return accuracy.compute(predictions=predictions, references=labels)
# 預訓練參數
training_args = TrainingArguments(
output_dir='./results',
logging_dir='./logs',
optim='adamw_torch',
learning_rate=2e-5,
per_device_train_batch_size=8,
per_device_eval_batch_size=8,
num_train_epochs=10,
weight_decay=0.01,
warmup_steps=500,
evaluation_strategy='epoch',
save_strategy='epoch',
load_best_model_at_end=True,
metric_for_best_model='accuracy',
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_set,
eval_dataset=test_set,
tokenizer=tokenizer,
data_collator=data_collator,
compute_metrics=compute_metrics
)
trainer.train()
trainer.evaluate()
def inference():
# 推導，真實label=1/neutral，訓練完這個case預測錯了0v0
premise = 'This church choir sings to the masses as they sing joyous songs from the book at a church.'
hypothesis = 'The church has cracks in the ceiling.'
# pipline類型: https://huggingface.co/docs/transformers/main_classes/pipelines
classifier = pipeline('text-classification', model='./results/checkpoint-68700', tokenizer='bert-base-uncased')
result = classifier({"text": premise, "text_pair": hypothesis}, top_k=None)
print('premise: {}\nhypothesis: {}\nresult: {}\n'.format(premise, hypothesis, result))
print('開始預測整個測試集')
test_set = load_dataset('snli', split='test[:100]').filter(lambda x: x['label'] != -1)
predictions = []
for out in classifier(KeyPairDataset(test_set, 'premise', 'hypothesis')):
predictions.append(label2id[out['label']])
print('predictions: {}'.format(predictions))
labels = []
for label in KeyDataset(test_set, 'label'):
labels.append(label)
print('labels: {}'.format(labels))
# 有時候需要開代理, 不然會卡住
accuracy = evaluate.load('accuracy')
print(accuracy.compute(predictions=predictions, references=labels))
# premise: This church choir sings to the masses as they sing joyous songs from the book at a church.
# hypothesis: The church has cracks in the ceiling.
# result: [{'label': 'contradiction', 'score': 0.9999877214431763}, {'label': 'neutral', 'score': 1.0622567970131058e-05}, {'label': 'entailment', 'score': 1.7016249103107839e-06}]
# 開始預測整個測試集
# predictions: [2, 0, 2, 1, 1, 1, 0, 1, 2, 1, 0, 2, 1, 0, 2, 2, 0, 1, 1, 2, 1, 2, 0, 1, 1, 2, 0, 0, 1, 2, 2, 1, 0, 0, 1, 2, 0, 0, 1, 0, 1, 2, 2, 1, 1, 1, 1, 0, 1, 2, 1, 0, 2, 1, 2, 0, 0, 2, 1, 0, 2, 1, 2, 0, 2, 0, 0, 2, 2, 1, 0, 2, 0, 1, 2, 0, 1, 0, 1, 0, 2, 2, 0, 0, 2, 1, 2, 1, 0, 1, 2, 0, 1, 0, 2, 2, 0, 1, 1]
# labels: [1, 0, 2, 1, 0, 2, 0, 1, 2, 1, 0, 2, 0, 0, 2, 2, 0, 1, 0, 2, 1, 2, 0, 1, 1, 2, 0, 0, 1, 2, 2, 0, 0, 0, 1, 2, 1, 0, 2, 0, 1, 2, 2, 0, 1, 1, 2, 0, 1, 2, 1, 0, 2, 1, 2, 0, 0, 2, 1, 0, 2, 1, 1, 0, 2, 0, 1, 1, 2, 1, 0, 2, 0, 1, 2, 0, 1, 0, 0, 2, 2, 2, 0, 0, 2, 1, 2, 1, 0, 1, 2, 0, 1, 0, 2, 2, 0, 1, 1]
# {'accuracy': 0.8484848484848485}
參考資料
[1]. BERT基礎教程Transformer大模型實戰
[2]. pipline文檔: https://huggingface.co/docs/transformers/main_classes/pipelines
[3]. snli數據集文檔: https://huggingface.co/datasets/snli
[4]. tokenizer函數的參數truncation=True是什么意思：https://blog.csdn.net/weixin_54227557/article/details/130308052
