在 multimodal Twitter dataset 上使用 vilbert-multi-task

目录

1. 图片特征提取：
2. 依赖安装

https://github.com/facebookresearch/vilbert-multi-task

https://gitlab.com/vedanuj/vqa-maskrcnn-benchmark

http://lil.nlp.cornell.edu/nlvr/NLVR2BiasAnalysis.html

https://github.com/lil-lab/nlvr/tree/master/nlvr2

Foil 也是二分类问题，所以我从 Foil 任务改造出 multimodal Twitter dataset 任务

Foil 数据集：

https://foilunitn.github.io/

图片特征提取：

说明：https://github.com/facebookresearch/vilbert-multi-task/tree/master/data

代码：https://github.com/facebookresearch/vilbert-multi-task/blob/master/script/extract_features.py

使用了 maskrcnn_benchmark 得到图片中物体的位置

依赖安装

项目的 requirements.txt 有问题

https://github.com/facebookresearch/vilbert-multi-task/issues/14

pytorch-transformers==1.0.0 要改为 pytorch-transformers==1.1.0

需要安装 h5py

(vilbert-multi-task) (base) xxx@xxxx-xxx:~/multimodal_sarcasm_detection/Code/related_work/vilbert-multi-task/vilbert-multi-task$ python train_tasks.py –bert_model bert-base-uncased –from_pretrained ../multi_task_model.bin –config_file config/bert_base_6layer_6conect.json –tasks 19 –lr_scheduler ‘warmup_linear’ –train_iter_gap 4 –task_specific_tokens –save_name multi_task_model

python eval_tasks.py –bert_model bert-base-uncased –from_pretrained ../multi_task_model.bin –config_file config/bert_base_6layer_6conect.json –tasks 19 –lr_scheduler ‘warmup_linear’ –train_iter_gap 4 –task_specific_tokens –save_name multi_task_model

python train_tasks.py --bert_model bert-base-uncased --from_pretrained ../multi_task_model.bin --config_file config/bert_base_6layer_6conect.json --tasks 19 --lr_scheduler 'warmup_linear' --train_iter_gap 4 --task_specific_tokens --save_name multi_task_model