Missing vocab dependency for reproducing anti-scaling

[rerngvit]

First of all, thank you a lot for working on ParlAI and also contributing this Anti-scaling for knowledge distillation.

I have been experimenting with distilled model like Blender 400M Distilled, which seems to work fine. However, I run into an unexpected issue when I try to reproduce the distillation process according to the documentation. The error says something like "OSError: File /checkpoint/parlai/zoo/meena/20200319_meenav0data_tall_2.7B_adamoptimizer/20200319_13.3ppl_200kupdates/model.dict-vocab.json does not exist. --bpe-vocab must be pretrained.". It would be great if you could help clarify things (@EricMichaelSmith).

The steps to reproduce the issue are the following.

1. Download master version git clone https://github.com/facebookresearch/ParlAI.git
2. Change directory into the cloned folder
3. Setup for development !python setup.py develop
4. Manually download the BlenderBot3B model python -c "from parlai.zoo.blender.blender_3B import download; download('data')". Until this step, everything seems to be working as expected.
5. For the last step of starting the distillation, this is where the problem occurs.
   (I assume that SAVE_PATH environment variable can be any path inside the local file system, which I put "data/models/blender_distilled_test" as the value.)
   !parlai train_model \ --allow-missing-init-opts True \ --init-model None \ --init-opt data/models/blender/blender_3B/model.opt \ --model projects.anti_scaling.distillation:DistillNarrowTransformerAgent \ --teacher-model data/models/blender/blender_3B/model \ --batchsize 16 \ --embedding-size 2560 \ --ffn-size 10240 \ --fp16 True \ --gpu -1 \ --learningrate 0.0001 \ --lr-scheduler reduceonplateau \ --max-lr-steps -1 \ --max-train-time -1 \ --model-parallel False \ --save-after-valid True \ --skip-generation True \ --task blended_skill_talk,wizard_of_wikipedia,convai2:normalized,empathetic_dialogues \ -veps -1 \ -vmm min \ -vmt ppl \ -vp 20 \ -vtim 900 \ --n-encoder-layers 2 \ --n-decoder-layers 9 \ --embedding-loss-coeff 4 \ --hidden-loss-coeff 64 \ --self-attn-loss-coeff 4 \ --enc-dec-attn-loss-coeff 64 \ --encoder-loss-coeff 0 \ --pred-loss-coeff 64 \ --task-loss-coeff 1 \ --model-file data/models/blender_distilled_test

From here on, I assume that the training process should be started. However, I get the following error. It seems to be like local development dependency is an issue ("/checkpoint/parlai/zoo/meena/20200319_meenav0data_tall_2.7B_adamoptimizer/20200319_13.3ppl_200kupdates/model.dict-vocab.json does not exist. "). Could you please guide on how to resolve this or obtain this JSON file?
20:59:42 | The "show_advanced_args" key in data/models/blender/blender_3B/model.opt will not be loaded, because it does not exist in the target opt. 20:59:42 | The "numthreads" key in data/models/blender/blender_3B/model.opt will not be loaded, because it does not exist in the target opt. 20:59:42 | The "max_lr_steps" key in data/models/blender/blender_3B/model.opt will not be loaded, because it does not exist in the target opt. 20:59:42 | The "starttime" key in data/models/blender/blender_3B/model.opt will not be loaded, because it does not exist in the target opt. 20:59:42 | building dictionary first... Traceback (most recent call last): File "/usr/local/bin/parlai", line 33, in <module> sys.exit(load_entry_point('parlai', 'console_scripts', 'parlai')()) File "/content/ParlAI/parlai/__main__.py", line 14, in main superscript_main() File "/content/ParlAI/parlai/core/script.py", line 306, in superscript_main return SCRIPT_REGISTRY[cmd].klass._run_from_parser_and_opt(opt, parser) File "/content/ParlAI/parlai/core/script.py", line 89, in _run_from_parser_and_opt return script.run() File "/content/ParlAI/parlai/scripts/train_model.py", line 912, in run self.train_loop = TrainLoop(self.opt) File "/content/ParlAI/parlai/scripts/train_model.py", line 339, in __init__ build_dict(opt, skip_if_built=True) File "/content/ParlAI/parlai/scripts/build_dict.py", line 79, in build_dict dictionary = str2class(opt['dict_class'])(opt) File "/content/ParlAI/parlai/core/dict.py", line 322, in __init__ self.bpe = bpe_factory(opt, shared) File "/content/ParlAI/parlai/utils/bpe.py", line 62, in bpe_factory bpe_helper = HuggingFaceBpeHelper(opt, shared) File "/content/ParlAI/parlai/utils/bpe.py", line 841, in __init__ f'File {self.vocab_path} does not exist. --bpe-vocab must be pretrained.' OSError: File /checkpoint/parlai/zoo/meena/20200319_meenav0data_tall_2.7B_adamoptimizer/20200319_13.3ppl_200kupdates/model.dict-vocab.json does not exist. --bpe-vocab must be pretrained.

[stephenroller]

Lol quite the filename

Try adding --dict-file zoo:blender/blender_3B/model.dict and see if that helps

[rerngvit]

Thank you @stephenroller for your prompt response. Adding the dict file property as you suggested did solve the problem.

[stephenroller]

Assigning issue to @EricMichaelSmith to update docs

[EricMichaelSmith]

Yes, good catch - just opened #3561 for this