[Perf] Skip creating attention mask in llama dataloader

[billishyahao]

This patch is to skip creating attention mask in llama dataloader by adding flag --no-create-attention-mask-in-dataloader. With this patch, we can see multiple benefits below:

1. This could bring 4%~6% performance gain
2. Also address observed data loader crash issue when dealing with long sequence case e.g. [BUG] Long context training using context-parallel hangs/crashes NVIDIA/Megatron-LM#1025
3. We also see new megatron model example adopt this flag as well. e.g. https://github.com/NVIDIA/Megatron-LM/blob/40db706d37a25787b0fb6b7b561327e5d2b4b2e4/examples/mamba/train.sh#L102

[wenchenvincent]

@lizamd Are you aware of this change? Do we use this setting for testing?

[wenchenvincent]

@billishyahao Could you give some more details on the behavior --no-create-attention-mask-in-dataloader? For example, if attention mask is not created in dataloader, where is it created?

[lizamd]

@billishyahao could you provide more data on the 4-5% perf gain and address @wenchenvincent 's question? we can have a call too