Fixed configuration of both llama2 and llama3

[indianspeedster]

Corrected the configuration for llam2-7b in train_llama2.sh.

The old configuration were different from what mentioned in Original llama2 and on Huggingface.

Modified changes:

FFN_HIDDEN_SIZE = 11008
NUM_KV_HEADS = 32

[lcskrishna]

@indianspeedster Thanks for fixing the values of 7B inside our scripts. Can you review train_llama3.sh as well just to avoid such errors in future.

[indianspeedster]

@indianspeedster Thanks for fixing the values of 7B inside our scripts. Can you review train_llama3.sh as well just to avoid such errors in future.

@lcskrishna I checked all the configurations. max_position_embeddings needed to be changed to match the original configuration of llama2 and llama3. I have changed those and rest all other configuration looks fine to me.

For reviewers:

Official llama2 70 b config : https://huggingface.co/meta-llama/Llama-2-70b-hf/blob/main/config.json
Official llama2 7 b config : https://huggingface.co/meta-llama/Llama-2-7b-hf/blob/main/config.json

Official llama3 8b config: https://huggingface.co/meta-llama/Llama-3.1-8B/blob/main/config.json
Official llama3 70b config: https://huggingface.co/meta-llama/Llama-3.1-70B/blob/main/config.json

[gurpreet-dhami]

We can omit this line if it is using the same value as defined initially at the top. isn't it ?
MAX_POSITION_EMBEDDINGS

[indianspeedster]

Hi @gurpreet-dhami,

Yes, MAX_POSITION_EMBEDDINGS is being repeated. I have removed it and also realized that SEQ_LENGTH is also being repetitive. Will remove that too.

[wenchenvincent]

@indianspeedster Could you write clear description in the commit message telling what each commit does instead of the general message "Update train_llama2.sh"?

[indianspeedster]

@wenchenvincent Modified the commit message of all the commits.