[retry] Use pytorch-labs/tokenizers and remove tokenizer/ (#1401)

[larryliu0820]

Summary: Retry of #1401

Test Plan: Re-run the repro command in #1413:

python3 torchchat.py generate llama3.2-1b-base --prompt "write me a story about a boy and his bear"

Reviewers:

Subscribers:

Tasks:

Tags:

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/1443

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 1 Pending

As of commit 2075fd7 with merge base 019f76f ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[larryliu0820]

Getting "model file path should not be empty." from sentencepiece processor.

According to chatgpt:

The _GLIBCXX_USE_CXX11_ABI macro is related to the Application Binary Interface (ABI) used by the GNU C++ Standard Library.
In 2011, the C++11 standard introduced a new ABI for C++ objects, which changed the way that standard library objects, including std::string, are laid out in memory. This new ABI is enabled by default in GCC 5 and later versions.
However, when _GLIBCXX_USE_CXX11_ABI=0 is defined, it tells the compiler to use the older, pre-C++11 ABI instead. This can cause issues with std::string objects because the older ABI uses a different memory layout for these objects.
When you pass a const char* pointer to a function that expects a std::string object, the compiler needs to create a temporary std::string object from the pointer. However, if the ABI is set to the older version, the compiler may not be able to correctly create this temporary object, resulting in an empty string being passed to the function.
This issue is likely due to the fact that the older ABI uses a different memory layout for std::string objects, which includes a pointer to the string data, a length field, and a capacity field. When the compiler creates a temporary std::string object from a const char* pointer, it needs to initialize these fields correctly. However, if the ABI is set to the older version, the compiler may not be able to correctly initialize these fields, resulting in an empty string.

[Jack-Khuu]

Thanks for the fix!!

xrf: #1440