Simplify lang detection #299
Conversation
ManyTheFish
force-pushed
the
simplify-lang-detection
branch
from
9efb459 to
f70289d
Compare
ManyTheFish
force-pushed
the
simplify-lang-detection
branch
from
f70289d to
599bb3b
Compare
| assert_eq!(Language::Eng.code(), "eng"); | ||
| assert_eq!(Language::from_code("eng"), Some(Language::Eng)); | ||
| assert_eq!(Language::Jpn.code(), "jpn"); | ||
| assert_eq!(Language::from_code("jpn"), Some(Language::Jpn)); | ||
| assert_eq!(Language::Cmn.code(), "cmn"); | ||
| assert_eq!(Language::from_code("cmn"), Some(Language::Cmn)); |
There was a problem hiding this comment.
not sure if this is relevant, but I thought we were using https://en.wikipedia.org/wiki/List_of_ISO_639_language_codes. I would have expected just en for English, ja for Japanese, etc.
Looks like we're not using set 1, but the other sets? Is set 1 usable?
There was a problem hiding this comment.
These codes are inherited from whatlang, which follows the ISO-639-3, and the bigrams codes come from the ISO-639-1, which is an anterior version.
Is there any issue with using a 3-gram code?
There was a problem hiding this comment.
This is about user expection. Maybe that's just me, but as a user, I would expect my language codes to be "EN" or "en", not "ENG"
The best solution would be to support both en and eng.
Co-authored-by: Louis Dureuil <[email protected]>
|
Build succeeded:
|
allow_listfrom a map of script->language to an array of allowed languages