Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TCR chain type special tokens #127

Merged
merged 96 commits into from
Jun 2, 2024
Merged

TCR chain type special tokens #127

merged 96 commits into from
Jun 2, 2024

Conversation

floccinauc
Copy link
Collaborator

No description provided.

VADIM RATNER [email protected] and others added 30 commits May 18, 2023 09:45
from_file for modular tokenizer
a5e842d
from_file for modular tokenizer
5a93fc8
add EOS token to modulartokenizer
9ad130b
modular tokenizer ops for t5 PPI
9aa0107
modular tokenizer ops adjustments
8e48881
modular tokenizer additions: ops
ecc12a6
adding special tokens to existing ModularTokenizer
dd896cb
adding special tokens to existing ModularTokenizer
08ec610
modular tokenizer ops bug fix
6233f40
mod tokenizer additions
5390a71
pr comments
d6ee4ea
modulartokenizer fixes, special tokens added
371baa0
modulartokenizer fixes, special tokens added
df9508b
modulartokenizer fixes, special tokens added
5c51567
formatting fixes
d33d9cc
PR fixes
33e4cad
readme update
d2d50b2
relative links test
b627873
relative links test
72e731a
@floccinauc
Merge branch 'main' into PPI_t5
34913e6
code cleanup
c012e23
Merge branch 'PPI_t5' of https://github.com/BiomedSciAI/fuse-drug int…
2de1cec 
…o PPI_t5
cleanup
b3e977b
cleanup
650ff6e
@floccinauc
Merge branch 'main' into PPI_t5
e4343f5
PR fix
4d66f49
PR fix
cc10b08
added special token
3819aaa
remove unused FastModularTokenizerOp operator
6587606
Merge branch 'main' of https://github.com/BiomedSciAI/fuse-drug into …
0d3b88b 
…PPI_t5
VADIM RATNER [email protected] and others added 25 commits February 6, 2024 05:01
Merge branch 'main' of https://github.com/BiomedSciAI/fuse-drug into …
626102d 
…PPI_t5
add a warning/exception on encountering unknown tokens and a method t…
32303fa 
…o count them to modular tokenizer
add a warning/exception on encountering unknown tokens and a method t…
9398cb7 
…o count them to modular tokenizer
PR fix
348df7f
update
6f9ad97
update
0e6fa55
update
4624474
detail level in unk token warnings during ModularTokenizer encode/enc…
0f1fa67 
…ode_list
detail level in unk token warnings during ModularTokenizer encode/enc…
2da0cdb 
…ode_list
detail level in unk token warnings during ModularTokenizer encode/enc…
e89f473 
…ode_list
detail level in unk token warnings during ModularTokenizer encode/enc…
52742da 
…ode_list
add verbose 0 version to ModularTokenizer.encode
2f733db
add verbose 0 version to ModularTokenizer.encode
40b58bd
main merge
22e5f64
Merge branch 'main' of https://github.com/BiomedSciAI/fuse-drug into …
6b1e229 
…PPI_t5
Merge branch 'main' of https://github.com/BiomedSciAI/fuse-drug into …
746f750 
…PPI_t5
save path compatibility fix for modular tokenizer
a4f309e
save path compatibility fix for modular tokenizer
4dc0e0f
Merge branch 'main' of https://github.com/BiomedSciAI/fuse-drug into …
aedce8b 
…PPI_t5
adding special tokens
d3348e4
add mutated token
407a9d1
travis fixes
4c87f49
Merge branch 'main' of https://github.com/BiomedSciAI/fuse-drug into …
b98584a 
…PPI_t5
TCR chain tokens
2a14ae4
TCR chain type tokens: alpha, gamma, delta and their CDR3 regions
14e1b7b
@floccinauc floccinauc requested a review from SagiPolaczek June 2, 2024 10:34
SagiPolaczek
SagiPolaczek approved these changes Jun 2, 2024
View reviewed changes
Copy link
Collaborator

@SagiPolaczek SagiPolaczek left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!! Please make sure to run the pre-commit hooks once again (pre-commit run --all-files for example).

Couple of general questions:

  1. We starting to have more and more tokenizers to sync between. Do we verify it / have some solution for it? Maybe inheritance?
  2. Alpha and Beta doesn't have _VAR (variable) region? I noticed you didn't add this kind of token for them.

@floccinauc
Copy link
Collaborator Author

LGTM, thanks!! Please make sure to run the pre-commit hooks once again (pre-commit run --all-files for example).

Couple of general questions:

  1. We starting to have more and more tokenizers to sync between. Do we verify it / have some solution for it? Maybe inheritance?
  2. Alpha and Beta doesn't have _VAR (variable) region? I noticed you didn't add this kind of token for them.
  1. Regarding the synching of multiple tokenizers - it's being taken care of by this script, and consistency between them is being tested in a unittest here, courtesy of
    Matan Ninio.
  2. They are all VAR regions, it's just that ALPHA and BETA already had existing tokens that are being used in that context by other tasks.

@floccinauc floccinauc merged commit 7fc56d9 into main Jun 2, 2024
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SagiPolaczek SagiPolaczek SagiPolaczek approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@floccinauc @SagiPolaczek