Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release new GliNER models #42

Merged
merged 43 commits into from
Aug 9, 2024
Merged

Release new GliNER models #42

merged 43 commits into from
Aug 9, 2024

Conversation

ljvmiranda921
Copy link
Owner

@ljvmiranda921 ljvmiranda921 commented Aug 8, 2024

Reference: #40

Release new GLiNER models

The evaluation results for TLUnified-NER are shown in the table below (reported numbers are F1-scores):

PER ORG LOC Overall
tl_gliner_small 86.76 78.72 86.78 84.83
tl_gliner_medium 87.46 79.71 86.75 85.40
tl_gliner_large 86.75 80.20 86.76 85.72
tl_calamancy_trf 91.95 84.84 88.92 88.03
span-marker 92.57 82.04 90.56 89.62

In general, GliNER gets decent scores, but nothing beats regular finetuning on BERT-based models as seen in tl_calamancy_trf and span_marker. The performance on Universal NER is generally worse (the highest is around ~50%), compared to the reported results in the Universal NER paper (we finetuned on RoBERTa as well). One possible reason is that the annotation guidelines for TULunified-NER are more loose, because we consider some entities that Universal NER ignores. At the same time, the text distribution of the two datasets are widely different.

Nevertheless, I'm still releasing these GliNER models as they are very extensible to other entity types (and it's also nice to have a finetuned version of GliNER for Tagalog!). I haven't done any extensive hyperparameter tuning here so it might be nice if someone can contribute better config parameters to bump up these scores.

@ljvmiranda921 ljvmiranda921 changed the title WIP: Experiments on GliNER models Release new GliNER models Aug 9, 2024
@ljvmiranda921 ljvmiranda921 marked this pull request as ready for review August 9, 2024 18:26
@ljvmiranda921 ljvmiranda921 merged commit 7c31f98 into master Aug 9, 2024
14 checks passed
@ljvmiranda921 ljvmiranda921 deleted the updated_ner_evals branch August 9, 2024 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant