The EVA algorithms described in van den Akker et al. (2020) justifies and proposes a number of general disambiguation procedures and the sequence of their execution. This general algorithm can be tuned in many ways for specific needs at hand. Below we demonstrate a comparison between different versions of the algorithm and discuss some implications.
The sample test data is based on the data used to test another disambiguation algorithm described by in Tekles & Bornmann (2019). The test data is a bibliographic records from Web of Science. We reproduced the smaller random sample of the Tekles & Bornmann (2019) by using the same Researcher IDs provided to us courtesy by the authors. We use these Researcher IDs and the same query parameters to export the sample data from Web of Science.
The table below presents comparison of EVA algorithms. The motivation for these versions is described in van den Akker et al. (2020). We used several common metrics to assess the efficiency and performance of each version.
| name | true_positives | false_positives | false_negatives | true_negatives | pw_presision | pw_recall | pw_f1 | pw_accuracy | dur_mins |
|---|---|---|---|---|---|---|---|---|---|
| eva-slow-dl11 | 2977 | 89 | 2460 | -5524 | 0.971 | 0.548 | 0.7 | -1274 | 35 |
| eva-slow-dl01 | 2976 | 38 | 2461 | -5473 | 0.987 | 0.547 | 0.704 | -1248 | 36.3 |
| eva-slow-dl124 | 2977 | 102 | 2460 | -5537 | 0.967 | 0.548 | 0.699 | -1280 | 37.6 |
| eva-slow-dl125 | 2977 | 89 | 2460 | -5524 | 0.971 | 0.548 | 0.7 | -1274 | 34.2 |
| eva-slow-dl126 | 2977 | 89 | 2460 | -5524 | 0.971 | 0.548 | 0.7 | -1274 | 33.9 |
| eva-slow-dl11-c | 83 | 4 | 5354 | NA | 0.954 | 0.0153 | 0.0301 | NA | 0.73 |
| eva-slow-dl11-e | 1294 | 0 | 4143 | 1667705 | 1 | 0.238 | 0.384 | 0.998 | 0.741 |
| eva-slow-dl11-k | 503 | 6 | 4934 | NA | 0.988 | 0.0925 | 0.169 | NA | 3.98 |
| eva-slow-dl11-x | 233 | 0 | 5204 | NA | 1 | 0.0429 | 0.0822 | NA | 14.4 |
| eva-slow-dl11-eic2s | 2977 | 89 | 2460 | -5524 | 0.971 | 0.548 | 0.7 | -1274 | 33.9 |
| eva-slow-dl11-eic2sk2 | 2981 | 92 | 2456 | -5527 | 0.97 | 0.548 | 0.701 | -1273 | 34 |
| eva-slow-dl11-eicsk2 | 2981 | 92 | 2456 | -5527 | 0.97 | 0.548 | 0.701 | -1273 | 33.9 |
| eva-slow-dl11-i | 2362 | 77 | 3075 | 5940768 | 0.968 | 0.434 | 0.6 | 0.999 | 0.757 |
The barchart below shows duration profiles of procedure for different EVA algorithms.
The tentative results shows that the baseline version of the algorithm eva-slow-dl01 has relatively better performance (based on F1 index) but it is also one of the most time consuming.
The reproducible code snippets for each variations to EVA algorithm that we considered for the analysis are below. More in depth investigation will follow.
options(disambr_get_output_set = TRUE)
options(disambr_read_output_set = TRUE)
options(disambr_mess_pretty = TRUE)
options(disambr_save_as = TRUE)
options(disambr_save_set_dir = "../eva-slow-dl11")
ds <-
disambr_read("../data/wos-slow-export-subset"
, save_sets_as = "wos-slow-export-subset.rds"
, save_sets_dir = "../data") %>%
disambr_set_tekles_bornmann(
file_path = "./inst/testdata/tekles-bornmann-researcher-ids.txt") %>%
disambr_set_on_same_paper %>%
disambr_set_similar_initials %>%
disambr_set_similar_last_names %>%
disambr_set_same_email %>%
disambr_set_same_affiliation %>%
disambr_set_cite_others_paper %>%
disambr_set_common_references %>%
disambr_set_cite_self_citation %>%
disambr_set_common_keywords %>%
disambr_set_same_researcher_idsoptions(disambr_get_output_set = TRUE)
options(disambr_read_output_set = TRUE)
options(disambr_mess_pretty = TRUE)
options(disambr_save_as = TRUE)
options(disambr_save_set_dir = "../eva-slow-dl01")
ds <-
disambr_read("../data/wos-slow"
, save_sets_as = "wos-slow.rds"
, save_sets_dir = "../disambr-data") %>%
disambr_set_tekles_bornmann(file_path = "../data/tekles-bornmann-researcher-ids.txt") %>%
disambr_set_on_same_paper %>%
disambr_set_similar_initials %>%
disambr_set_similar_last_names(max_dist = 1
, max_dist_short = 0
, min_length = 4) %>%
disambr_set_same_email %>%
disambr_set_same_affiliation %>%
disambr_set_cite_others_paper %>%
disambr_set_common_references %>%
disambr_set_cite_self_citation %>%
disambr_set_common_keywords %>%
disambr_set_same_researcher_idsoptions(disambr_get_output_set = TRUE)
options(disambr_read_output_set = TRUE)
options(disambr_mess_pretty = TRUE)
options(disambr_save_as = TRUE)
options(disambr_save_set_dir = "../eva-slow-dl124")
ds <-
disambr_read("../data/wos-slow"
, save_sets_as = "wos-slow.rds"
, save_sets_dir = "../disambr-data") %>%
disambr_set_tekles_bornmann(file_path = "../data/tekles-bornmann-researcher-ids.txt") %>%
disambr_set_on_same_paper %>%
disambr_set_similar_initials %>%
disambr_set_similar_last_names(max_dist = 2
, max_dist_short = 1
, min_length = 4) %>%
disambr_set_same_email %>%
disambr_set_same_affiliation %>%
disambr_set_cite_others_paper %>%
disambr_set_common_references %>%
disambr_set_cite_self_citation %>%
disambr_set_common_keywords %>%
disambr_set_same_researcher_idsoptions(disambr_get_output_set = TRUE)
options(disambr_read_output_set = TRUE)
options(disambr_mess_pretty = TRUE)
options(disambr_save_as = TRUE)
options(disambr_save_set_dir = "../eva-slow-dl125")
ds <-
disambr_read("../data/wos-slow"
, save_sets_as = "wos-slow.rds"
, save_sets_dir = "../disambr-data") %>%
disambr_set_tekles_bornmann(file_path = "../data/tekles-bornmann-researcher-ids.txt") %>%
disambr_set_on_same_paper %>%
disambr_set_similar_initials %>%
disambr_set_similar_last_names(max_dist = 2
, max_dist_short = 1
, min_length = 5) %>%
disambr_set_same_email %>%
disambr_set_same_affiliation %>%
disambr_set_cite_others_paper %>%
disambr_set_common_references %>%
disambr_set_cite_self_citation %>%
disambr_set_common_keywords %>%
disambr_set_same_researcher_idsoptions(disambr_get_output_set = TRUE)
options(disambr_read_output_set = TRUE)
options(disambr_mess_pretty = TRUE)
options(disambr_save_as = TRUE)
options(disambr_save_set_dir = "../eva-slow-dl126")
ds <-
disambr_read("../data/wos-slow"
, save_sets_as = "wos-slow.rds"
, save_sets_dir = "../disambr-data") %>%
disambr_set_tekles_bornmann(file_path = "../data/tekles-bornmann-researcher-ids.txt") %>%
disambr_set_on_same_paper %>%
disambr_set_similar_initials %>%
disambr_set_similar_last_names(max_dist = 2
, max_dist_short = 1
, min_length = 6) %>%
disambr_set_same_email %>%
disambr_set_same_affiliation %>%
disambr_set_cite_others_paper %>%
disambr_set_common_references %>%
disambr_set_cite_self_citation %>%
disambr_set_common_keywords %>%
disambr_set_same_researcher_idsoptions(disambr_get_output_set = TRUE)
options(disambr_read_output_set = TRUE)
options(disambr_mess_pretty = TRUE)
options(disambr_save_as = TRUE)
options(disambr_save_set_dir = "../eva-slow-dl11-eic2s")
ds <-
disambr_read("../data/wos-slow"
, save_sets_as = "wos-slow.rds"
, save_sets_dir = "../disambr-data") %>%
disambr_set_tekles_bornmann(file_path = "../data/tekles-bornmann-researcher-ids.txt") %>%
disambr_set_on_same_paper %>%
disambr_set_similar_initials %>%
disambr_set_similar_last_names(max_dist = 1
, max_dist_short = 0
, min_length = 0) %>%
disambr_set_same_email %>%
disambr_set_same_affiliation %>%
disambr_set_cite_others_paper %>%
disambr_set_common_references(references_in_common = 2) %>%
disambr_set_cite_self_citation %>%
disambr_set_common_keywords %>%
disambr_set_same_researcher_idsoptions(disambr_get_output_set = TRUE)
options(disambr_read_output_set = TRUE)
options(disambr_mess_pretty = TRUE)
options(disambr_save_as = TRUE)
options(disambr_save_set_dir = "../eva-slow-dl11-eicsk2")
ds <-
disambr_read("../data/wos-slow"
, save_sets_as = "wos-slow.rds"
, save_sets_dir = "../disambr-data") %>%
disambr_set_tekles_bornmann(file_path = "../data/tekles-bornmann-researcher-ids.txt") %>%
disambr_set_on_same_paper %>%
disambr_set_similar_initials %>%
disambr_set_similar_last_names(max_dist = 1
, max_dist_short = 0
, min_length = 0) %>%
disambr_set_same_email %>%
disambr_set_same_affiliation %>%
disambr_set_cite_others_paper %>%
disambr_set_common_references %>%
disambr_set_cite_self_citation %>%
disambr_set_common_keywords(keywords_in_common = 2) %>%
disambr_set_same_researcher_idsoptions(disambr_get_output_set = TRUE)
options(disambr_read_output_set = TRUE)
options(disambr_mess_pretty = TRUE)
options(disambr_save_as = TRUE)
options(disambr_save_set_dir = "../eva-slow-dl11-eic2sk2")
ds <-
disambr_read("../data/wos-slow"
, save_sets_as = "wos-slow.rds"
, save_sets_dir = "../disambr-data") %>%
disambr_set_tekles_bornmann(file_path = "../data/tekles-bornmann-researcher-ids.txt") %>%
disambr_set_on_same_paper %>%
disambr_set_similar_initials %>%
disambr_set_similar_last_names(max_dist = 1
, max_dist_short = 0
, min_length = 0) %>%
disambr_set_same_email %>%
disambr_set_same_affiliation %>%
disambr_set_cite_others_paper %>%
disambr_set_common_references(references_in_common = 2) %>%
disambr_set_cite_self_citation %>%
disambr_set_common_keywords(keywords_in_common = 2) %>%
disambr_set_same_researcher_idsoptions(disambr_get_output_set = TRUE)
options(disambr_read_output_set = TRUE)
options(disambr_mess_pretty = TRUE)
options(disambr_save_as = TRUE)
options(disambr_save_set_dir = "../data/disambr-data/eva-slow-dl11-e")
ds <-
disambr_read("../data/wos-slow"
, save_sets_as = "wos-slow.rds"
, save_sets_dir = "../data/disambr-data/disambr-data") %>%
disambr_set_tekles_bornmann(file_path = "./inst/testdata/tekles-bornmann-researcher-ids.txt") %>%
disambr_set_on_same_paper %>%
disambr_set_similar_initials %>%
disambr_set_similar_last_names %>%
disambr_set_same_email %>%
## disambr_set_same_affiliation %>%
## disambr_set_cite_others_paper %>%
## disambr_set_common_references %>%
## disambr_set_cite_self_citation %>%
## disambr_set_common_keywords %>%
disambr_set_same_researcher_idsoptions(disambr_get_output_set = TRUE)
options(disambr_read_output_set = TRUE)
options(disambr_mess_pretty = TRUE)
options(disambr_save_as = TRUE)
options(disambr_save_set_dir = "../eva-slow-dl11-i")
ds <-
disambr_read("../data/wos-slow"
, save_sets_as = "wos-slow.rds"
, save_sets_dir = "../disambr-data") %>%
disambr_set_tekles_bornmann(file_path = "../data/tekles-bornmann-researcher-ids.txt") %>%
disambr_set_on_same_paper %>%
disambr_set_similar_initials %>%
disambr_set_similar_last_names %>%
## disambr_set_same_email %>%
disambr_set_same_affiliation %>%
## disambr_set_cite_others_paper %>%
## disambr_set_common_references %>%
## disambr_set_cite_self_citation %>%
## disambr_set_common_keywords %>%
disambr_set_same_researcher_idsoptions(disambr_get_output_set = TRUE)
options(disambr_read_output_set = TRUE)
options(disambr_mess_pretty = TRUE)
options(disambr_save_as = TRUE)
options(disambr_save_set_dir = "../eva-slow-dl11-c")
ds <-
disambr_read("../data/wos-slow"
, save_sets_as = "wos-slow.rds"
, save_sets_dir = "../disambr-data") %>%
disambr_set_tekles_bornmann(file_path = "../data/tekles-bornmann-researcher-ids.txt") %>%
disambr_set_on_same_paper %>%
disambr_set_similar_initials %>%
disambr_set_similar_last_names %>%
## disambr_set_same_email %>%
## disambr_set_same_affiliation %>%
disambr_set_cite_others_paper %>%
## disambr_set_common_references %>%
## disambr_set_cite_self_citation %>%
## disambr_set_common_keywords %>%
disambr_set_same_researcher_idsoptions(disambr_get_output_set = TRUE)
options(disambr_read_output_set = TRUE)
options(disambr_mess_pretty = TRUE)
options(disambr_save_as = TRUE)
options(disambr_save_set_dir = "../eva-slow-dl11-x")
ds <-
disambr_read("../data/wos-slow"
, save_sets_as = "wos-slow.rds"
, save_sets_dir = "../disambr-data") %>%
disambr_set_tekles_bornmann(file_path = "../data/tekles-bornmann-researcher-ids.txt") %>%
disambr_set_on_same_paper %>%
disambr_set_similar_initials %>%
disambr_set_similar_last_names %>%
## disambr_set_same_email %>%
## disambr_set_same_affiliation %>%
## disambr_set_cite_others_paper %>%
disambr_set_common_references %>%
## disambr_set_cite_self_citation %>%
## disambr_set_common_keywords %>%
disambr_set_same_researcher_idsThis one is skipped as the procedure requires prior matched (strong sets) authors to identify self citations.
options(disambr_get_output_set = TRUE)
options(disambr_read_output_set = TRUE)
options(disambr_mess_pretty = TRUE)
options(disambr_save_as = TRUE)
options(disambr_save_set_dir = "../eva-slow-dl11-s")
ds <-
disambr_read("../data/wos-slow"
, save_sets_as = "wos-slow.rds"
, save_sets_dir = "../disambr-data") %>%
disambr_set_tekles_bornmann(file_path = "../data/tekles-bornmann-researcher-ids.txt") %>%
disambr_set_on_same_paper %>%
disambr_set_similar_initials %>%
disambr_set_similar_last_names %>%
## disambr_set_same_email %>%
## disambr_set_same_affiliation %>%
## disambr_set_cite_others_paper %>%
## disambr_set_common_references %>%
disambr_set_cite_self_citation %>%
## disambr_set_common_keywords %>%
disambr_set_same_researcher_idsoptions(disambr_get_output_set = TRUE)
options(disambr_read_output_set = TRUE)
options(disambr_mess_pretty = TRUE)
options(disambr_save_as = TRUE)
options(disambr_save_set_dir = "../eva-slow-dl11-k")
ds <-
disambr_read("../data/wos-slow"
, save_sets_as = "wos-slow.rds"
, save_sets_dir = "../disambr-data") %>%
disambr_set_tekles_bornmann(file_path = "../data/tekles-bornmann-researcher-ids.txt") %>%
disambr_set_on_same_paper %>%
disambr_set_similar_initials %>%
disambr_set_similar_last_names %>%
## disambr_set_same_email %>%
## disambr_set_same_affiliation %>%
## disambr_set_cite_others_paper %>%
## disambr_set_common_references %>%
## disambr_set_cite_self_citation %>%
disambr_set_common_keywords %>%
disambr_set_same_researcher_idsstats <- c("../eva-slow-dl01"
, "../eva-slow-dl11"
, "../eva-slow-dl124"
, "../eva-slow-dl125"
, "../eva-slow-dl126"
, "../eva-slow-dl11-c"
, "../eva-slow-dl11-e"
, "../eva-slow-dl11-i"
, "../eva-slow-dl11-k"
, "../eva-slow-dl11-x"
, "../eva-slow-dl11-eic2s"
, "../eva-slow-dl11-eic2sk2"
, "../eva-slow-dl11-eicsk2"
) %>%
lapply(function(d) {
print(d)
disambr_stats(sets_dir = d, save_rds = FALSE)
}) %>%
rbindlist
stats_dur <-
as.data.table(stats$dur_sets) %>%
setnames(stats$name) %>%
as.matrix %>%
barplot(legend.text = paste("procedure", 1:(nrow(.))), las = 2)
library(magrittr)
library(data.table)
source("R/disambr_sets.r")
source("R/disambr_stats.r")
options("digits" = 3)
c("../eva-slow-dl01"
, "../eva-slow-dl11"
, "../eva-slow-dl11-c"
, "../eva-slow-dl11-e"
, "../eva-slow-dl11-eic2s"
, "../eva-slow-dl11-eic2sk2"
, "../eva-slow-dl11-eicsk2"
, "../eva-slow-dl11-i"
, "../eva-slow-dl11-k"
, "../eva-slow-dl11-x"
, "../eva-slow-dl124"
, "../eva-slow-dl125"
, "../eva-slow-dl126") %>%
lapply(function(d) {
print(d)
disambr_stats(sets_dir = d, save_rds = FALSE)
}) %>%
rbindlist %>%
as.matrix| name | true_positives | false_positives | false_negatives | true_negatives | pw_presision pw_recall pw_f1 pw_accuracy dur_mins | w_presision | pw_recall | pw_f1 | pw_accuracy dur_mins |
|---|---|---|---|---|---|---|---|---|---|
| eva-slow-dl01” | 2976 | 38 | 2461 | -5473 | 0.987 | 0.547 | 0.704 | -1248 | 16.3 |
| eva-slow-dl11” | 2977 | 89 | 2460 | -5524 | 0.971 | 0.548 | 0.7 | -1274 | 35 |
| eva-slow-dl11-c” | 83 | 4 | 5354 | NA | 0.954 | 0.0153 | 0.0301 | NA | 0.73 |
| eva-slow-dl11-e” | 1294 | 0 | 4143 | 1667705 | 1 | 0.238 | 0.384 | 0.998 | 0.741 |
| eva-slow-dl11-eic2s” | 2977 | 89 | 2460 | -5524 | 0.971 | 0.548 | 0.7 | -1274 | 33.9 |
| eva-slow-dl11-eic2sk2” | 2981 | 92 | 2456 | -5527 | 0.97 | 0.548 | 0.701 | -1273 | 34 |
| eva-slow-dl11-eicsk2” | 2981 | 92 | 2456 | -5527 | 0.97 | 0.548 | 0.701 | -1273 | 33.9 |
| eva-slow-dl11-i” | 2362 | 77 | 3075 | 5940768 | 0.968 | 0.434 | 0.6 | 0.999 | 0.757 |
| eva-slow-dl11-k” | 503 | 6 | 4934 | NA | 0.988 | 0.0925 | 0.169 | NA | 3.98 |
| eva-slow-dl11-x” | 233 | 0 | 5204 | NA | 1 | 0.0429 | 0.0822 | NA | 14.4 |
| eva-slow-dl124” | 2977 | 102 | 2460 | -5537 | 0.967 | 0.548 | 0.699 | -1280 | 37.6 |
| eva-slow-dl125” | 2977 | 89 | 2460 | -5524 | 0.971 | 0.548 | 0.7 | -1274 | 34.2 |
| eva-slow-dl126” | 2977 | 89 | 2460 | -5524 | 0.971 | 0.548 | 0.7 | -1274 | 33.9 |
[2022-04-15 Fri]
library(magrittr)
library(data.table)
source("R/disambr_sets.r")
source("R/disambr_stats.r")
options("digits" = 3)
"/mnt/raid5/data/disambr-data" |>
file.path(c("eva-slow-dl01"
, "eva-slow-dl11"
, "eva-slow-dl11-c"
, "eva-slow-dl11-e"
, "eva-slow-dl11-eic2s"
, "eva-slow-dl11-eic2sk2"
, "eva-slow-dl11-eicsk2"
, "eva-slow-dl11-i"
, "eva-slow-dl11-k"
, "eva-slow-dl11-x"
, "eva-slow-dl124"
, "eva-slow-dl125"
, "eva-slow-dl126")) |>
lapply(function(d) {
print(d)
disambr_stats(sets_dir = d, save_rds = FALSE)
}) |>
data.table::rbindlist() |>
as.matrix()setwd("S:/disambr")
options(disambr_get_output_set = TRUE)
options(disambr_mess_pretty = TRUE)
options(disambr_save_as = TRUE)
options(disambr_save_set_dir = "eva-1")
ts_eva_full <-
disambr_read("wos-slow-export-subset"
, save_sets_as = "wos-slow-export-subset-2.rds"
, save_sets_dir = "eva-1") %>%
disambr_set_tekles_bornmann %>% #59149 vs_1962896
disambr_set_on_same_paper %>%
disambr_set_similar_initials %>%
disambr_set_similar_last_names %>%
disambr_set_same_email %>%
disambr_set_same_affiliation %>%
disambr_set_cite_others_paper %>%
disambr_set_common_references %>%
disambr_set_cite_self_citation %>%
disambr_set_common_keywords %>%
disambr_set_same_researcher_ids
ts_eva_full_try <-
disambr_read("../data/wos-slow-export-subset"
, save_sets_as = "wos-slow-export-subset-2.rds"
, save_sets_dir = "../data")van den Akker, O. R., Epskamp, Sacha, & Vlasov, S. A. (2020). The AEV Algorithm—Author name disambiguation for large Web of Science datasets.
Tekles, A., & Bornmann, L. (2019). Author name disambiguation of bibliometric data: A comparison of several unsupervised approaches. ArXiv:1904.12746 [Cs]. http://arxiv.org/abs/1904.12746