Comparing first name using the nicknames package

[Ahosseinzadeh723]

I'm currently facing a challenge in linking individual records due to the use of nicknames (in member_first_name). I found a Python package called nicknames, which includes a database of over a thousand nicknames. I'm considering integrating this package into my data linkage process to improve the accuracy of individual matching.

Could anyone share their experiences or suggest the best practices for incorporating such a package into a data linkage settings? Specifically, I am looking for advice on how to effectively utilize this nickname database in my settings.

settings = {
    "link_type": "dedupe_only",
    "comparisons": [
        ctl.name_comparison("member_first_name", term_frequency_adjustments=True),
        cl.exact_match("member_middle_initial"),
        ctl.name_comparison("member_last_name", term_frequency_adjustments=True),
        cl.levenshtein_at_thresholds("member_date_of_birth_vc", distance_threshold_or_thresholds=[1, 2], term_frequency_adjustments=True)
    ],
    "blocking_rules_to_generate_predictions": [
        block_on(["member_last_name", "member_date_of_birth_vc"]),
        block_on(["member_first_name", "member_date_of_birth_vc"]),
        block_on(["member_first_name", "member_last_name"])
    ],
    "retain_matching_columns": False,
    "retain_intermediate_calculation_columns": False,
    "unique_id_column_name": "enrollment_id",
    "max_iterations": 30,
    "em_convergence": 0.0001
}

For example, "Bill" is short for "William". If record 1: member_first_name is Bill and record 2: member_first_name is William.

pip install nicknames
from nicknames import NickNamer
nn = NickNamer()
nicks = nn.nicknames_of("william")

which results: {'bela', 'bell', 'bill', 'billy', 'wil', 'will', 'willie', 'willy'}
Using this package I expect the algorithm to consider record 1 and record 2 member_first_name as exact match.
Any insights or examples of similar implementations would be greatly appreciated!

[Ahosseinzadeh723]

Alternatively, the list of names available at nicknames can be used independently.

[RobinL]

In your input data, you need to derive an array column containing the name and each nickname. Then you need a comparison level for exact match on first name, and then a second for an array intersection on the array column of nicknames. Sorry on my phone so can't easily suggest code, but hopefully that's enough of a pointer that you can look up the rest in the docs

[Ahosseinzadeh723]

Thank you so much Robin! Sounds very interesting. Based on what I have, I will have a schema of my base data for linkage and also the nicknames data. I will join the nicknames to my data and store them in a new column as an array. What I am struggling with is how to construct a comparison with different levels using two fields. I would appreciate your help with it!

[RobinL]

There's an example here: https://moj-analytical-services.github.io/splink/topic_guides/comparisons/customising_comparisons.html?h=cust#method-3-comparisonlevels. You will want cll.array_intersect_level on the nickname rather than the exact match on dmeta

[RobinL]

Thinking about it i think you might want something like this:

   {
            "sql_condition": "list_contains(nicknames_l, first_name_r) OR list_contains(nicknames_r, first_name_l)",
            "label_for_charts": "Nickname match",
        },

A pure array intersection compares elements that exist in both lists. However, in the context of nicknames, there is often an asymmetry. Two different first names can share a nickname, but you wouldn't want to count that as a match. So you need to check that either the left name is a nickname of the right name, or the right name is a nickname of the left