Add rare typo hep->heap, help,

[skangas]

Found in Emacs.

[henryiii]

Ahh! HEP stands for High Energy Physics, and is extremely common, such as in https://scikit-hep.org, https://iris-hep.org, etc. The Scientific-Python developer guidelines were originally the Scikit-HEP developer pages, etc. Just noticed this in yesterday's release.

[DimitriPapadopoulos]

Actually, hep is also an entry in the OED.

I'm not sure how to handle this. Perhaps the rare dictionary should not be selected by default.

[henryiii]

How about removing this, and just leaving all the other ones (like heping, which is not valid for either HEP or "hep" the word)?

[DimitriPapadopoulos]

It could lead to false negatives in other contexts.

[henryiii]

Many correctly spelled words could be a misspelling for another word, but those aren't listed. Forgetting the "n" in "many" gives "may", but "may" isn't listed as a misspelled word, for example. I thought the idea was to just list ones that are not valid words. "hep" is a major branch of physics that has had a lot of impact in programming (at least in C++ and Python), and an English word as well. HEP is used in 40+ repos across the Scikit-HEP project and in other HEP related projects, as well as in lots of Python packaging repos, etc; all told I'm likely to have to ignore this now in 60+ repos. That's why I was wondering if it could have just been left out, as older versions didn't have this.

(I'd also be fine if "hep" was invalid but "HEP" was valid, by the way, which doesn't help the English usage but does fix the HEP case. At least some of the time - not scikit-hep.org links or links to scikit-hep repos, but at least the normal usages. Nevermind, it ignores scikit-hep, only triggering on standalone hep, that helps, especially in the lower-case case.)

[jpivarski]

I'd like to emphasize how common "HEP" is as a term. From my physics background, I can say that it's one of the major subdivisions of the field: 4 of the 13 physics categories in the preprint archive (https://arxiv.org) are "High Energy Physics," but the project started at Los Alamos, so that's a bias.

Are there some quantitative ways to say how common this is? Well, in Google Books, "HEP" closely tracks its meaning, "High Energy Physics", and although the peak use of that term was in the late 20th century, it's on the same scale as phrases like "quantum mechanics." (The term is giving way to "particle physics," since that's more inclusive of experiments that don't use colliders, not literally "high energy.")

In Google searches, "HEP" is much more common than "quantum mechanics" (a lot to type!), but within a factor of 3 of just the word "quantum" by itself.

Maybe more important for typos, specifically "hep" when "help" was intended, is that "hep" is often included in filenames in GitHub. A typo in a filename is much less likely (to survive scrutiny) than a typo in the text itself. In GitHub code searches,

• 0.754 million matches to "hep"
• 17.7 million matches to "help"

which is 4%. There are also a lot of repositories with "hep" in their name (1% compared to "help"). For a comparison between a subdivision of a scientific discipline versus a common English word, that's a lot!

I bet there are ways of determining how often people type "hep" when they mean "help" (spellchecker logs?). How rare is this "rare typo"? What's the ROC curve threshold for our tolerance of type 1 and type 2 errors on this? Because it would be quite annoying to have to patch all HEP-related software with a rule that counters this rule.

[DimitriPapadopoulos]

Note that I am not the maintainer. @larsoner Do not hesitate to jump in.

I am fully aware that HEP is a major field of physics. Yet, for someone outside of physics it's yet another acronym and I wouldn't be surprised if any (short) typo was an acronym. The rule of thumb I've been using so far is to:

• rely on curated (online) dictionaries such as the OED, Cambridge, Collins, Oxford Learner's, Macquarie, Merriam-Webster,
• use SCOWL (And Friends), the "reference" open source dictionary, as the definitive reference.

That's perhaps an excessive simplification, but words that are not part of these dictionaries are not taken into account. In this specific case:

• Unless I am mistaken, except for the OED, hep is not a word.
• I would argue that you need to add HEP to SCOWL (And Friends), the open source reference dictionary.
• Nevertheless, since hep is in the OED, it has been added to the rare dictionary.
• Again, I think the best solution would be to remove the rare dictionary from the default dictionaries.

Any way, I am not hell-bent to refuse this change - and again I am not the maintainer. I just don't feel like taking the decision. Why not submit a PR? Someone else, perhaps @larsoner, will have a look.

[larsoner]

I'm okay with reverting the addition to rare. Seems like it'll be easier for people to add it to their own custom dict in the rare cases it's helpful.

At some point maybe we don't want rare in the defaults but that's a bigger change

@jpivarski want to open a PR? I could look merge and cut a quick release

[peternewman]

Random slightly drive-by thoughts:

• Do we need a science dictionary, to mirror the code one ( https://github.com/codespell-project/codespell/blob/main/codespell_lib/data/dictionary_code.txt )? For science-specific conflicting typos like hep, which you could then skip the whole dictionary in a science context (I feel like there were a few others back in the day)
• Should non-core dictionaries (rare etc) never auto-correct, even if they've only got one entry?
• Back in the day I suggested splitting rare in half based on some criteria (e.g. which SCOWL list they were in) so you've got rare and really rare, where the latter may be enabled by default, but not the former.

I appreciate it's a balance, but personally my worry would be if you disable that dictionary by default, most people will just run with the defaults and don't get the benefit, whereas it's pretty trivial to remove the dictionary if you find you're getting too many false positives (does it need better signposting in the log messages though)?

FWIW on my domain specific typos, I've just added them to an ignore list (I think thead and wronly have subsequently been moved to code given they impact anyone there):
https://github.com/OpenLightingProject/ola/blob/master/.codespellignorewords

[hugovk]

• Unless I am mistaken, except for the OED, hep is not a word.

• rely on curated (online) dictionaries such as the OED, Cambridge, Collins, Oxford Learner's, Macquarie, Merriam-Webster,

It's also in Collins and Merriam-Webster. (I don't have access to Macquarie.)

[DimitriPapadopoulos]

Note that SCOWL (and friends) include a scale of rarity for words, extracted from Google Ngram viewer:

By the way, this screen capture shows that hep is in the en_GB-large dictionary.