Readd(ed) corrects to itself

[cfi-gb]

The dictionary.txt currently contains the following:

readd->readd, re-add, read,
readded->readded, read,

which corrects to itself:

$ cat test.txt
readd
readded

$ codespell test.txt
test.txt:1: readd  ==> readd, re-add, read
test.txt:2: readded  ==> readded, read

If the correct spelling of "readded" would be "re-added" then there is also the following found a few lines below:

readed->read, readd, readded,

[larsoner]

Related discussion in #410, it's not clear how to deal with this case yet

[peternewman]

As I mentioned in #410, I think something needs to be done. Personally if things like wit->wit, with, are being included, these ones need to be excluded by default. Only some sort of magic context based AI can decide if you wanted wit or with in a particular sentence.

[larsoner]

Maybe what we should do is have support for multiple dictionary files, which will use the union of all entries (probably just taking the entries from ones listed later if there are overlaps). Then the default can be dictionary.txt and dictionary_rare.txt or so, where the latter holds all of these self-corrections that are likely mistakes but could be correct.

[peternewman]

Maybe what we should do is have support for multiple dictionary files, which will use the union of all entries (probably just taking the entries from ones listed later if there are overlaps). Then the default can be dictionary.txt and dictionary_rare.txt or so, where the latter holds all of these self-corrections that are likely mistakes but could be correct.

Sounds good to me, multiple files should give some much better options for validation during CI. The only real question is how they get merged or overwritten regards suggestions etc.

[larsoner]

The only real question is how they get merged or overwritten regards suggestions etc.

I would try to keep it simple and let the later entries take precendence (overwriting any exitsing entries). From a user perspective it's clear what happens, and if we need some other behavior let's add it later and not over-engineer (assume YAGNI for now).

@peternewman do you want to try to implement this?

[peternewman]

Yes, I guess I should given I keep banging on about it! I've got some other stuff I need to sort first, but I'll try and take a look at some point soon. In the meantime, I guess people can start preparing additional dictionaries...

[peternewman]

I think I've unfortunately found a scenario where suggestions may need merging (or maybe I just need a working camel case filter).

If you find dictionary entries matching [^,] ., some are just adding spaces, some are also fixing typos, some offer a space and a non-space suggestion. It might make sense to split the space only ones out, but then some people would want both options merged.

[peternewman]

Okay, so after some hacking around, I've got a test of the dictionary. There are currently 112 valid words (according to aspell's English dictionary) listed as typos in the dictionary file, so anyone not using interactive mode will potentially be getting these offered up as typos:
https://travis-ci.org/codespell-project/codespell/jobs/539006897#L515

I've not yet checked how many of these offer themselves as corrections, as discussed in #410 .

[peternewman]

Another rare word, but a valid word nonetheless chack->check