Allow use of truncated Dictionary for coherence measures

[macks22]

Description

I used the make_wikicorpus.py script to parse the latest English Wikipedia. This script filters the extremes in the Dictionary using wiki.dictionary.filter_extremes(no_below=20, no_above=0.1, keep_n=100000). I then trained an LDA model on the corpus (see code below). Afterwards, I tried to compute the c_v coherence on this corpus using the Dictionary produced by the make_wikicorpus.py script. This failed because the Dictionary does not contain all terms in the Wikipedia corpus. It should not fail, because the terms it is missing are not terms in the top_ids set from the actual topics, and are therefore not relevant to the coherence computation.

Coherence calculation should be possible with the same dictionary used to train the model, even if it does not contain all tokens in the texts.

Steps/Code/Corpus to Reproduce

import gensim
id2word = gensim.corpora.Dictionary.load_from_text('wiki_en_wordids.txt.bz2')
mm = gensim.corpora.MmCorpus('wiki_en_tfidf.mm')
lda = gensim.models.ldamulticore.LdaMulticore(corpus=mm, id2word=id2word, num_topics=100, chunksize=10000, passes=1)
topics = [[t[0] for t in lda.get_topic_terms(i, 20)] for i in range(100)]
texts = gensim.corpora.WikiCorpus('enwiki-latest-pages-articles.xml.bz2', dictionary=id2word).get_texts()
cm = gensim.models.coherencemodel.CoherenceModel(topics=topic_words, texts=texts, dictionary=id2word, coherence='c_v', topn=20, window_size=110)

Expected Results

Coherence measure computed from wikipedia corpus using the Dictionary built by the make_wikicorpus.py script.

Actual Results

cm.get_coherence()
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-78-a170469ad280> in <module>()
----> 1 cm.get_coherence()

/Users/vru959/anaconda2/lib/python2.7/site-packages/gensim/models/coherencemodel.pyc in get_coherence(self)
    207                 self.window_size = sliding_windows_dict[self.coherence]
    208             per_topic_postings, num_windows = measure.prob(texts=self.texts, segmented_topics=segmented_topics,
--> 209                                                            dictionary=self.dictionary, window_size=self.window_size)
    210             if self.coherence == 'c_v':
    211                 confirmed_measures = measure.conf(self.topics, segmented_topics, per_topic_postings, 'nlr', 1, num_windows)

/Users/vru959/anaconda2/lib/python2.7/site-packages/gensim/topic_coherence/probability_estimation.pyc in p_boolean_sliding_window(texts, segmented_topics, dictionary, window_size)
     96         it = iter(document)
     97         window = tuple(islice(it, window_size))
---> 98         window_id, per_topic_postings = add_topic_posting(top_ids, window, per_topic_postings, window_id, token2id_dict)
     99         for elem in it:
    100             window = window[1:] + (elem,)

/Users/vru959/anaconda2/lib/python2.7/site-packages/gensim/topic_coherence/probability_estimation.pyc in add_topic_posting(top_ids, window, per_topic_postings, window_id, token2id_dict)
     84     def add_topic_posting(top_ids, window, per_topic_postings, window_id, token2id_dict):
     85         for word in window:
---> 86             word_id = token2id_dict[word]
     87             if word_id in top_ids:
     88                 if word_id in per_topic_postings:

KeyError: 'of'

Versions

Darwin-16.5.0-x86_64-i386-64bit
('Python', '2.7.13 |Anaconda custom (x86_64)| (default, Dec 20 2016, 23:05:08) \n[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]')
('NumPy', '1.12.1')
('SciPy', '0.19.0')
('gensim', '2.1.0')
('FAST_VERSION', 1)

[menshikh-iv]

Hello @macks22, thank you for your report. I need some time to reproduce the bug.

It seems to me that the problem in the texts variable (i think it contains tokens that not exists in the dictionary, because of dictionary parameter in gensim.corpora.WikiCorpus used only when you called method __iter__ (not get_texts()) for returning doc2bow representation)

Could you please filtering texts manually (remove tokens that dictionary does not contain) and try again?

[macks22]

I'm fairly certain that is the issue; I've modified the topic_coherence.probability_estimation.p_boolean_sliding_window method to only try to convert words to their ids if they are actually in the set accumulated from the topics. So far, this appears to have fixed the issue, though I can't be sure until the get_coherence method finishes. It is slow on the full wikipedia corpus of texts.

[menshikh-iv]

@macks22 coherence="c_v" so slow, you can try coherence="u_mass" (it works significantly faster).

[macks22]

@menshikh-iv thank you for the suggestion, but unfortunately the code path for coherence=u_mass does not use p_boolean_sliding_window, which I suspect is why it is much faster.

[tmylk]

c_v is the best/recommended coherence so we should make it as fast as possible