stop_words assigned but not used?

[ExplodingCabbage]

Maybe I'm being dense, but when I search the entire repo case-insensitively for stop_words, it looks like you're defining a list of stop words but never using it. Every match in the Sublime search below is an assignment; all you seem to do is define STOP_WORDS constants in language_data.py files and then assign those constants to the stop_words class property of a Language's Defaults, without ever then reading from it:

Searching 535 files for "stop_words"

/home/mark/spaCy/spacy/language.py:
  153      tagger_features = Tagger.feature_templates # TODO -- fix this
  154  
  155:     stop_words = set()
  156  
  157      lex_attr_getters = {

/home/mark/spaCy/spacy/de/__init__.py:
   24          tag_map = dict(language_data.TAG_MAP)
   25  
   26:         stop_words = set(language_data.STOP_WORDS)
   27  
   28  

/home/mark/spaCy/spacy/de/language_data.py:
    4  
    5  
    6: STOP_WORDS = set()
    7  
    8  

/home/mark/spaCy/spacy/en/__init__.py:
   29          tag_map = dict(language_data.TAG_MAP)
   30  
   31:         stop_words = set(language_data.STOP_WORDS)
   32  

/home/mark/spaCy/spacy/en/language_data.py:
    4  
    5  # improved list from Stone, Denis, Kwantes (2010)
    6: STOP_WORDS = set("""
    7  a about above across after afterwards again against all almost alone 
    8  along already also although always am among amongst amoungst amount 

/home/mark/spaCy/spacy/es/__init__.py:
   24          tag_map = dict(language_data.TAG_MAP)
   25  
   26:         stop_words = set(language_data.STOP_WORDS)
   27  

/home/mark/spaCy/spacy/es/language_data.py:
    4  
    5  
    6: STOP_WORDS = set()
    7  
    8  

/home/mark/spaCy/spacy/fr/__init__.py:
   24          tag_map = dict(language_data.TAG_MAP)
   25  
   26:         stop_words = set(language_data.STOP_WORDS)
   27  
   28  

/home/mark/spaCy/spacy/fr/language_data.py:
    4  
    5  
    6: STOP_WORDS = set()
    7  
    8  

/home/mark/spaCy/spacy/it/__init__.py:
   24          tag_map = dict(language_data.TAG_MAP)
   25  
   26:         stop_words = set(language_data.STOP_WORDS)
   27  
   28  

/home/mark/spaCy/spacy/it/language_data.py:
    4  
    5  
    6: STOP_WORDS = set()
    7  
    8  

/home/mark/spaCy/spacy/pt/__init__.py:
   24          tag_map = dict(language_data.TAG_MAP)
   25  
   26:         stop_words = set(language_data.STOP_WORDS)
   27  
   28  

/home/mark/spaCy/spacy/pt/language_data.py:
    4  
    5  
    6: STOP_WORDS = set()
    7  
    8  

19 matches across 13 files

Does this list still have a purpose, or should it be culled? I thought I'd flag this up before any of you dutifully hunt down stop word lists for the new languages you're adding!

Apologies if there's some reason for this to exist that I'm missing.

[ExplodingCabbage]

Aside: the English STOP_WORDS list contains some surprising entries like "computer", "fire", and "mill" that it seems bizarre and arbitrary to treat as stop words. I've tracked down the source of this to http://onlinelibrary.wiley.com/store/10.1111/j.1756-8765.2010.01108.x/asset/supinfo/TOPS_1108_sm_supmat.pdf?v=1&s=715bd019aab0c2df0c269b487209c1342143a0a6, and it seems that this was indeed the stop word list used in http://onlinelibrary.wiley.com/doi/10.1111/j.1756-8765.2010.01108.x/full; regardless, it's bizarre and if this list is sticking around perhaps the presence of these seemingly inappropriate entries should be addressed.

[honnibal]

Thanks.

What's supposed to happen is the IS_STOP attribute in the Language class should be mapping to a function that looks up the word in the stop list. I see that this got broken somewhere.

Agree about the English stop list.

[ines]

Re English stopwords: I'm currently in the process of reorganising the language data. Just posted an update here: #649

[honnibal]

Put a band-aid on this for now. A more satisfying fix will come alongside the data reorganisation.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.