MemoryError: Error allocating memory for feature

[schlichtanders]

Dear spaCy Team,

I tried to further train the default spacy entity recognizer, however I immediately run into a Memory Error.

import spacy
nlp = spacy.load("de")
nlp.entity.model.resume_training()

throws

MemoryError                               Traceback (most recent call last)
<ipython-input-29-6298edf1c0b7> in <module>()
----> 1 nlp.entity.model.resume_training()

C:\tools\Anaconda3\lib\site-packages\thinc\linear\avgtron.pyx in thinc.linear.avgtron.AveragedPerceptron.resume_training (thinc/linear/avgtron.cpp:4177)()

MemoryError: Error allocating memory for feature: 13665621275175813120

Have you ever encountered such things? Any suggestions how to proceed?

My Environment

• Operating System: Windows 10
• Python Version Used: 3.5
• spaCy Version Used: 1.6.0
• Environment Information: 32GB RAM

[honnibal]

Hi,

Labelling this bug, because no matter the root cause that shouldn't be the expected behaviour.

Do you have the data installed?

Matt

[schlichtanders]

thank you very much for your fast response

I already had run long ago python -m spacy.de.download all and tried to redownload it using --force, however this behaviour is currently broken in spacy 1.6 (on master it is already fixed), hence I might have an older download from spacy 1.2

I executed python -m spacy.en.download all and downloaded all english data.

Still the same error. (also with the english equivalent code)

[schlichtanders]

I tried it again with the new thinc master adapted for #799 however this memory allocation error is unfortunately still there.

[akalino]

I am encountering similar issues now - upgrading to thinc-6.4.0 solved my memory issues, however now I am unable to save the models I update. Traceback below:

  File "/mnt/alex/NER/train_general_model.py", line 277, in <module>

    train_client_brand_model(output_path, td)

  File "/mnt/alex/NER/train_general_model.py", line 211, in train_client_brand_model

    ner.model.dump(str(model_dir / 'model'))

  File "thinc/linear/avgtron.pyx", line 81, in thinc.linear.avgtron.AveragedPerceptron.dump 
(thinc/linear/avgtron.cpp:2852)

AttributeError: 'preshed.maps.PreshMap' object has no attribute 'capacity' ```

[akalino]

Actually, looks like an upgrade of preshed to 1.0.0 solves the issue I was seeing.

[honnibal]

This should be fixed now -- see #910

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.