Wrapper for Varembed Models

.travis.yml

@@ -21,5 +21,6 @@ install:
   - pip install annoy
   - pip install testfixtures
   - pip install unittest2
+  - pip install Morfessor==2.0.2a4
   - python setup.py install
 script: python setup.py test

docs/notebooks/Varembed.ipynb

@@ -0,0 +1,163 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# VarEmbed Tutorial\n",
+    "\n",
+    "Varembed is a word embedding model incorporating morphological information, capturing shared sub-word features. Unlike previous work that constructs word embeddings directly from morphemes, varembed combines morphological and distributional information in a unified probabilistic framework. Varembed thus yields improvements on intrinsic word similarity evaluations. Check out the original paper, [arXiv:1608.01056](https://arxiv.org/abs/1608.01056) accepted in [EMNLP 2016](http://www.emnlp2016.net/accepted-papers.html).\n",
+    "\n",
+    "Varembed is now integrated into [Gensim](http://radimrehurek.com/gensim/) providing ability to load already trained varembed models into gensim with additional functionalities over word vectors already present in gensim.\n",
+    "\n",
+    "# This Tutorial\n",
+    "\n",
+    "In this tutorial you will learn how to train, load and evaluate varembed model on your data.\n",
+    "\n",
+    "# Train Model\n",
+    "\n",
+    "The authors provide their code to train a varembed model. Checkout the repository [MorphologicalPriorsForWordEmbeddings](https://github.com/rguthrie3/MorphologicalPriorsForWordEmbeddings) for to train a varembed model. You'll need to use that code if you want to train a model. \n",
+    "\n",
+    "# Load Varembed Model\n",
+    "\n",
+    "Now that you have an already trained varembed model, you can easily load the varembed word vectors directly into Gensim. <br>\n",
+    "For that, you need to provide the path to the word vectors pickle file generated after you train the model and run the script to [package varembed embeddings](https://github.com/rguthrie3/MorphologicalPriorsForWordEmbeddings/blob/master/package_embeddings.py) provided in the [varembed source code repository](https://github.com/rguthrie3/MorphologicalPriorsForWordEmbeddings).\n",
+    "\n",
+    "We'll use a varembed model trained on [Lee Corpus](https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/test/test_data/lee.cor) as the vocabulary, which is already available in gensim.\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "from gensim.models.wrappers import varembed\n",
+    "\n",
+    "vector_file = '../../gensim/test/test_data/varembed_leecorpus_vectors.pkl'\n",
+    "model = varembed.VarEmbed.load_varembed_format(vectors=vector_file)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This loads a varembed model into Gensim. Also if you want to load with morphemes added into the varembed vectors, you just need to also provide the path to the trained morfessor model binary as an argument. This works as an optional parameter, if not provided, it would just load the varembed vectors without morphemes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "morfessor_file = '../../gensim/test/test_data/varembed_leecorpus_morfessor.bin'\n",
+    "model_with_morphemes = varembed.VarEmbed.load_varembed_format(vectors=vector_file, morfessor_model=morfessor_file)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This helps load trained varembed models into Gensim. Now you can use this for any of the Keyed Vector functionalities, like 'most_similar', 'similarity' and so on, already provided in gensim. \n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[(u'launch', 0.2694973647594452),\n",
+       " (u'again', 0.2564533054828644),\n",
+       " (u'gun', 0.2521245777606964),\n",
+       " (u'response', 0.24817466735839844),\n",
+       " (u'swimming', 0.23348823189735413),\n",
+       " (u'bombings', 0.23146548867225647),\n",
+       " (u'transformed', 0.2289058119058609),\n",
+       " (u'used', 0.2224646955728531),\n",
+       " (u'weeks,', 0.21905183792114258),\n",
+       " (u'scheduled', 0.2170265018939972)]"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "model.most_similar('government')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "0.022313305789051038"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "model.similarity('peace', 'grim')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Conclusion\n",
+    "In this tutorial, we learnt how to load already trained varembed models vectors into gensim and easily use and evaluate it. That's it!\n",
+    "\n",
+    "# Resources\n",
+    "\n",
+    "* [Varembed Source Code](https://github.com/rguthrie3/MorphologicalPriorsForWordEmbeddings)\n",
+    "* [Gensim](http://radimrehurek.com/gensim/)\n",
+    "* [Lee Corpus](https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/test/test_data/lee.cor)\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "anaconda-cloud": {},
+  "kernelspec": {
+   "display_name": "Python [default]",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}

docs/src/apiref.rst

@@ -45,6 +45,7 @@ Modules:
     models/wrappers/dtmmodel
     models/wrappers/ldavowpalwabbit.rst
     models/wrappers/wordrank
+    models/wrappers/varembed
     similarities/docsim
     similarities/index
     topic_coherence/aggregation

docs/src/models/wrappers/varembed.rst

@@ -0,0 +1,9 @@
+:mod:`models.wrappers.varembed` -- VarEmbed Word Embeddings
+================================================================================================
+
+.. automodule:: gensim.models.wrappers.varembed
+    :synopsis: VarEmbed Word Embeddings
+    :members:
+    :inherited-members:
+    :undoc-members:
+    :show-inheritance:

gensim/models/wrappers/__init__.py

@@ -7,3 +7,4 @@
 from .ldavowpalwabbit import LdaVowpalWabbit
 from .fasttext import FastText
 from .wordrank import Wordrank
+from .varembed import VarEmbed

gensim/models/wrappers/varembed.py

@@ -0,0 +1,113 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# Copyright (C) 2017 Anmol Gulati <[email protected]>
+# Copyright (C) 2017 Radim Rehurek <[email protected]>
+
+"""
+Python wrapper around word representation learning from Varembed models, a library for efficient learning of word representations
+and sentence classification [1].
+
+This module allows ability to obtain word vectors for out-of-vocabulary words, for the Varembed model[2].
+
+The wrapped model can NOT be updated with new documents for online training -- use gensim's `Word2Vec` for that.
+
+.. [1] https://github.com/rguthrie3/MorphologicalPriorsForWordEmbeddings
+
+.. [2] http://arxiv.org/pdf/1608.01056.pdf
+"""
+
+import logging
+import sys
+
+import numpy as np
+
+from gensim.models.keyedvectors import KeyedVectors
+
+# utility fnc for pickling, common scipy operations etc
+from gensim import utils
+from gensim.models.word2vec import Vocab
+
+logger = logging.getLogger(__name__)
+
+
+class VarEmbed(KeyedVectors):
+    """
+    Class for word vectors using Varembed models. Contains methods to load a varembed model and implements
+    functionality like `most_similar`, `similarity` by extracting vectors into numpy matrix.
+    Refer to [Varembed]https://github.com/rguthrie3/MorphologicalPriorsForWordEmbeddings for
+    implementation of Varembed models.
+    """
+
+    def __init__(self):
+        self.vector_size = 0
+        self.vocab_size = 0
+
+    @classmethod
+    def load_varembed_format(cls, vectors, morfessor_model=None):
+        """
+        Load the word vectors into matrix from the varembed output vector files.
+        Using morphemes requires Python 2.7 version or above.
+
+        'vectors' is the pickle file containing the word vectors.
+        'morfessor_model' is the path to the trained morfessor model.
+        'use_morphemes' False(default) use of morpheme embeddings in output.
+        """
+        result = cls()
+        if vectors is None:
+            raise Exception("Please provide vectors binary to load varembed model")
+        D = utils.unpickle(vectors)
+        word_to_ix = D['word_to_ix']
+        morpho_to_ix = D['morpho_to_ix']
+        word_embeddings = D['word_embeddings']
+        morpho_embeddings = D['morpheme_embeddings']
+        result.load_word_embeddings(word_embeddings, word_to_ix)
+        if morfessor_model:
+            if sys.version_info >= (2, 7):  #Morfessor is only supported for Python 2.7 and above.
+                try:
+                    import morfessor
+                    morfessor_model = morfessor.MorfessorIO().read_binary_model_file(morfessor_model)
+                    result.add_morphemes_to_embeddings(morfessor_model, morpho_embeddings, morpho_to_ix)
+                except ImportError:
+                    # Morfessor Package not found.
+                    logger.error('Could not import morfessor. Not using morpheme embeddings')
+                    raise ImportError('Could not import morfessor.')
+            else:
+                # Raise exception in Python 2.6 or earlier.
+                raise Exception('Using Morphemes requires Python 2.7 and above. Morfessor is not supported in python 2.6')
+
+        logger.info('Loaded varembed model vectors from %s', vectors)
+        return result
+
+    def load_word_embeddings(self, word_embeddings, word_to_ix):
+        """ Loads the word embeddings """
+        logger.info("Loading the vocabulary")
+        self.vocab = {}
+        self.index2word = []
+        counts = {}
+        for word in word_to_ix:
+            counts[word] = counts.get(word, 0) + 1
+        self.vocab_size = len(counts)
+        self.vector_size = word_embeddings.shape[1]
+        self.syn0 = np.zeros((self.vocab_size, self.vector_size))
+        self.index2word = [None]*self.vocab_size
+        logger.info("Corpus has %i words", len(self.vocab))
+        for word_id, word in enumerate(counts):
+            self.vocab[word] = Vocab(index=word_id, count=counts[word])
+            self.syn0[word_id] = word_embeddings[word_to_ix[word]]
+            self.index2word[word_id] = word
+        assert((len(self.vocab), self.vector_size) == self.syn0.shape)
+        logger.info("Loaded matrix of %d size and %d dimensions", self.vocab_size, self.vector_size)
+
+
+    def add_morphemes_to_embeddings(self, morfessor_model, morpho_embeddings, morpho_to_ix):
+        """ Method to include morpheme embeddings into varembed vectors
+            Allowed only in Python versions 2.7 and above.
+        """
+        for word in self.vocab:
+            morpheme_embedding = np.array(
+                    [morpho_embeddings[morpho_to_ix.get(m, -1)] for m in
+                     morfessor_model.viterbi_segment(word)[0]]).sum(axis=0)
+            self.syn0[self.vocab[word].index] += morpheme_embedding
+        logger.info("Added morphemes to word vectors")
+