Merge branch 'master' into gal_dropout

CMakeLists.txt

@@ -1,7 +1,6 @@
 project(dynet)
 cmake_minimum_required(VERSION 2.8 FATAL_ERROR)
 
-set(CMAKE_MACOSX_RPATH 1)
 set(CMAKE_MODULE_PATH ${PROJECT_SOURCE_DIR}/cmake)
 
 # DYNET uses Eigen which exploits modern CPU architectures. To get the
@@ -12,6 +11,11 @@ set(CMAKE_MODULE_PATH ${PROJECT_SOURCE_DIR}/cmake)
 #      flags (the compiler does not always make the best configuration
 #      decisions without help)
 
+# NOTE: This seems to be causing problems with linking before using
+#       make install. It is allegedly preferred, but probably doesn't
+#       suit our model of not installing the library most of the time.
+set(CMAKE_MACOSX_RPATH 0)
+
 function(find_mkl)
   set(MKL_ARCH intel64)
   find_path(MKL_INCLUDE_DIR mkl.h

README.md

@@ -7,3 +7,5 @@ Read the [documentation](http://dynet.readthedocs.io/en/latest/) to get started,
 
 * **Latest Code:** Can be found on the github page master branch.
 * **Latest Release:** [v1.0-rc1](https://github.com/clab/dynet/releases/tag/v1.0-rc1)
+
+[![Build Status](https://travis-ci.org/clab/dynet.svg?branch=master)](https://travis-ci.org/clab/dynet)

doc/source/commandline.rst

@@ -1,5 +1,7 @@
-DyNet Command Line Options
-==========================
+.. _command-line-options:
+
+Command Line Options
+====================
 
 All programs using DyNet have a few command line options. These must be
 specified at the very beginning of the command line, before other
@@ -14,8 +16,11 @@ options.
    variables ``--dynet-mem FOR,BACK,PARAM``. This is useful if, for
    example, you are performing testing and don't need to allocate any
    memory for backward calculation.
--  ``--dynet-l2 NUMBER``: Specifies the level of l2 regularization to
-   use (default 1e-6).
+-  ``--dynet-weight-decay NUMBER``: Adds weight decay to the parameters,
+   which modifies each parameter w such that `w *= (1-weight_decay)` after
+   every update. This is similar to L2 regularization, but different in a
+   couple ways, which are noted in detail in the "Unorthodox Design"
+   section.
 -  ``--dynet-gpus NUMBER``: Specify how many GPUs you want to use, if
    DyNet is compiled with CUDA. Currently, only one GPU is supported.
 -  ``--dynet-gpu-ids X,Y,Z``: Specify the GPUs that you want to use by

doc/source/index.rst

@@ -34,6 +34,7 @@ Mode advanced topics are below:
 
    minibatch
    multiprocessing
+   unorthodox
 
 And we welcome your contributions!
 

doc/source/install.rst

@@ -6,16 +6,16 @@ How to build DyNet and link it with your programs
 Prerequisites
 -------------
 
-DyNet relies on a number of external libraries including Boost, cmake,
-Eigen, and mercurial (to install Eigen). Boost, cmake, and mercurial can
-be installed from standard repositories, for example on Ubuntu linux:
+DyNet relies on a number of external libraries including Boost, CMake,
+Eigen, and Mercurial (to install Eigen). Boost, CMake, and Mercurial can
+be installed from standard repositories, for example on Ubuntu Linux:
 
 ::
 
     sudo apt-get install libboost-all-dev cmake mercurial
 
-Or on OS X, first make sure the Apple Command Line Tools are installed, then
-get boost, cmake, and mercurial with either homebrew or macports:
+Or on OSX, first make sure the Apple Command Line Tools are installed, then
+get Boost, CMake, and Mercurial with either homebrew or macports:
 
 ::
 
@@ -31,7 +31,10 @@ the following command:
 
 ::
 
-    hg clone https://bitbucket.org/eigen/eigen/
+    hg clone https://bitbucket.org/eigen/eigen/ -r 346ecdb
+    
+The `-r NUM` specified a revision number that is known to work.
+Adventurous users can remove it and use the very latest version, at the risk of the code breaking / not compiling.
 
 Building
 --------
@@ -77,7 +80,7 @@ the ``dynet`` directory to the compile path:
 
     -I/path/to/dynet
 
-and link with the dynet library:
+and link with the DyNet library:
 
 ::
 
@@ -95,7 +98,7 @@ If you have a build problem and want to debug, please run
 
 then examine the commands in the ``make.log`` file to see if anything
 looks fishy. If you would like help, send this ``make.log`` file via the
-"Issues" tab on github, or to the dynet-users mailing list.
+"Issues" tab on GitHub, or to the dynet-users mailing list.
 
 
 GPU/MKL support and build options
@@ -123,19 +126,20 @@ CPU or GPU, but this is not fully implemented yet.)
 MKL support
 ~~~~~~~~~~~
 
-DyNet can leverage Intel's MKL library to speed up computation on the CPU. As an example, we've seen 3x speedup in seq2seq training when using MKL. To use MKL, include the following cmake option: 
+DyNet can leverage Intel's MKL library to speed up computation on the CPU.
+As an example, we've seen 3x speedup in seq2seq training when using MKL. To use MKL, include the following cmake option:
 
 ::
 
     -DMKL=TRUE
 
-If cmake is unable to find MKL automatically, try setting `MKL_ROOT`, such as
+If CMake is unable to find MKL automatically, try setting `MKL_ROOT`, such as
 
 ::
 
     -DMKL_ROOT="/path/to/MKL"
 
-If either MKL or MKL_ROOT are set, CMake will look for MKL.
+If either `MKL` or `MKL_ROOT` are set, CMake will look for MKL.
 
 By default, MKL will use all CPU cores. You can control how many cores MKL uses by setting the environment
 variable `MKL_NUM_THREADS` to the desired number. The following is the total time to process 250 training 
@@ -165,9 +169,9 @@ there are diminishing returns or even slowdown.
 Non-standard Boost location
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-``dynet`` supports boost, and will find it if it is in the standard
-location. If boost is in a non-standard location, say ``$HOME/boost``,
-you can specify the location by adding the following to your cmake
+``dynet`` requires Boost, and will find it if it is in the standard
+location. If Boost is in a non-standard location, say ``$HOME/boost``,
+you can specify the location by adding the following to your CMake
 options:
 
 ::
@@ -177,11 +181,13 @@ options:
 
 Note that you will also have to set your ``LD_LIBRARY_PATH`` to point to
 the ``boost/lib`` directory.
+Note also that Boost must be compiled with the same compiler version as
+you are using to compile DyNet.
 
 Building for Windows
 ~~~~~~~~~~~~~~~~~~~~
 
-DYNET has been tested to build in Windows using Microsoft Visual Studio
+DyNet has been tested to build in Windows using Microsoft Visual Studio
 2015. You may be able to build with MSVC 2013 by slightly modifying the
 instructions below.
 
@@ -201,10 +207,10 @@ installed Eigen and Boost (for example, at c:\\libs\\Eigen and c:\\libs\\boost_1
     cd build
     cmake .. -DEIGEN3_INCLUDE_DIR=c:\libs\Eigen -DBOOST_ROOT=c:\libs\boost_1_61_0 -DBOOST_LIBRARYDIR=c:\libs\boost_1_61_0\lib64-msvc-14.0 -DBoost_NO_BOOST_CMAKE=ON -G"Visual Studio 14 2015 Win64"
 
-This will generate dynet.sln and a bunch of \*.vcxproj files (one for
-the DYNET library, and one per example). You should be able to just open
-dynet.sln and build all. **Note: multi-process functionality is
-currently not supported in Windows, so the multi-process examples (*-mp) will not be included 
+This will generate `dynet.sln` and a bunch of `*.vcxproj` files (one for
+the DyNet library, and one per example). You should be able to just open
+`dynet.sln` and build all. **Note: multi-process functionality is
+currently not supported in Windows, so the multi-process examples (`*-mp`) will not be included
 in the generated solution**
 
 The Windows build also supports CUDA with the latest version of Eigen (as of Oct 28, 2016), with the following code change: 

doc/source/minibatch.rst

@@ -1,4 +1,6 @@
-Mini-batching in DyNet
-======================
+.. _minibatching:
+
+Minibatching
+============
 
 How to perform minibatching to improve efficiency

doc/source/operations.rst

@@ -1,5 +1,7 @@
-DyNet Operations
-================
+.. _operations:
+
+Operations
+==========
 
 Operation Interface
 -------------------

doc/source/python.rst

@@ -3,7 +3,7 @@ Installing the Python DyNet module.
 
 (for instructions on installing on a computer with GPU, see below)
 
-Python bindings to DyNet are currently only supported under python 2.
+Python bindings to DyNet are supported for both Python 2.x and 3.x.
 
 TL;DR
 -----
@@ -12,14 +12,14 @@ TL;DR
 
 .. code:: bash
 
-    # Installing python DyNet on a machine with python 2.7:
+    # Installing Python DyNet:
 
-    pip install cython # if you don't have it already.
+    pip install cython  # if you don't have it already.
     mkdir dynet-base
     cd dynet-base
     # getting dynet and eigen
     git clone https://github.com/clab/dynet.git
-    hg clone https://bitbucket.org/eigen/eigen
+    hg clone https://bitbucket.org/eigen/eigen -r 346ecdb  # -r NUM specified a known working revision
     cd dynet
     mkdir build
     cd build
@@ -33,7 +33,7 @@ TL;DR
     python setup.py install  # or `python setup.py install --user` for a user-local install.
     
     # this should suffice, but on some systems you may need to add the following line to your
-    # init files in order for the compiled .so files be accessible to python.
+    # init files in order for the compiled .so files be accessible to Python.
     # /path/to/dynet/build/dynet is the location in which libdynet.dylib resides.
     export DYLD_LIBRARY_PATH=/path/to/dynet/build/dynet/:$DYLD_LIBRARY_PATH
 
@@ -58,7 +58,10 @@ Then get Eigen:
 
     cd $HOME
     cd dynet-base
-    hg clone https://bitbucket.org/eigen/eigen/
+    hg clone https://bitbucket.org/eigen/eigen/ -r 346ecdb
+    
+(`-r NUM` specifies a known working revision of Eigen. You can remove this in order to get the bleeding
+edge Eigen, with the risk of some compile breaks, and the possible benefit of added optimizations.)
 
 We also need to make sure the ``cython`` module is installed. (you can
 replace ``pip`` with your favorite package manager, such as ``conda``,
@@ -82,9 +85,9 @@ Compile DyNet.
 
 This is pretty much the same process as compiling DyNet, with the
 addition of the ``-DPYTHON=`` flag, pointing to the location of your
-python interpreter.
+Python interpreter.
 
-If boost is installed in a non-standard location, you should add the
+If Boost is installed in a non-standard location, you should add the
 corresponding flags to the ``cmake`` commandline, see the `DyNet
 installation instructions page <install.md>`__.
 
@@ -98,20 +101,20 @@ installation instructions page <install.md>`__.
     make -j 2
 
 Assuming that the ``cmake`` command found all the needed libraries and
-didn't fail, the ``make`` command will take a while, and compile dynet
-as well as the python bindings. You can change ``make -j 2`` to a higher
+didn't fail, the ``make`` command will take a while, and compile DyNet
+as well as the Python bindings. You can change ``make -j 2`` to a higher
 number, depending on the available cores you want to use while
 compiling.
 
-You now have a working python binding inside of ``build/dynet``. To
+You now have a working Python binding inside of ``build/dynet``. To
 verify this is working:
 
 .. code:: bash
 
     cd $PATH_TO_DYNET/build/python
     python
 
-then, within python:
+then, within Python:
 
 .. code:: bash
 
@@ -127,10 +130,10 @@ in the system, run the following:
     cd $PATH_TO_DYNET/build/python
     python setup.py install --user
 
-(the ``--user`` switch will install the module in your local
-site-packages, and works without root privilages. To install the module
-to the system site-packages (for all users), run
-``python setup.py install`` without this switch)
+The ``--user`` switch will install the module in your local
+site-packages, and works without root privileges. To install the module
+to the system site-packages (for all users), or to the current `virtualenv`
+(if you are on one), run ``python setup.py install`` without this switch.
 
 You should now have a working python binding (the dynet module).
 
@@ -142,7 +145,6 @@ Now, check that everything works:
 
 .. code:: bash
 
-    # check that it works:
     cd $PATH_TO_DYNET
     cd pyexamples
     python xor.py
@@ -162,8 +164,10 @@ If it doesn't work and you get an error similar to the following:
     Referenced from: /Users/sneharajana/.python-eggs/dyNET-0.0.0-py2.7-macosx-10.11-intel.egg-tmp/_dynet.so
     Reason: image not found``
 
-then you may need to run the following (add it to your shell init files):
-``export DYLD_LIBRARY_PATH=/path/to/dynet/build/dynet/:$DYLD_LIBRARY_PATH``
+then you may need to run the following (and add it to your shell init files):
+
+    export DYLD_LIBRARY_PATH=/path/to/dynet/build/dynet/:$DYLD_LIBRARY_PATH
+
 
 Windows Support
 ------------------
@@ -204,7 +208,7 @@ The installation process is pretty much the same, while adding the
 find it, you can specify also
 ``-DCUDA_TOOLKIT_ROOT_DIR=/path/to/cuda``.)
 
-Now, build the python modules (as above, we assume cython is installed):
+Now, build the Python modules (as above, we assume Cython is installed):
 
 After running ``make -j 2``, you should have the files ``_dynet.so`` and
 ``_gdynet.so`` in the ``build/python`` folder.
@@ -225,7 +229,10 @@ In order to use the GPU support, you can either:
 Running with MKL
 ~~~~~~~~~~~~~~~~
 
-If you've built dynet to use MKL (using -DMKL or -DMKL_ROOT), python sometimes has difficulty finding the MKL shared libraries. You can try setting LD_LIBRARY_PATH to point to your MKL library directory. If that doesn't work, try setting the following environment variable (supposing, for example, your MKL libraries are located at /opt/intel/mkl/lib/intel64):
+If you've built DyNet to use MKL (using ``-DMKL`` or ``-DMKL_ROOT``), Python sometimes has difficulty finding
+the MKL shared libraries. You can try setting ``LD_LIBRARY_PATH`` to point to your MKL library directory.
+If that doesn't work, try setting the following environment variable (supposing, for example,
+your MKL libraries are located at ``/opt/intel/mkl/lib/intel64``):
 
 .. code:: bash