Overparameterization

Code for Subquadratic Overparameterization for Shallow Neural Networks NeurIPS 2021.

Usage

For possible arguments see:

python runner.py -h

Example:

source .env
python runner.py --exp-name mnist-test --d1 1000 --dataset MNIST --lr 0.01 --w1 0.03 --w2 0.04 --epochs 300 --batch-size 128 --mse-loss --remote

Note that --remote requires the Slurm workload manager on the server.

Recreating experiments

To recreate Figure 1 from the paper see notebook.ipynb by running the following:

source .env
jupyter notebook notebook.ipynb

Installation

• CPU:

  conda env create -f environment_cpu.yml
  source activate overparameterization

• GPU enabled:

  conda env create -f environment_gpu.yml
  source activate overparameterization

Server install

Note: The server setup requires the server to use Slurm as its queuing manager.

1. Locally change the .env file to:

   PYTHONPATH=.
   export SERVER_USERNAME="myusername"
   export SERVER_NAME="myserver.com"

2. On the server create the necessary folder:

   source .env
   ssh $(SERVER_USERNAME)@$(SERVER_NAME)
   mkdir ~/overparameterization
   mkdir ~/overparameterization/output

3. Push the repository to the server:

   source .env
   make push

4. Install the python dependencies on the server:

   ssh $(SERVER_USERNAME)@$(SERVER_NAME)
   cd ~/overparameterization
   conda env create -f environment.yml