running LipSDP on a single layer

[trevoravant]

Hi,

I am wondering if it is possible to apply your method to a single layer, i.e., relu(Wx + b). When I try to give LipSDP one layer, I get an "Inner matrix dimensions must agree." error. More specifically, I used the following code to create the weight file (which is adapted from the example code in the README):

import os
import numpy as np
from scipy.io import savemat

weights = []
net_dims = [2, 10]
num_layers = len(net_dims) - 1
norm_const = 1 / np.sqrt(num_layers)

for i in range(1, len(net_dims)):
  weights.append(norm_const * np.random.rand(net_dims[i], net_dims[i-1]))

fname = os.path.join(os.getcwd(), 'examples/saved_weights/random_weights.mat')
data = {'weights': np.array(weights, dtype=np.object)}
savemat(fname, data)

Then I ran:

$ python solve_sdp.py --form neuron --weight-path examples/saved_weights/random_weights.mat
Error using * (line 41)
Inner matrix dimensions must agree.

Error in lipschitz_multi_layer (line 161)
        (A_on_B' * Q * A_on_B) - M <= 0;

Error in solve_LipSDP (line 41)
        L = lipschitz_multi_layer(weights, lip_params.formulation, ...

Traceback (most recent call last):
  File "solve_sdp.py", line 112, in <module>
    main(args)
  File "solve_sdp.py", line 35, in main
    L = eng.solve_LipSDP(network, lip_params, nargout=1)
  File "/home/trevor/.local/lib/python3.8/site-packages/matlab/engine/matlabengine.py", line 70, in __call__
    return FutureResult(self._engine(), future, nargs, _stdout,
  File "/home/trevor/.local/lib/python3.8/site-packages/matlab/engine/futureresult.py", line 67, in result
    return self.__future.result(timeout)
  File "/home/trevor/.local/lib/python3.8/site-packages/matlab/engine/fevalfuture.py", line 82, in result
    self._result = pythonengine.getFEvalResult(self._future,self._nargout, None, out=self._out, err=self._err)
matlab.engine.MatlabExecutionError:
  File /home/trevor/Documents/MATLAB/cvx/builtins/@cvx/mtimes.m, line 41, in mtimes

  File /home/trevor/LipSDP/LipSDP/matlab_engine/lipschitz_multi_layer.m, line 161, in lipschitz_multi_layer

  File /home/trevor/LipSDP/LipSDP/matlab_engine/solve_LipSDP.m, line 41, in solve_LipSDP
Inner matrix dimensions must agree.

I get the error show above.

I also know that there is a --splitoption, and my understanding is that with --split --split-size 1LipSDP would be applied to each layer individually, and then the results would be multiplied together. When I use this option on the one-layer network, no error is thrown, but a Lipschitz constant of 0 is returned.

[arobey1]

To answer your first question, the program expects at least two linear transformations in the definition of a neural network, i.e. f(x) = W_2 * phi(W_1 * x + b_1) + b_2. So for your use case, I would recommend setting W_2 = I (the identity matrix) and b_2 = 0 (the zero vector). Let me know if that works for you.

To answer the second question, the --split-size argument is expecting an integer >= 2. That sounds like a bug though -- perhaps the easiest fix is to ensure that --split is used for networks of appropriate size.

[trevoravant]

Ok, that makes sense, thanks for the info. So how do you specify the bias of a layer? Looking at your random_weights.mat and mnist_weights.mat files, there seems to be a weights key, but nothing related to bias.

[sarosijbose]

Hi @arobey1, are the number of hidden fc layers in mnist_weights.mat 5 as shown in the paper or is it 2 because on loading the weights, I am getting the latter.
EDIT: Is there any chance you ( or anyone else) remember the naive upper bound for the random_weights.mat you have provided? I have found out a value but it is way too close to the tight bound.....albeit it is for 4 hidden layers only.