This is the experiment code for the paper:
"Newton Methods for Convolutional Neural Networks".

Here we show how to reproduce the experiment results for subsampled Newton
methods and SG methods in the paper. However, running on different machine may
produce slightly different results due to the round-off error.


How to run the experiments
==========================

First of all, download data.
$ ./prepare_data.sh

1. To generate Table 5:

(1) Enter Table5 directory.
$ cd Table5;

(2) Make libsvmread.c and libsvmwrite.c.
$ matlab -r "make; exit"

For Octave users,
$ octave-cli --eval "make; exit"

(3) For each script file in the `script' directory, EXP_HOME variable (in line 3
of each file) must be set to the experiment home path (The absolute path where
`experiment.m' resides).

Moreover, for Octave users, replace the command,
    matlab -nodesktop -nosplash -r
with
    octave-cli --eval

Then, the jobs configure file must be re-generate by the command as follows.
This command will also create a directory named `logs'.

$ make all

(4) Run the python code to generate results for Table 5 in the paper.

* (4.1) Cross Validation

For SG five-folds cross-validation, run the following command in the terminal.
$ python3 run_CV.py --cv_splits 5

After the cross validation procedure finished, the best learning rate will be
output to ``logs/CV_result.log''.

If you would like to deploy the experiments in a workstation, please see the
following instruction. Otherwise you can skip the rest of this section.

To deploy the experiments in a workstation, first make sure the passwordless SSH
login is setup. You should be able to connect to the workstation from localhost
without password. Then add `<username>@<hostname>' in the `config/hosts.config'
line by line. In the end, the `config/hosts.config' should look like this:

my_login_account@host1.csie.ntu.edu.tw
my_login_account@host2.csie.ntu.edu.tw

In each host, put this experiment directory at the path where EXP_HOME
is being assigned to. Then run the command with `--host' argument:
$ python3 run_CV.py --host config/hosts.config --cv_splits 5

* (4.2) Newton and SG

After cross validation, fill in the best learning rate accordingly in line 11-12
in ``script/gen_exp1_newton_sg.sh''. Then, run the script file to generate the config
file.
$ make configure

Run the experiment and generate Table 5 with the command

$ python3 exp1.py --out_filename newton_sg_exp1.tex

or

$ python3 exp1.py --host config/hosts.config --out_filename newton_sg_exp1.tex

See Section (4.1) above for more information about `--host' argument. The Table
will be save to ``newton_sg_exp1.tex''. If `--out_filename' argument is not
given, it will print the table to stdout.

2. To draw Figure 3:

(1) Enter Figure3 directory
$ cd Figure3;

(2) Make sure the MATLAB path (in line 3) in the following script file are correct.
 - `simpleNN/experiment/run_newton.sh'
 - `simpleNN/experiment/run_sg.sh'

(3) Generate Newton and SG method logs
$ cd simpleNN/experiment/;
$ ln -s ../../../data data;
$ ./run.sh;
$ cd -;

(4) Plot Figure3
$ mkdir graph;
# The graphs should be stored at ``graph'' directory.
$ ./plot.sh mnist;
$ ./plot.sh cifar10;
$ ./plot.sh smallNORB-32x32;
$ ./plot.sh SVHN;

3. To generate Table 6:

(1) Enter Table6 directory
$ cd Table6;

(2) Make sure the MATLAB path (in line 3) in the following script file are correct.
 - `cpuversion/simpleNN/experiment/run_newton.sh'
 - `gpuversion/simpleNN/experiment/run_sg.sh'

(3) Generate Newton method logs
$ cd cpuversion/simpleNN/experiment;
$ ln -s ../../../../data data;
$ ./run.sh;
$ cd -;

(4) Generate SG method logs
$ cd gpuversion/simpleNN/experiment;
$ ln -s ../../../../data data;
$ ./run.sh;
$ cd -;

(5) Generate table tex file
$ ./plot_robust.sh > output.tex
