For some unbalanced data sets, accuracy may not be a good criterion for evaluating a model. This tool enables LIBSVM to conduct cross-validation and prediction with respect to different criteria (e.g., F-score, AUC..).

What can this tool do?

- Cross-validation with different criteria (F-score, AUC, or BAC)
- Using different evaluations in prediction (precision, recall, F-score, AUC, or BAC)

Please note that precision or recall may not be a good criterion for cross validation because you can easily get 100% precision/recall by predicting all data in one class. - Support parameter search with grid.py
- An simple framework for designing new evaluation functions
- Easy extension for LIBLINEAR
- MATLAB support

precision

- Precision = true_positive / (true_positive + false_positive)
recall

- Recall = true_positive / (true_positive + false_negative)
fscore

- F-score = 2 * Precision * Recall / (Precision + Recall)
bac

- BAC (Balanced ACcuracy) = (Sensitivity + Specificity) / 2,

where Sensitivity = true_positive / (true_positive + false_negative)

and Specificity = true_negative / (true_negative + false_positive) auc

- AUC (Area Under Curve) is the area under the ROC curve.
ap

- AP (Average Precision) approximates the area under the Precision-Recall curve.

In the scenario of highly-unbalanced data (such as information retrieval area), AP metric is commonly used.

In the implementation, we assume that the majority of the data is labeled as negetive.

Note: This tool is designed only for binary-class C-SVM with labels {1,-1}.
Multi-class, regression and probability estimation are not supported.

Note: When using accuracy as the
evaluation criterion, the cross validation accuracy may be
different from that by standard LIBSVM. The reason is that
LIBSVM internally group data in the same class, while this
tool does not.

- Download eval.cpp, eval.h, and Makefile, and put them to the directory of LIBSVM (overwrite the old Makefile).
- Add
#include "eval.h"

to svm-train.c and svm-predict.c. - Replace
do_cross_validation();

in the main() of svm-train.c withdouble cv = binary_class_cross_validation(&prob, ¶m, nr_fold); printf("Cross Validation = %g%%\n",100.0*cv);

Note that the percentage mark is necessary in order to use grid.py. No need to change other places where do_cross_validation() appears. - Replace
predict(input,output);

in main() of svm-predict.c withbinary_class_predict(input, output);

- Assign the global variable
double (*validation_function)(const dvec_t&, const ivec_t&) = auc;

in eval.cpp to the evaluation function you preferred. You can also assign precision, recall, fscore, or bac here. - Recompile LIBSVM with the new Makefile.
make clean; make

To display various evaluation results in prediction, you can displace

validation_function(dec_values, true_labels);in binary_class_predict() of eval.cpp. For example, to see accuracy, precision, and recall, you can write

accuracy(dec_values, true_labels); precision(dec_values, true_labels); recall(dec_values, true_labels);The output will be like

Accuracy = 86.6667% (234/270) Precision = 88.1818% (97/110) Recall = 80.8333% (97/120)

The best parameters vary among different performance evaluations.
Using grid.py (at tools/ in LIBSVM), you can choose the best parameters with respect to any specified evaluation function.
grid.py will search best parameters *C* and *g* by cross-validation.

Here is an example output of grid.py for the data set heart_scale when the evaluation function is AUC:

512.0 0.00048828125 90.7111The best cross-validation AUC is 90.7111% when (

Because grid.py maximizes the evaluation value, the evaluation function should satisfy the property that a better model gives a higher value.

New evaluation functions should be added in eval.cpp. The prototype of an evaluation function is

typedef std::vector<double> dvec_t; typedef std::vector<int> ivec_t; double eval_func(const dvec_t& dec_values, const ivec_t& ty);where dec_values is a vector of decision values and ty is a vector of true labels (+1 or -1). This function returns the evaluation value.

Here is an example showing how recall is implemented.

- Add a function prototype in eval.cpp.
double recall(const dvec_t& dec_values, const ivec_t& ty);

- Implement the recall function.
double recall(const dvec_t& dec_values, const ivec_t& ty){ size_t size = dec_values.size(); size_t i; int tp, fn; // true_positive and false negative double recall; tp = fn = 0; for(i = 0; i < size; ++i) if(ty[i] == 1){ // true label is 1 if(dec_values[i] >= 0) ++tp; // predict label is 1 else ++fn; // predict label is -1 } recall = tp / (double) (tp + fn); // print result in case of invocation in prediction printf("Recall = %g%%\n", 100.0 * recall); return recall; // return the evaluation value }

- Assign the global variable
double (*validation_function)(const dvec_t&, const ivec_t&) = recall;

- Modify grid.py if your criterion is the smaller the better.

Modifying LIBLINEAR is similar to modifying LIBSVM. Please download the files for LIBLINEAR: eval.cpp, eval.h, and Makefile. Please note that svm-train.c and svm-predict.c in LIBSVM become train.c and predict.c in LIBLINEAR. Besides, predict(input,output) in LIBSVM becomes do_predict(input,output,model_) in LIBLINEAR.

Please download the files do_binary_cross_validation.m, do_binary_predict.m, and validation_function.m. Put them to the matlab directory of LIBSVM.

Assign the variable

valid_function = @(dec, labels) auc(dec, labels);in validation_function.m to the evaluation function you preferred. You can assign auc, precision, recall, fscore, or bac here. You can use the following two functions.

Usage:

> do_binary_cross_validation(training_label_vector, training_instance_matrix, 'libsvm_options', n_fold); > [predicted_label, evaluation_result, decision_values] = do_binary_predict(testing_label_vector, testing_instance_matrix, model);

Examples:

[trainY trainX] = libsvmread('./data.scale'); [testY testX] = libsvmread('./data.scale.t'); do_binary_cross_validation(trainY, trainX, '-c 8 -g 4', 5); model = svmtrain(trainY, trainX); [pred eval_ret dec] = do_binary_predict(testY, testX, model);These files can be used for LIBLINEAR, though you need to replace svmtrain and svmpredict with train and predict, respectively.

Please contact Chih-Jen Lin for any question.