orngWrap: Wrappers for Tuning Parameters and Thresholds

This module contains classes for two very useful purposes: tuning learning algorithm's parameters using internal validation and tuning the threshold for classification into positive class.

Tuning parameters

The module provides two classes, one for fitting a single parameter and one fitting multiple parameters at once, trying all possible combinations. When called with examples and, optionally, id of meta attribute with weights, they find the optimal setting of arguments using the cross validation. The classes can also be used as ordinary learning algorithms - they are in fact derived from orange.Learner.

Both classes have a common parent, TuneParameters, and a few common attributes.

Attributes

object
The learning algorithm whose parameters are to be tuned. This can be, for instance, orngTree.TreeLearner. You will usually use the wrapped learners from modules, not the built-in classifiers, such as orange.TreeLearner directly, since the arguments to be fitted are easier to address in the wrapped versions. But in principle it doesn't matter.
evaluate
The statistics to evaluate. The default is orngStat.CA, so the learner will be fit for the optimal classification accuracy. You can replace it with, for instance, orngStat.AUC to optimize the AUC. Statistics can return either a single value (classification accuracy), a list with a single value (this is what orngStat.CA actually does), or arbitrary objects which the compare function below must be able to compare.
folds
The number of folds used in internal cross-validation. Default is 5.
compare
The function used to compare the results. The function should accept two arguments (e.g. two classification accuracies, AUCs or whatever the result of evaluate is) and return a positive value if the first argument is better, 0 if they are equal and a negative value if the first is worse than the second. The default compare function is cmp. You don't need to change this if evaluate is such that higher values mean a better classifier.
returnWhat
Decides what should be result of tuning. Possible values are
  • TuneParameters.returnNone (or 0): tuning will return nothing,
  • TuneParameters.returnParameters (or 1): return the optimal value(s) of parameter(s),
  • TuneParameters.returnLearner (or 2): return the learner set to optimal parameters,
  • TuneParameters.returnClassifier (or 3): return a classifier trained with the optimal parameters on the entire data set. This is the default setting.
Regardless of this, the learner (given as object) is left set to the optimal parameters.
verbose
If 0 (default), the class doesn't print anything. If set to 1, it will print out the optimal value found, if set to 2, it will print out all tried values and the related

If tuner returns the classifier, it behaves as a learning algorithm. As the examples below will demonstrate, it can be called, given the examples and the result is a "trained" classifier. It can, for instance, be used in cross-validation.

Out of these attributes, the only necessary argument is object. The real tuning classes add two additional - the attributes that tell what parameter(s) to optimize and which values to use.

Tune1Parameter

Class Tune1Parameter tunes a single parameter.

Attributes

parameter
The name of the parameter (or a list of names, if the same parameter is stored at multiple places - see the examples) to be tuned.
values
A list of parameter's values to be tried.

To show how it works, we shall fit the minimal number of examples in a leaf for a tree classifier.

part of tuning1.py

import orange, orngTree, orngWrap learner = orngTree.TreeLearner() data = orange.ExampleTable("voting") tuner = orngWrap.Tune1Parameter(object=learner, parameter="minSubset", values=[1, 2, 3, 4, 5, 10, 15, 20], evaluate = orngStat.AUC) classifier = tuner(data) print learner.minSubset

Set up like this, when the tuner is called, set learner.minSubset to 1, 2, 3, 4, 5, 10, 15 and 20, and measure the AUC in 5-fold cross validation. It will then reset the learner.minSubset to the optimal value found and, since we left returnWhat at the default (returnClassifier), construct and return the classifier from the entire data set. So, what we get is a classifier, but if we'd also like to know what the optimal value was, we can get it from learner.minSubset.

Tuning is of course not limited to setting numeric parameters. You can, for instance, try to find the optimal criteria for assessing the quality of attributes by tuning parameter="measure", trying settings like values=[orange.MeasureAttribute_gainRatio(), orange.MeasureAttribute_gini()].

Since the tuner returns a classifier and thus behaves like a learner, we can used in a cross-validation. Let us see whether a tuning tree indeed enhances the AUC or not. We shall reuse the tuner from above, add another tree learner, and test them both.

import orngTest untuned = orngTree.TreeLearner() res = orngTest.crossValidation([untuned, tuner], data) AUCs = orngStat.AUC(res) print "Untuned tree: %5.3f" % AUCs[0] print "Tuned tree: %5.3f" % AUCs[1]

This will take some time: for each of 8 values for minSubset it will perform 5-fold cross validation inside a 10-fold cross validation - altogether 400 trees. Plus, it will learn the optimal tree afterwards for each fold. Add a tree without tuning, and you get 420 trees build.

Well, not that long, and the results are good:

Untuned tree: 0.930 Tuned tree: 0.986

We mentioned that we will normally use wrapped learners from orng modules, not directly from Orange. Why is this? Couldn't we use orange.TreeLearner instead of orngTree.TreeLearner. Well, we can, except that minSubset is not only deeper, but also appears on two places. To optimize it, we'd do the same as above, except that we'd specify the parameter to optimize with

parameter=["split.continuousSplitConstructor.minSubset", "split.discreteSplitConstructor.minSubset"]

TuneMParameters

The use of TuneMParameters differs from Tune1Parameter only in specification of tuning parameters.

Attributes

parameters
A list of two-element tuples, each containing the name of a parameter and its possible values.

For exercise we can try to tune both settings mentioned above, the minimal number of examples in leaves and the splitting criteria by setting the tuner as follows:

tuningm.py

tuner = orngWrap.TuneMParameters(object=learner, parameters=[("minSubset", [2, 5, 10, 20]), ("measure", [orange.MeasureAttribute_gainRatio(), orange.MeasureAttribute_gini()])], evaluate = orngStat.AUC)

Everything else stays like above, in examples for Tune1Parameter.

Setting Optimal Thresholds

Some models may perform well in terms of AUC which measures the ability to distinguish between examples of two classes, but have low classifications accuracies. The reason may be in the threshold: in binary problems, classifiers usually classify into the more probable class, while sometimes, when class distributions are highly skewed, a modified threshold would give better accuracies. Here are two classes that can help.

Threshold Learner and Classifier

ThresholdLearner is a class that wraps around another learner. When given the data, it calls the wrapped learner to build a classifier, than it uses the classifier to predict the class probabilities on the training examples. Storing the probabilities, it computes the threshold that would give the optimal classification accuracy. Then it wraps the classifier and the threshold into an instance of ThresholdClassifier.

Note that the learner doesn't perform internal cross-validation. Also, the learner doesn't work for multivalued classes. If you don't understand why, think harder. If you still don't, try to program it yourself, this should help. :)

ThresholdLearner has the same interface as any learner: if the constructor is given examples, it returns a classifier, else it returns a learner. It has two attributes.

learner
The wrapped learner, for example an instance of orange.BayesLearner.
storeCurve
If set, the resulting classifier will contain an attribute curve, with a list of tuples containing thresholds and classification accuracies at that threshold.

There's also a dumb variant of ThresholdLearner, a class call ThreshholdLearner_fixed. Instead of finding the optimal threshold it uses a prescribed one. So, it has the following two attributes.

learner
The wrapped learner, for example an instance of orange.BayesLearner.
threshold
Threshold to use in classification.

What this guy does is therefore simple: to learn, it calls the learner and puts the resulting classifier together with the threshold into an instance of ThresholdClassifier.

ThresholdClassifier, used by both ThredholdLearner and ThresholdLearner_fixed is therefore another wrapper class, containing a classifier and a threshold. When it needs to classify an example, it calls the wrapped classifier to predict probabilities. The example will be classified into the second class only if the probability of that class is above the threshold.

classifier
The wrapped classifier, normally the one related to the ThresholdLearner's learner, e.g. an instance of orange.BayesClassifier.
threshold
The threshold for classification into the second class.

The two attributes can be specified set as attributes or given to the constructor as ordinary arguments.

This is how you use the learner.

thresholding1.py

import orange, orngWrap, orngTest, orngStat data = orange.ExampleTable("bupa") learner = orange.BayesLearner() thresh = orngWrap.ThresholdLearner(learner = learner) thresh80 = orngWrap.ThresholdLearner_fixed(learner = learner, threshold = .8) res = orngTest.crossValidation([learner, thresh, thresh80], data) CAs = orngStat.CA(res) print "W/out threshold adjustement: %5.3f" % CAs[0] print "With adjusted thredhold: %5.3f" % CAs[1] print "With threshold at 0.80: %5.3f" % CAs[2]

The output,

W/out threshold adjustement: 0.633 With adjusted thredhold: 0.659 With threshold at 0.80: 0.449 shows that fitting threshold is good (well, although 2.5 percent increase in the accuracy absolutely guarantees you a publication at ICML, the difference is still unimportant), while setting it at 80% is a bad idea. Or is it?

thresholding2.py

import orange, orngWrap, orngTest, orngStat data = orange.ExampleTable("bupa") ri2 = orange.MakeRandomIndices2(data, 0.7) train = data.select(ri2, 0) test = data.select(ri2, 1) bayes = orange.BayesLearner(train) thresholds = [.2, .5, .8] models = [orngWrap.ThresholdClassifier(bayes, thr) for thr in thresholds] res = orngTest.testOnData(models, test) cm = orngStat.confusionMatrices(res) print for i, thr in enumerate(thresholds): print "%1.2f: TP %5.3f, TN %5.3f" % (thr, cm[i].TP, cm[i].TN)

The script first divides the data into training and testing examples. It trains a naive Bayesian classifier and than wraps it into ThresholdClassifiers with thresholds of .2, .5 and .8. The three models are tested on the left-out examples, and we compute the confusion matrices from the results. The printout,

0.20: TP 60.000, TN 1.000 0.50: TP 42.000, TN 24.000 0.80: TP 2.000, TN 43.000 shows how the varying threshold changes the balance between the number of true positives and negatives.