This module contains classes for two very useful purposes: tuning learning algorithm's parameters using internal validation and tuning the threshold for classification into positive class.
The module provides two classes, one for fitting a single parameter and one fitting multiple parameters at once, trying all possible combinations. When called with examples and, optionally, id of meta attribute with weights, they find the optimal setting of arguments using the cross validation. The classes can also be used as ordinary learning algorithms - they are in fact derived from orange.Learner.
Both classes have a common parent,
, and a few common attributes.
Attributes
orngTree.TreeLearner
. You will usually use the wrapped learners from modules, not the built-in classifiers, such as orange.TreeLearner
directly, since the arguments to be fitted are easier to address in the wrapped versions. But in principle it doesn't matter.orngStat.CA
, so the learner will be fit for the optimal classification accuracy. You can replace it with, for instance, orngStat.AUC
to optimize the AUC. Statistics can return either a single value (classification accuracy), a list with a single value (this is what orngStat.CA
actually does), or arbitrary objects which the compare
function below must be able to compare.evaluate
is) and return a positive value if the first argument is better, 0 if they are equal and a negative value if the first is worse than the second. The default compare
function is cmp
. You don't need to change this if evaluate
is such that higher values mean a better classifier.TuneParameters.returnNone
(or 0): tuning will return nothing,TuneParameters.returnParameters
(or 1): return the optimal value(s) of parameter(s),TuneParameters.returnLearner
(or 2): return the learner set to optimal parameters,TuneParameters.returnClassifier
(or 3): return a classifier trained with the optimal parameters on the entire data set. This is the default setting.object
) is left set to the optimal parameters.
If tuner returns the classifier, it behaves as a learning algorithm. As the examples below will demonstrate, it can be called, given the examples and the result is a "trained" classifier. It can, for instance, be used in cross-validation.
Out of these attributes, the only necessary argument is object
. The real tuning classes add two additional - the attributes that tell what parameter(s) to optimize and which values to use.
Class
tunes a single parameter.
Attributes
To show how it works, we shall fit the minimal number of examples in a leaf for a tree classifier.
part of tuning1.py
Set up like this, when the tuner is called, set learner.minSubset
to 1, 2, 3, 4, 5, 10, 15 and 20, and measure the AUC in 5-fold cross validation. It will then reset the learner.minSubset
to the optimal value found and, since we left returnWhat
at the default (returnClassifier
), construct and return the classifier from the entire data set. So, what we get is a classifier, but if we'd also like to know what the optimal value was, we can get it from learner.minSubset
.
Tuning is of course not limited to setting numeric parameters. You can, for instance, try to find the optimal criteria for assessing the quality of attributes by tuning parameter="measure"
, trying settings like values=[orange.MeasureAttribute_gainRatio(), orange.MeasureAttribute_gini()]
.
Since the tuner returns a classifier and thus behaves like a learner, we can used in a cross-validation. Let us see whether a tuning tree indeed enhances the AUC or not. We shall reuse the tuner
from above, add another tree learner, and test them both.
This will take some time: for each of 8 values for minSubset
it will perform 5-fold cross validation inside a 10-fold cross validation - altogether 400 trees. Plus, it will learn the optimal tree afterwards for each fold. Add a tree without tuning, and you get 420 trees build.
Well, not that long, and the results are good:
We mentioned that we will normally use wrapped learners from orng modules, not directly from Orange. Why is this? Couldn't we use orange.TreeLearner
instead of orngTree.TreeLearner
. Well, we can, except that minSubset
is not only deeper, but also appears on two places. To optimize it, we'd do the same as above, except that we'd specify the parameter to optimize with
The use of
differs from Tune1Parameter
only in specification of tuning parameters.
Attributes
For exercise we can try to tune both settings mentioned above, the minimal number of examples in leaves and the splitting criteria by setting the tuner as follows:
Everything else stays like above, in examples for Tune1Parameter
.
Some models may perform well in terms of AUC which measures the ability to distinguish between examples of two classes, but have low classifications accuracies. The reason may be in the threshold: in binary problems, classifiers usually classify into the more probable class, while sometimes, when class distributions are highly skewed, a modified threshold would give better accuracies. Here are two classes that can help.
is a class that wraps around another learner. When given the data, it calls the wrapped learner to build a classifier, than it uses the classifier to predict the class probabilities on the training examples. Storing the probabilities, it computes the threshold that would give the optimal classification accuracy. Then it wraps the classifier and the threshold into an instance of ThresholdClassifier
.
Note that the learner doesn't perform internal cross-validation. Also, the learner doesn't work for multivalued classes. If you don't understand why, think harder. If you still don't, try to program it yourself, this should help. :)
ThresholdLearner
has the same interface as any learner: if the constructor is given examples, it returns a classifier, else it returns a learner. It has two attributes.
orange.BayesLearner
.curve
, with a list of tuples containing thresholds and classification accuracies at that threshold.There's also a dumb variant of ThresholdLearner
, a class call ThreshholdLearner_fixed
. Instead of finding the optimal threshold it uses a prescribed one. So, it has the following two attributes.
orange.BayesLearner
.What this guy does is therefore simple: to learn, it calls the learner and puts the resulting classifier together with the threshold into an instance of ThresholdClassifier
.
, used by both ThredholdLearner
and ThresholdLearner_fixed
is therefore another wrapper class, containing a classifier and a threshold. When it needs to classify an example, it calls the wrapped classifier to predict probabilities. The example will be classified into the second class only if the probability of that class is above the threshold.
ThresholdLearner
's learner
, e.g. an instance of orange.BayesClassifier
.The two attributes can be specified set as attributes or given to the constructor as ordinary arguments.
This is how you use the learner.
The output,
The script first divides the data into training and testing examples. It trains a naive Bayesian classifier and than wraps it into ThresholdClassifier
s with thresholds of .2, .5 and .8. The three models are tested on the left-out examples, and we compute the confusion matrices from the results. The printout,