Earlier, we manually tried different metacodes and asked Vizier to evaluate them with LOO-XVE so we could use the best one. Its much easier, though, to give Blackbox a list of metacodes and have it automatically go through them and report back the best one. In this example, we'll use Blackbox to determine the best number of nearest neighbors to use in a kernel regression.
File -> Open -> c1.mbl
Edit -> Metacode -> A01:9
Model -> Blackbox -> Search Algorithm Best Neighbors
Cross Validation Police Use Testset
Testset: None
Reset ON
Launch!
(Repeat for files k1.mbl and i1.mbl)
First, we set the metacode to one that we'd like the search to begin from. Then we choose which search algorithm to use. The default is a totally autonomous search. ``Best Neighbors'' means that the search will try all the different number of nearest neighbors in conjunction with the other settings specified by the metacode we just chose. The CV Police is set to use a separate test set. This feature allows you to separate your data manually into training and testing data. The filename ``None'' is a special case which means do not use anything for the CV Police. We chose that in this example because we want to choose a metacode based strictly on LOO-XVE. The reset feature means we want to clear all previous Blackbox work. If we had run Blackbox before on this data set, the search would begin from the best previous metacode found. Since we want it to start from the metacode just specified, we tell Blackbox to erase any previous results first. Launch begins the search. Depending on the speed of your computer, you will see the metacodes being tried and their performance in the status bar at the bottom of Vizier's window. At the end, Blackbox displays its choice as the best metacode.
Similarly, we can use Blackbox to find the best kernel width for a local linear regression. In order to do that, we would start by setting the metacode to ``L10:9'', and then choosing the ``Best Smoothing'' search algorithm in the Blackbox dialogue box.
It is also possible to use the Blackbox dialogue box to eliminate some kinds of metacodes from consideration. This is done with the array of check boxes. For example, suppose we have a high dimensional data set and we know it will be impractical to fit quadratic models to it. Quadratic models can be eliminated from the search by unchecking the ``Quadratic'' check box.