Given the above representation and learning method, the obvious question is ``How well can WebWatcher learn to advise the user?'' To estimate the answer to this question, the available data was split into training and testing sets. Each learning method was applied to the set of training sessions and evaluated according to how frequently it recommended the hyperlink taken by the user in the separate testing sessions.
In order to obtain more statistically significant estimates of learning accuracy, the training data was separated into 29 training sessions and one test session, in each of the 30 possible ways. Each learning method was then applied to each training session collection and evaluated on the test session. The results of these 30 experiments were averaged. This procedure was run for each of the four learning methods.
Figure: Accuracy of advice for different methods. The vertical axis
indicates the fraction of pages for which the recommended hyperlinks included
the link chosen by the user. The horizontal axis indicates the number of
hyperlinks recommended per page. The worst case line shows the fraction of
pages having N or fewer links total.
Figure 6 plots the results of this experiment. The vertical axis indicates the fraction of test cases in which the user-selected hyperlink was among those recommended by the learned knowledge. The horizontal axis indicates the number of hyperlinks that the learner was allowed to recommend for each page. Thus, the leftmost point of each line indicates the fraction of cases in which the user chose the learner's highest-rated link. The second point to the left indicates the fraction of cases in which the user chose one of the two highest-rated links, and so on.
Notice that all three learning methods significantly outperform randomly
generated advice. For example, Winnow recommends the user-selected
link as its first choice in 30% of the test cases, and among its top three
choices in 54% of the cases. Given the mean of 16 links per page in this
data, random advice chooses the user-selected link only 6% of the time.