Tutorial
We outline below the toolkit's functionality by following through a concrete example applying the toolkit in a Matlab environment. The example below demonstrates the usability of the toolkit in the different stages of predicting and evaluating collaborative filtering.
| The first step is to read the data set and store it in
memory. The toolkit may be used to parse EachMovie data and read it to
memory by issuing the routine eachMovieReader: >>userVoteMat=eachMovieReader(300); The number 300 represents the number of users that are read from the dataset. The dataset contains data gathered from 80000 users - to read all of them replace 300 by 80000. This the entire dataset should take several minutes and exhaust a large part of the physical memory. Reading a subset of 300 users should a few seconds. The output userVoteMat is a sparse matrix in which userVoteMat(i,j) is the recorded preference of the user i for movie j. If no preference with respect to that item was reported by the user the entry will be 0. To view the preference of user i type userVoteMat(i,:) in the matlab prompt. To view the sparsity pattern of the matrix type spy(userVoteMat). |
| The next step is to split the data set into a set of
active users and a set of non-active users. The latter set will be used to
provide predictions for the active users preference. This division is
analogous to a training set/testing set division in classification
scenario. The division basically chooses some of the rows randomly and
form a new matrix activeMat with the
remaining rows forming the new matrix otherMat.
This may be done using the routine splitUsers
below >>numNonActive=size(userVoteMat,1)-150; The second and third arguments in the function call are the numbers of active and non-active users respectively. Other numbers may be selected of course, but the sum of both numbers should equal the number of users. Note that userVoteMat may contain less users than 300. This is because the routine eachMovieReader post-process the users and removes users with extremely low number of preference. For example, users with a vote of only one item are completely unusable in a collaborative filtering framework. |
| The next step is to partition the set of items to
observed and predicted items. The first set is the set of items that the active
user reports his preference on to the system. The system is then trained
based on that information. The second set of items that was
held out for the training phase is used for evaluating the system's responses. The system recommendations for
the second set of items for the active user is compared to the active user
preference and using one of the evaluation measures, a numeric score is
calculated. Partitioning the items is done inside the routine
EachMovieComparison. To
perform it manually outside the routing, simply cut and paste the code to
the Matlab environment. |
| To obtain a prediction of a user's preference use the
routines predictPreferencePD for personality diagnosis prediction and
predictPreferenceMemBased for memory based prediction. For example,
>>[pred,meanPred]=predictPreferencePD(activeUser, otherUserMat,numValues,sigma) returns the probability of different preference for different items (a stochastic matrix) in pred and the mean predicted preference (row vector) for each item in meanPred. activeUser is a row vector denoting the preference of the active user on the reported items and otherUserMat denotes the preference of the other non-active users. numValues is the number of different preference values (6 in EachMovie dataset) and sigma is the standard deviation of the Gaussian distribution in the PD model. This is a free parameter and requires some tuning for optimal performance. |
| The prediction output from predictPreferncePD and
predictPreferneceMemBased may be evaluated using one of the evaluation
metrics as described in the toolkit description. For example, the routine
rankedEvalCF evaluates the given
recommendations by ranking the items according to the output mean predicted
values and then computing the expected utility of a user browsing the list,
with the probability of the user selecting an item on the list decays
exponentially with the rank of the item in the list. |
| It is also possible to evaluate the methods directly,
without calling predictPreferencePD or
predcitPreferenceMemBased by using the
supplied routines evalPDEachMovie, evalMemBasedEachMovie, evalConstEachMovie, evalAvgEachMovie
with each routine returning the evaluation L1, L2
and ranked evaluation measures for different prediction strategies.
For example, [L1err,L2err,rankedErr]=evalMemBasedEachMovie(activeMatTrain,... returns the L1 error, the L2 error and the ranked evaluation measure
for the memory based method described by simMethod. |