AI Seminar 2004/2005

(please see the main page for schedule information)

Speaker: Rich Caruana

Which Supervised Learning Method Works Best For What? 
 An Empirical Comparison of Ten Learning Algorithms.

Abstract:

Decision trees may be intelligible, but can they cut the mustard?
Have SVMs replaced neural nets, or are neural nets still best for
regression, and SVMs best for classification?  Boosting maximizes a
margin much like SVMs, but can boosting compete with SVMs?  And is it
better to boost weak models, as theory suggests, or to boost stronger
models?  Bagging is easier than boosting, so how well does bagging
stack up against boosting?  Breiman says Random Forests is better than
bagging and as good as boosting.  Is he right?  And what about old
friends like memory-based learning, logistic regression, and naive
bayes -- should they be put out to pasture, or do they still fill an
important niche?

In this talk we'll empirically compare the performance of ten
supervised learning methods on nine different performance criteria:
Accuracy, F-score, Lift, Precision/Recall Break-Even Point, Area under
the ROC, Average Precision, Squared Error, Cross-Entropy, and
Probabilistic Calibration.  The results show that no one learning
method does it all, but it is possible to "repair" some of them so
that they do very well on all performance metrics.  We'll then
describe NACHOS, an ensemble method that combines models from these
ten learning methods to yield even better performance.  Finally, if
time permits, we'll discuss how the nine performance metrics relate to
each other, and which metrics you probably should or shouldn't use.

Speaker Bio:

Some of you may remember Rich from the "old" days -- he worked with
Tom Mitchell and Herb Simon, and finished his Ph.D. in 1997.  After a
few glorious years at Scott Fahlman's Justsystem Pittsburgh Research
Lab (a.k.a. JustResearch), Rich began commuting from Pittsburgh to
UCLA where he was faculty at the Med School and Computer Science
Department.  UCLA was great, but all that concrete didn't look like a
good place to put down roots, so he left UCLA and become the first
Research Faculty hired by CALD.  A year later Rich left the nest and
moved to anti-L.A., otherwise known as Ithaca, NY, where he is an
assistant professor of Computer Science at Cornell.  Rich's research
is in machine learning and data mining.  His current focus is ensemble
learning, inductive transfer, meta clustering, and applications of
these to problems in bioinformatics, medical decision-making, and
weather forecasting.  As always, he likes to mix algorithm development
with applications to insure that the methods he develops really work.
The empirical comparison he'll talk about results from a massive study
he and his students did "by accident" while developing a new ensemble
learning algorithm.

---

Maintainer is

Jack Mostow

Last modified: Tue May 10 14:19:28 EDT 2005