NIPS*2000 Workshop: Cross-Validation, Bootstrap and Model Selection

NIPS*2000 Workshop:
Cross-Validation, Bootstrap and Model Selection

December 1, 2000: Beaver Run Resort (Peaks 9/10), Breckenridge, Colorado
Final schedule

	Workshop Organizers: Rahul Sukthankar, Compaq CRL and Robotics Institute, Carnegie Mellon Larry Wasserman, Department of Statistics, Carnegie Mellon Rich Caruana, Center for Automated Learning and Discovery, Carnegie Mellon
Workshops

Description

Cross-validation and bootstrap are popular methods for estimating generalization error based on resampling a limited pool of data, and have become widely-used for model selection. The aim of this workshop is to bring together researchers from both matchine learning and statistics in an informal setting to discuss current issues in resampling-based techniques. These include:

Improving theoretical bounds on cross-validation, bootstrap or other resampling-based methods;
Empirical or theoretical comparisons between resampling-based methods and other forms of model selection;
Exploring the issue of overfitting in sampling-based methods;
Efficient algorithms for estimating generalization error;
Novel resampling-based approaches to model selection.

This one-day workshop will begin with an invited talk by Brad Efron, Professor of Statistics (Stanford) and inventor of the bootstrap algorithm. The remainder of the morning session will feature four talks focusing primarily on theoretical issues. The evening session will consist of four talks describing applications and new algorithms. Each 20 minute presentation will be followed by 10 minutes of discussion where interesting questions raised during the talk can be explored in some detail. Extended abstracts and other materials will be made available here, in advance so that workshop participants may prepare for the discussion sessions.

Call for Papers

The workshop submissions are closed. Accepted papers are listed in the schedule below.

Workshop Schedule

	Morning Session
7:30-7:45	Introduction (Rahul Sukthankar)
7:45-8:45	Invited speaker: Brad Efron, The Cost of Model Selection
8:45-9:15	Olivier Chapelle, Vladimir Vapnik, Yoshua Bengio: Model Selection for Small Sample Regression
9:15-9:45	John Langford: PAC Bounds for Holdout Procedures
9:45-10:00	Break
10:00-10:30	Carl Rasmussen, Pedro Højen-Sørensen: Empirical Model Comparison: Bayesian Analysis of Disjoint Cross Validation for Continuous Losses
10:30-11:00	Masashi Sugiyama, Hidemitsu Ogawa: Subspace Information Criterion - Unbiased Generalization Error Estimator for Linear Regression

	Evening Session
16:30-17:00	Bart Bakker, Tom Heskes: Model Clustering and Resampling
17:00-17:30	Matthew Mullin, Rahul Sukthankar: Complete Cross Validation for Nearest Neighbor Classifiers
17:30-17:45	Break
17:45-18:15	Ashutosh Garg, Ira Cohen, Thomas Huang: Sampling Based EM Algorithm
18:15-18:45	Koji Tsuda, Gunnar Rätsch, Sebastian Mika, Klaus-Robert Müller: Learning to Predict the Leave-one-out Error
18:45-19:00	Wrap-up discussion

Additional Downloads

Extended abstracts are linked to the paper title above. This section contains pre-prints to full papers and additional related work.

Brad Efron talk slides

Olivier Chapelle, Vladimir Vapnik, Yoshua Bengio preprint

John Langford COLT-99 paper
Webpage with additional information (including NIPS presentation)

Masashi Sugiyama, Hidemitsu Ogawa Neural Computation article

Matthew Mullin, Rahul Sukthankar ICML paper

Contact Information

The workshop organizers can be contacted by email at rahuls=nips@cs.cmu.edu, or at the phone/fax numbers listed below.

Email:

Phone:

Fax:

Rahul Sukthankar

rahuls@cs.cmu.edu

+1-617-551-7694

+1-617-551-7650

Larry Wasserman

larry@stat.cmu.edu

+1-412-268-8727

+1-412-268-7828

Rich Caruana

caruana@cs.cmu.edu

+1-412-268-7664

Rahul Sukthankar (rahuls@cs.cmu.edu)

Last updated: November 26, 2000

Brad Efron	talk slides
Olivier Chapelle, Vladimir Vapnik, Yoshua Bengio	preprint
John Langford	COLT-99 paper Webpage with additional information (including NIPS presentation)
Masashi Sugiyama, Hidemitsu Ogawa	Neural Computation article
Matthew Mullin, Rahul Sukthankar	ICML paper