Journal of Artificial Intelligence Research 4 (1996) 397-417. Submitted 12/95; published 5/96.
© 1996 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.

Further Experimental Evidence against the Utility of Occam's Razor

Geoffrey I. Webb webb@deakin.edu.au
School of Computing and Mathematics
Deakin University
Geelong, Vic, 3217, Australia.

Abstract:

This paper presents new experimental evidence against the utility of Occam's razor. A systematic procedure is presented for post-processing decision trees produced by C4.5. This procedure was derived by rejecting Occam's razor and instead attending to the assumption that similar objects are likely to belong to the same class. It increases a decision tree's complexity without altering the performance of that tree on the training data from which it is inferred. The resulting more complex decision trees are demonstrated to have, on average, for a variety of common learning tasks, higher predictive accuracy than the less complex original decision trees. This result raises considerable doubt about the utility of Occam's razor as it is commonly applied in modern machine learning.

 
• Introduction
• Previous Theoretical and Experimental Work
  • The Law of Conservation of Generalization Performance
  • Other Theoretical Objections to the Occam Thesis
  • Previous Experimental Evidence Against the Occam Thesis
  • Theoretical and Experimental Support for the Occam Thesis
• New Experimental Evidence Against the Occam Thesis
  • Theoretical Basis for the Decision Tree Post-processor
  • The Post-processor
  • Evaluation
• Discussion
  • Directions for Future Research
  • Other Related Research
  • Bias Versus Variance
  • Minimum Encoding Length Induction
  • Appropriate Application of Grafting and Pruning
• Conclusion
• Acknowlegements
• References
• About this document ...