Monday, Jan. 12. Organizational meeting; introduction to neural nets. [ps, pdf]

• Hertz, Krogh & Palmer, chapter 1.

Wednesday, Jan. 14. Perceptrons and the LMS Algorithm. [ps, pdf]

• Hertz, Krogh & Palmer, chapter 5.

• Optional: Anderson, J. A., and Rosenfeld, E. (1998) Talking Nets: An Oral History of Neural Networks, chapter 3. Cambridge, MA: MIT Press.

Problem set 1: learning with linear units.

Monday, Jan. 19. No class. Martin Luther King's birthday.

Wednesday, Jan. 21. Pattern Recognition I. [ps, pdf]

Problem set 1 due.

• Bishop, C. M. (1995) Neural Networks for Pattern Recognition, chapter 1. Oxford University Press.

Monday, Jan. 26. Pattern Recognition II. [ps, pdf]

• Bishop, C. M. (1995) Neural Networks for Pattern Recognition, chapter 2, sections 2.1 through 2.5. Oxford University Press.

Wednesday, Jan. 28. Backpropagation Learning. [ps, pdf]

• Hertz, Krogh & Palmer sections 5.4, and 6.1-6.3.

• Optional: Rumelhart, D. E., Hinton, G. E, and Williams, R. J. (1986) Learning internal representations by error propagation. In D. E. Rumelhart and J. L. McClelland (eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition, chapter 8. Cambridge, MA: MIT Press.

Problem set 2: backpropagation learning.

Monday, Feb. 2. Visually-Guided Robot Control. [ps, pdf]

• Pomerleau, D.A. (1991) Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88-97.

• Pomerleau, D.A. (1994) Neural network based vision for precise control of a walking robot. Machine Learning, 15(2):125-135.

Wednesday, Feb. 4. Optimization Techniques. [Kornel Laskowski] [ps, pdf]

• Bishop, C. M. (1995) Neural Networks for Pattern Recognition, chapter 7. Oxford University Press.

Problem set 2 due.

Problem set 3: ALVINN.

Monday, Feb. 9. Overfitting and Early Stopping. [ps, pdf]

• Caruana, R., Lawrence, S., and Giles, C. L. (2001) Overfitting in neural networks trained with backpropagation, conjugate gradient, and early stopping. In T. K. Leen, T. G. Dietterich, and V. Tresp (eds.), Advances in Neural Information Processing Systems 13. MIT Press.

• Weigend, A. S. (1993) On overfitting and the effective number of hidden units. In M. Mozer, P. Smolensky, D. Touretzky, J. Elman, and A. Weigend (eds.), Proceedings of the 1993 Connectionist Models Summer School, pp. 335-342.

Wednesday, Feb. 11. Recurrent Backprop Networks.

• Hertz, Krogh & Palmer, section 7.3 up to the middle of p. 184.

• Elman, J. L. (1991) Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning 7(2,3):195-225.

Monday, Feb. 16. No class (President's Day).

Wednesday, Feb. 18. Neural Networks for Control. [ps, pdf]

Problem set 3 due.

• Nguyen, D., and Widrow, B. (1990) The truck backer-upper: an example of self-learning in neural networks. In W. T. Miller III, R. S. Sutton, and P. J. Werbos (Eds.), Neural Networks for Control, ch. 12, pp. 287-299. Cambridge, MA: MIT Press.

• Jordan, M. I., and Rumelhart, D. E. (1992) Forward models: supervised learning with a distal teacher. Cognitive Science 16(3):307-354.

Monday, Feb. 23. Shared-Weight Networks.

• Lang, K., Waibel, A. and Hinton, G. E. (1990) A Time-Delay Neural Network Architecture for Isolated Word Recognition. Neural Networks 3:23-43.

• LeCun, Y, Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. (1989) Backpropagation applied to handwritten zip code recognition. Neural Computation 1(4):541-551.

Wednesday, Feb. 25. Time Series Prediction. [Kornel Laskowski] [ps, pdf]

• Fessant, F., Bengio, S., and Collobert, D. (1995) On the prediction of solar activity using different neural network models. Annales Geophysicae.

Monday, March 1. Midterm Exam.

Wednesday, March 3. Radial Basis Functions. [ps, pdf]

• Moody, J.. and Darken, C. (1989) Fast learning in networks of locally-tuned processing units. Neural Computation 1(2):281-294.

• Note: radial basis functions are also covered briefly in Hertz, Krogh, & Palmer section 9.7.

Monday, March 8. Spring break. No class.

Wednesday, March 10. Spring break. No class.

Monday, March 15. Object Recognition with Radial Basis Functions. [ps, pdf]

• Duvdevani-Bar, S., and Edelman, S. (1999) Visual recognition and categorization on the basis of similarities to multiple class prototypes. International Journal of Computer Vision, 33:201-228.

Wednesday, March 17. Competitive Learning and Kohonen Nets. [ps, pdf]

• Hertz, Krogh & Palmer, chapter 9.

• Competitive learning demos at Ruhr University Bochum.

• Optional reading: Rumelhart, D. E., and Zipser, D. (1986) Feature discovery by competitive learning. In D. E. Rumelhart and J. L. McClelland (eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1, chapter 5. Cambridge, MA: MIT Press.

Problem set 4: learning with a distal teacher.

Monday, March 22. The EM (Expectation-Maximization) Algorithm. [ps, pdf]

• Bishop, chapter 2, just section 2.6.

• Williamson, J. R. (1997) A constructive, incremental-learning network for mixture modeling and classification. Neural Computation 9(7):1517-1543.

Wednesday, March 24. Hebbian Learning and Principal Components Analysis. [Kornel Laskowski] [ps, pdf]

• Hertz, Krogh & Palmer, chapter 8.

• Tony Bell's Independent Components Analysis web page, at http://www.cnl.salk.edu/~tony/ica.html.

Monday, March 29. Hopfield Nets and Boltzmann Machines. [ps, pdf]

• Hertz, Krogh & Palmer, chapter 2 and section 7.1.

Problem set 4 due.

Wednesday, March 31. Boltzmann Machines and Mean Field Approximation

• Hopfield, J. J. (1984) Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Sciences, 81:3088-3092.

• Hopfield, J. J., and Tank, D. W. (1985) ``Neural'' computation of decisions in optimization problems. Biological Cybernetics, 52:141-152.

Monday, April 5. Helmholtz Machines; Minimum Description Length.

• Hinton, G. E., Dayan, P. Frey, B. J., and Neal, R. M. (1995) The wake-sleep algorithm for unsupervised neural networks. Science, 268:1158-1160.

• Dayan, P., Hinton, G. E., Neal, R. M., and Zemel, R. S. (1995) The Helmholtz machine. Neural Computation, 7(5):889-904.

• Brief tutorial on information theory, by Dave Touretzky.

Wednesday, April 7. Bayesian Networks. [ps, pdf]

• Bishop, chapter 10.

• Lecture slides (Postscript)

Monday, April 12. Computational Learning Theory. [ps, pdf]

• Kearns, M. J., and Vazirani, U. V. (1994) An introduction to Computational Learning Theory, chapter 1. Cambridge, MA: MIT Press.

Problem set 5: Hopfield and Boltzmann networks.

Wednesday, April 14. Recursive Structures 1: Backprop.

• Pollack, J. B. (1990) Recursive distributed representations. Artificial Intelligence 46(1):77-105.

Monday, April 19. Recursive Structures 2: Convolutional Approaches. [ps, pdf]

• Plate, T. (1998) Analogy retrieval and processing with distributed representations. Technical report CS-TR-98-4, Victoria University of Wellington, Computer Science.

• Touretzky, D. S. (in press) Connectionist and symbolic representations. In M. A. Arbib (Ed.), Handbook of Brain Theory and Neural Networks, 2nd edition. Cambridge, MA: MIT Press.

Problem set 5 due.

Problem set 6: digit recognition.

Wednesday, April 21. Attractor Bump Models.

• Seung, H. S. (1996) How the brain keeps the eyes still. Proceedings of the National Academy of Sciences, 93:13339-13344.

Monday, April 26. Neurophysiology for Computer Scientists.

• Churchland, P. S. (1986) Neurophilosphy: Toward a Unified Science of the Mind-Brain, chapter 2, Modern Theory of Neurons (pp. 48-77.) MIT Press.

  HHsim: Hodgkin-Huxley simulator

Wednesday, April 28. The Mammalian Visual System.

• Van Essen, D. C. and Anderson, C. H. (1990) Information processing strategies and pathways in the primate retina and visual cortex. In S. F. Zornetzer, J. L. Davis, and C. Lau (Eds.), An Introduction to Neural and Electronic Networks, pp. 43-72. San Diego: Academic Press.

• Van Essen, D. C., and DeYoe, E. A. (1995) Concurrent processing in the primate visual cortex. In M. S. Gazzaniga (Ed.), The Cognitive Neurosciences, pp. 383-400. MIT Press.

Problem set 6 due.

Monday, May 3. Open book/open notes Final Exam.

1:00 pm to 4:00 pm in Wean Hall 7500.