15-486/782: Artificial Neural Networks Dave Touretzky Fall 2006 - Course Syllabus Last modified: Fri Dec 1 04:18:23 EST 2006

Monday, August 28. Organizational meeting; introduction to neural nets. [pdf]

• Hertz, Krogh & Palmer, chapter 1.

Wednesday, August 30. Perceptrons and the LMS Algorithm. [pdf]

• Hertz, Krogh & Palmer, sections 5.1 through 5.4.

• Optional enrichment: Anderson, J. A., and Rosenfeld, E. (1998) Talking Nets: An Oral History of Neural Networks, chapter 3, by Bernard Widrow. Cambridge, MA: MIT Press. Available online at CogNet.

Problem set 1: learning with linear units.

Monday, Sep. 4. No class. Labor day.

Wednesday, Sep. 6. Backpropagation Learning. [pdf]

• Hertz, Krogh & Palmer sections 5.5, and 6.1-6.3.

• Handout: Derivation of the backprop learning rule

• Optional enrichment: Rumelhart, D. E., Hinton, G. E, and Williams, R. J. (1986) Learning internal representations by error propagation. In D. E. Rumelhart and J. L. McClelland (eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition, chapter 8. Cambridge, MA: MIT Press. (local copy)

Problem set 1 due.

Problem set 2: backpropagation learning.

Monday, Sep. 11. Visually-Guided Robot Control. [pdf] Note: class now meets in NSH 3002

• Pomerleau, D.A. (1991) Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88-97.

• Pomerleau, D.A. (1994) Neural network based vision for precise control of a walking robot. Machine Learning, 15(2):125-135.

Wednesday, Sep. 13. Optimization Techniques. [pdf]

• Bishop, C. M. (1995) Neural Networks for Pattern Recognition, chapter 7. Oxford University Press.

Problem set 2 due.

Problem set 3: ALVINN.

Monday, Sep. 18. Overfitting, Cross-Validation, and Early Stopping. [pdf]

• Caruana, R., Lawrence, S., and Giles, C. L. (2001) Overfitting in neural networks trained with backpropagation, conjugate gradient, and early stopping. In T. K. Leen, T. G. Dietterich, and V. Tresp (eds.), Advances in Neural Information Processing Systems 13. MIT Press.

• Weigend, A. S. (1993) On overfitting and the effective number of hidden units. In M. Mozer, P. Smolensky, D. Touretzky, J. Elman, and A. Weigend (eds.), Proceedings of the 1993 Connectionist Models Summer School, pp. 335-342.

Wednesday, Sep. 20. Simple Recurrent Networks. [pdf]

• Hertz, Krogh & Palmer, section 7.3 up to the middle of p. 184.

• Elman, J. L. (1991) Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning 7(2,3):195-225.

Monday, Sep. 25. Language Processing Models. [pdf]

• Hinton, G. E., Plaut, D. C., and Shallice, T. (1993) Simulating brain damage. Scientifi American 269:76-82.

Wednesday, Sep. 27. Pattern Classification I. [pdf]

• Bishop, C. M. (1995) Neural Networks for Pattern Recognition, chapter 1. Oxford University Press.

Problem set 3 due.

Monday, Oct. 2. Pattern Classification II. [pdf]

• Bishop, C. M. (1995) Neural Networks for Pattern Recognition, chapter 2, sections 2.1 through 2.5. Oxford University Press.

Wednesday, Oct. 4. Radial Basis Functions. [pdf]

• Moody, J.. and Darken, C. (1989) Fast learning in networks of locally-tuned processing units. Neural Computation 1(2):281-294.

• Note: radial basis functions are also covered briefly in Hertz, Krogh, & Palmer section 9.7.

• Optional enrichment: Kruschke, J. K. (1992) ALCOVE: An exmplar-based model of category learning. Psychological Review 99(1):22-44.

Monday, Oct. 9. The EM (Expectation-Maximization) Algorithm. [pdf]

• Bishop, chapter 2, just section 2.6.

• Optional enrichment: Ueda, N., Nakano, R., Ghahramani, Z., and Hinton, G. E. (2000) SMEM algorithms for mixture models. Neural Computation 12:2109-2128.

• Optional enrichment: Williamson, J. R. (1997) A constructive, incremental-learning network for mixture modeling and classification. Neural Computation 9(7):1517-1543.

Wednesday, Oct. 11. Neural Networks for Control. [pdf]

• Jordan, M. I., and Rumelhart, D. E. (1992) Forward models: supervised learning with a distal teacher. Cognitive Science 16(3):307-354.

• Optional enrichment: Nguyen, D., and Widrow, B. (1990) The truck backer-upper: an example of self-learning in neural networks. In W. T. Miller III, R. S. Sutton, and P. J. Werbos (Eds.), Neural Networks for Control, ch. 12, pp. 287-299. Cambridge, MA: MIT Press.

Monday, Oct. 16. Support Vector Machines. [Guest lecturer: Mark Fuhs] [PowerPoint]

• Haykin, S. (1999) Neural Networks: A Comprehensive Foundation, 2nd edition. Chapter 6, Support Vector Machines, sections 6.1-6.4 and 6.9. Prentice-Hall.

Wednesday, October 18. Midterm Exam.

Monday, Oct. 23. Time Series Prediction. [pdf]

• Fessant, F., Bengio, S., and Collobert, D. (1995) On the prediction of solar activity using different neural network models. Annales Geophysicae.

Wednesday, Oct. 25. Shared Weight Networks.

• LeCun, Y., Bottou, L, Bengio, Y., and Haffner, P. (1998) Gradient-based learning applied to document recognition, sections I through III. Proceedings of the IEEE, November 1998.

• LeNet-5 demos by Yann LeCun, at http://yann.lecun.com/exdb/lenet/index.html.

Problem set 4: Robot arm control.

Monday, Oct. 30. Competitive Learning and Kohonen Nets. [pdf]

• Hertz, Krogh & Palmer, chapter 9.

• Competitive learning demos at Ruhr University Bochum.

Wednesday, Nov. 1. Hebbian Learning and Principal Components Analysis. [pdf]

• Hertz, Krogh & Palmer, chapter 8.

• Tony Bell's Independent Components Analysis web page, at http://www.cnl.salk.edu/~tony/ica.html.

Problem set 4 due.

Monday, Nov. 6. Hopfield Nets and Boltzmann Machines. [pdf]

• Hertz, Krogh & Palmer, chapter 2 and section 7.1.

• Optional enrichment: Kirkpatrick, S., Gelatt, C. D., Jr., and Vecchi, M. P. (1983) Optimization by simulated annealing. Science 220:671-680.

Problem set 5: Hopfield networks.

Wednesday, Nov. 8. Mean Field Approximation [pdf]

• Hopfield, J. J. (1984) Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Sciences, 81:3088-3092.

• Hopfield, J. J., and Tank, D. W. (1985) ``Neural'' computation of decisions in optimization problems. Biological Cybernetics, 52:141-152.

Monday, Nov. 13. Helmholtz Machines; Minimum Description Length. [pdf]

• Dayan, P., Hinton, G. E., Neal, R. M., and Zemel, R. S. (1995) The Helmholtz machine. Neural Computation, 7(5):889-904.

• Hinton, G. E., Dayan, P. Frey, B. J., and Neal, R. M. (1995) The wake-sleep algorithm for unsupervised neural networks. Science, 268:1158-1160.

• Brief tutorial on information theory, by Dave Touretzky.

Problem set 5 due.

Wednesday, Nov. 15. Bayesian Networks. [pdf]

• Bishop, chapter 10, sections 10.1, 10.2, and 10.6.

Monday, Nov. 20. Computational Learning Theory. [pdf]

• Kearns, M. J., and Vazirani, U. V. (1994) An introduction to Computational Learning Theory, chapter 1. Cambridge, MA: MIT Press.

Wednesday, Nov. 22. No class. Thanksgiving Holiday.

Monday, Nov. 27. Connectionist Symbol Processing. [pdf]

• Plate, T. (1998) Analogy retrieval and processing with distributed representations. Technical report CS-TR-98-4, Victoria University of Wellington, Computer Science.

Wednesday, Nov. 29. Reinforcement Learning. [pdf]

• Tesauro, G. (2002) Programming backgammon using self-teaching neural nets. Artificial Intelligence 134(1):181-199.

Monday, Dec. 4. Neurophysiology for Computer Scientists. [Guest lecturer: Mark Fuhs]

• Churchland, P.S. (1986) Neurophilosophy: Toward a Unified Science of the Mind-Brain. Chapter 2, Modern Theory of Neurons, pp. 38-69 only. Cambridge, MA: MIT Press.

• Matlab demo: HHSim: Hodgkin-Huxley simulator.

Wednesday, Dec. 6. No class.

Tuesday, Dec. 12. Final exam, 1-4 pm, Wean Hall 5403.