Communication:
Slack will be used for discussion about the course and assignments. The link will be emailed to the students.
Image generated by chatgpt, including typos.
Course Description
This course is aimed to help students with a neuroscience background as well as those with a computational background to form a solid basis in Computational Cognitive Neuroscience research. The focus will be on acquiring the skills to both build models of brain activity and understand the implications of these models.
This PhD-level course explores the intersection of Neuroscience and AI. It evaluates the use of new AI methods as models for the brain, and the utility of inspirations from the brain for building better AI. The course has two main components: (a) a critical look at the current practice of computational neuroscience through lectures, readings and discussions and (b) an active, practical component (though homework and projects) that allows students to form their opinions based not only on readings and ideas imposed upon them, but also through their own experience and understanding. Specifically, the class has a project component, which is expected to be substantial enough that the work could lead to a publication if successful.
The focus of the class will be on representations in the brain and in AI models, specifically in the areas of language and vision (and their intersection). Students will learn the most recent approaches for using AI models as models of the brain, along with the arguments for and against these types of approaches. The course will also touch upon generation in both AI models and in the brain, and discuss possibilities of integrating them.
Learning Objectives:
The aim of this class is to enable students to:
know the state of the art in modeling brain responses using AI and key findings,
identify key fallacies in modeling and ways to avoid them,
form computational cognitive neuroscience research questions and a plan to address them,
have practical experience in modeling brain responses.
Prerequisites
Intermediate Statistics and/or machine learning, working knowledge of AI models, basic knowledge in cognitive neuroscience. For students who do not have the cognitive neuroscience background, readings will be provided to be covered at the beginning of the semester. The official requirement is having taken one of the machine learning classes (10301 or 10315 or 10601 or 10701 or 10715 or 10707 or 10417 or 10617 or 10414 or 10714). If you believe you have the necessary background from other coursework, please contact the instructor.
Resources
Some of the readings we will cover:
Mitchell, T.M., Shinkareva, S.V., Carlson, A., Chang, K.M., Malave, V.L., Mason, R.A. and Just, M.A., 2008. Predicting human brain activity associated with the meanings of nouns. science, 320(5880), pp.1191-1195.
Nishimoto, S., Vu, A.T., Naselaris, T., Benjamini, Y., Yu, B. and Gallant, J.L., 2011. Reconstructing visual experiences from brain activity evoked by natural movies. Current biology, 21(19), pp.1641-1646.
Wang, J., Miller, K., Marblestone, A., 2020, Where Neuroscience meets AI (And What’s in Store for the Future), Neurips Tutorial 2020.
Naselaris, T., Kay, K.N., Nishimoto, S. and Gallant, J.L., 2011. Encoding and decoding in fMRI. Neuroimage, 56(2), pp.400-410.
Kriegeskorte, N., Mur, M. and Bandettini, P.A., 2008. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, p.4.
Kornblith, S., Norouzi, M., Lee, H. and Hinton, G., 2019, May. Similarity of neural network representations revisited. In International conference on machine learning (pp. 3519-3529). PMLR.
Wehbe, L., Vaswani, A., Knight, K., and Mitchell, T., 2014. Aligning context-based statistical models of language with brain activity during reading. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 233-243.
Yamins, D.L. and DiCarlo, J.J., 2016. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3), pp.356-365.
Schrimpf, M., Kubilius, J., Hong, H., Majaj, N.J., Rajalingham, R., Issa, E.B., Kar, K., Bashivan, P., Prescott-Roy, J., Geiger, F. and Schmidt, K., 2018. Brain-score: Which artificial neural network for object recognition is most brain-like?. BioRxiv, p.407007.
Huth, A.G., De Heer, W.A., Griffiths, T.L., Theunissen, F.E. and Gallant, J.L., 2016. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600), pp.453-458.
Guest, O. and Martin, A.E., 2023. On logical inference over brains, behaviour, and artificial neural networks. Computational Brain & Behavior, pp.1-15.
Schrimpf, M., Blank, I.A., Tuckute, G., Kauf, C., Hosseini, E.A., Kanwisher, N., Tenenbaum, J.B. and Fedorenko, E., 2021. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45), p.e2105646118.
St-Yves, G., Allen, E.J., Wu, Y., Kay, K. and Naselaris, T., 2023. Brain-optimized deep neural network models of human visual areas learn non-hierarchical representations. Nature communications, 14(1), p.3329.
Toneva, M., Mitchell, T.M. and Wehbe, L., 2022. Combining computational controls with natural text reveals aspects of meaning composition. Nature computational science, 2(11), pp.745-757.
Reddy, A.J. and Wehbe, L., 2020. Syntactic representations in the human brain: beyond effort-based metrics. BioRxiv, pp.2020-06.
Makin, J.G., Moses, D.A. and Chang, E.F., 2020. Machine translation of cortical activity to text with an encoder–decoder framework. Nature neuroscience, 23(4), pp.575-582.
Willett, F.R., Kunz, E.M., Fan, C., Avansino, D.T., Wilson, G.H., Choi, E.Y., Kamdar, F., Glasser, M.F., Hochberg, L.R., Druckmann, S. and Shenoy, K.V., 2023. A high-performance speech neuroprosthesis. Nature, pp.1-6.
Cross, L., Cockburn, J., Yue, Y. and O’Doherty, J.P., 2021. Using deep reinforcement learning to reveal how the brain encodes abstract state-space representations in high-dimensional environments. Neuron, 109(4), pp.724-738.
Francl, A. and McDermott, J.H., 2022. Deep neural network models of sound localization reveal how perception is adapted to real-world environments. Nature human behaviour, 6(1), pp.111-133.
Tang, J., LeBel, A., Jain, S. and Huth, A.G., 2023. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience, pp.1-9.
Caucheteux, C., Gramfort, A. and King, J.R., 2023. Evidence of a predictive coding hierarchy in the human brain listening to speech. Nature human behaviour, 7(3), pp.430-441.
Schedule
Tentative schedule, might change according to class progress and interest.
Grading
The course is a 12 unit class. The graded components are as follows: (these percentages might change a bit before the start of the Spring semester)
25% reading and participation. Each week, a paper will be assigned and the class will begin with a short quiz on that paper (the 10 best scores will be counted towards the grade). Further, participation will be a large part of the grade.
25% Homework.
50% class project. The project is expected to target a new scientific problem or target an existing problem in a new way. Success in evaluating the project hypothesis is not necessary for a good project. The final deliverable will be a NeurIPS style paper and a class presentation.
Course Policies
Attendance
This course is a discussion-based course and therefore attendance is essential for benefiting from it and contributing to it. Attendance will not be directly graded but participation will. We understand that some events such as conferences will lead to absence. Please communicate your absence in advance to the course staff. There will be some allowance in the grade for a few absences.
Collaboration
Discussion of class material is heavily encouraged. Collaboration in homework assignments is allowed as long as it's properly reported. Project collaboration is expected (projects are done in groups), and is allowed across groups as well.
Academic Integrity
We have a zero tolerance policy for violation of class policies. If you are in any doubt in regards to the policy, please clarify with the course staff before proceeding.
Late homework policy
The maximum earnable points for each assignment will drop by 20% per late day.
Take care of yourself
Take care of yourself. Do your best to maintain a healthy lifestyle this semester by eating well,
exercising, avoiding drugs and alcohol, getting enough sleep and taking some time to relax. This
will help you achieve your goals and cope with stress.
All of us benefit from support during times of struggle. There are many helpful resources
available on campus and an important part of the college experience is learning how to ask for
help. Asking for support sooner rather than later is almost always helpful.
If you or anyone you know experiences any academic stress, difficult life events, or feelings like
anxiety or depression, we strongly encourage you to seek support. Counseling and
Psychological Services (CaPS) is here to help: call 412-268-2922 and visit their website
at http://www.cmu.edu/counseling/. Consider reaching out to a friend, faculty or family
member you trust for help getting connected to the support that can help.
If you or someone you know is feeling suicidal or in danger of self-harm, call someone
immediately, day or night:
CaPS: 412-268-2922
Re:solve Crisis Network: 888-796-8226
If the situation is life threatening, call the police
