Zhiqing Sun
孙之清

Research Scientist
at OpenAI
PhD Candidate at
CMU -> SCS -> LTI
Hey there, welcome!
I am a final-year Ph.D. candidate at CMU LTI, advised by Prof. Yiming Yang. My research is generously supported by the Google PhD Fellowship in Natural Language Processing (2023) and the OpenAI Superalignment Fast Grants (2024). I received my B.S. in Computer Science from Peking University.
Update (Feb 2025): I’ve joined OpenAI, where I trained the LLM that powers Deep Research, our latest AI agent that Sam estimates can do “a single-digit percentage of all economically valuable tasks in the world.”
Besides Deep Research, my recent research interests include scalable reasoning beyond human supervision, scalable alignment, and scalable training of more capable and trustworthy AI agents.
Note: The following content may be outdated.
Research Interests
I am generally interested in machine learning and artificial intelligence. My recent research focuses on scalable alignment of foundation models. I am particularly interested in enhancing the reliability of foundation models, including large language models (LLMs) and large multimodal models (LMMs), through minimal human supervision and scalable oversight. This can be achieved using human-defined principles, factual feedback from real-world interactions, or easy-to-hard generalization. A few of my recent projects include:
- Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision: Guided by the observation that evaluation is easier than generation, we enabled large language models to excel on hard math problems beyond human evaluation capabilities through the easy-to-hard generalization of evaluators (e.g., process reward models).
- SALMON: Self-Alignment with Instructable Reward Models: We developed an Instructable Reward Model that helps RLAIF fully replace RLHF to align language models from scratch (enhancing both their alignment and capabilities)!
- Aligning Large Multimodal Models with Factually Augmented RLHF: We proposed Factually Augmented RLHF (Fact-RLHF) that augments the reward model with additional factual information to alleviate the reward hacking phenomenon in RLHF.
News
- Apr. 2024: Received the OpenAI Superalignment Fast Grants ($100,000) to support our research on easy-to-hard generalization.
- Feb. 2024: Gave an invited talk at UIUC on scalable alignment.
- Jan. 2024: TAing for 11-741 Machine Learning with Graphs.
- Nov. 2023: Gave an invited talk at Caltech on scalable alignment, hosted by Prof. Yisong Yue
- Oct. 2023: Selected as the 2023 Rising Stars in Data Science and gave a talk on scalable alginment at the Rising Stars workshop in UChicago.
- Sept. 2023: Received the Microsoft Accelerate Foundation Models Research (AFMR) Initiative ($20,000).
- Sept. 2023: Received the Google PhD Fellowship in Natural Language Processing.
Education
Language Technologies Institute, Carnegie Mellon University
- Aug. 2019 - Present, M.S. / Ph.D. in Language Technologies
School of Electrical Engineering & Computer Science (EECS), Peking University
- Sept. 2015 - July 2019, B.S. in Computer Science (Summa Cum Laude)
Selected Publications
For a more complete list or preprints, see the publications page, or my google scholar page.
(*=equal contribution)
2024
2023
(Internship) Experience
- Allen Institute for Artificial Intelligence (AI2), Spring 2024
- MIT-IBM Watson AI Lab, Summer 2023
- Google Brain, Summer 2022
- Google Brain, Summer 2019
- Microsoft Research Asia, Spring 2019
- Mila & University of Montreal , Summer 2018