|
|
Jun-Yan Zhu Assistant Professor Email: junyanz at cs dot cmu dot edu |
|
|
Students | CV | Google Scholar | GitHub | arXiv |
|
|
|
||||||||
|
I am an Assistant Professor with The Robotics Institute in the School of Computer Science of Carnegie Mellon University. I also hold affiliated faculty appointments in the Computer Science Department and Machine Learning Department. I study computer vision, graphics, computational photography, and generative models. Prior to joining CMU, I was a Research Scientist at Adobe Research. I did a postdoc at MIT CSAIL, working with William T. Freeman, Josh Tenenbaum, and Antonio Torralba. I obtained my Ph.D. from UC Berkeley, under the supervision of Alexei A. Efros. I received my B.E. from Tsinghua University, working with Zhuowen Tu, Shi-Min Hu, and Eric Chang. |
Code & Events
Generative Intelligence Lab
|
Our lab studies the collaboration between Human Creators and Generative Models, with the goal of building intelligent machines capable of helping everyone tell their visual stories. We are studying the following questions:
Our lab is part of Carnegie Mellon Graphics Lab and Carnegie Mellon Computer Vision Group. |
Kangle Deng
Ruihan Gao
Maxwell Jones
Nupur Kumari
Beijia Lu
Gaurav Parmar
Ava Pun
Grace Su
Sheng-Yu Wang|
Former members and visitors: Sean Liu (Postdoc, now Research Scientist at Autodesk Research), Richa Mishra (MSCV, now at HeyGen), Aniruddha Mahapatra (MSCV, now at Adobe), Or Patashnik (Visiting PhD from TAU), Songwei Ge (Visiting PhD from UMD), Chonghyuk (Andrew) Song (MSR, now PhD student at MIT), Muyang Li (MSR, now PhD student at MIT), Daohan (Fred) Lu (MSCV, now PhD student at NYU), Mia Tang (Undergrad, now MS student at Stanford), Bingliang Zhang (Undergrad, now PhD student at Caltech), Rohan Agarwal (MSCV, now at Runway ML), George Cazenavette (MSR, now PhD student at MIT) |
Teaching
Software
|
Landscape Mixer Photoshop 2022's Landscape Mixer can transform landscape images in various ways. This feature is based on our work Swapping Autoencoder (NeurIPS 2020).
|
|
NVIDIA Canvas: Turn Simple Brushstrokes into Realistic Images Download Windows 10 app based on our work SPADE (CVPR 2019) and GauGAN demo (SIGGRAPH 2019).
|
|
Photoshop Neural Filters Photoshop 2021 introduces "Neural Filters". Several features are partly built on our work iGAN (ECCV 2016), ideepcolor (SIGGRAPH 2017), and CycleGAN (ICCV 2017).
|
Selected Publications
See the full list on Google Scholar
|
Generative Photomontage Sean J. Liu, Nupur Kumari, Ariel Shamir, Jun-Yan Zhu arXiv 2024
|
|
SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models Muyang Li*, Yujun Lin*, Zhekai Zhang*, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun-Yan Zhu, Song Han arXiv 2024
|
|
Tactile DreamFusion: Exploiting Tactile Sensing for 3D Generation Ruihan Gao, Kangle Deng, Gengshan Yang, Wenzhen Yuan, Jun-Yan Zhu NeurIPS 2024
|
|
Data Attribution for Text-to-Image Models by Unlearning Synthesized Images Sheng-Yu Wang, Aaron Hertzmann, Alexei A. Efros, Jun-Yan Zhu, Richard Zhang NeurIPS 2024
|
|
Customizing Text-to-Image Diffusion with Camera Viewpoint Control Nupur Kumari*, Grace Su*, Richard Zhang, Taesung Park, Eli Shechtman, Jun-Yan Zhu SIGGRAPH Asia 2024
|
|
Customizing Text-to-Image Models with a Single Image Pair Maxwell Jones, Nupur Kumari, Sheng-Yu Wang, David Bau, Jun-Yan Zhu SIGGRAPH Asia 2024
|
|
Consolidating Attention Features for Multi-view Image Editing Or Patashnik, Rinon Gal, Daniel Cohen-Or, Jun-Yan Zhu, Fernando De la Torre SIGGRAPH Asia 2024
|
|
Distilling Diffusion Models into Conditional GANs Minguk Kang, Richard Zhang, Connelly Barnes, Sylvain Paris, Jaesik Park, Suha Kwak, Eli Shechtman, Jun-Yan Zhu, Taesung Park ECCV 2024
Project | E-LatentLPIPS code | Paper |
BibTex
|
|
FlashTex: Fast Relightable Mesh Texturing with LightControlNet Kangle Deng, Timothy Omernick, Alexander Weiss, Deva Ramanan, Jun-Yan Zhu, Tinghui Zhou, Maneesh Agrawala ECCV 2024
|
|
CoFRIDA: Self-Supervised Fine-Tuning for Human-Robot Co-Painting Peter Schaldenbrand, Gaurav Parmar, Jun-Yan Zhu, James McCann, Jean Oh ICRA 2024 (Best Paper on Human-Robot Interaction) ICRA EXPO 2024 Best Demo Award Finalist
|
|
On the Content Bias in Fréchet Video Distance Songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Jun-Yan Zhu, Jia-Bin Huang CVPR 2024 Download: pip install cd-fvd
|
|
Content-Based Search for Deep Generative Models Daohan Lu*, Sheng-Yu Wang*, Nupur Kumari*, Rohan Agarwal*, Mia Tang, David Bau, Jun-Yan Zhu SIGGRAPH Asia 2023
|
|
Text-Guided Synthesis of Eulerian Cinemagraphs Aniruddha Mahapatra, Aliaksandr Siarohin, Hsin-Ying Lee, Sergey Tulyakov, Jun-Yan Zhu SIGGRAPH Asia 2023
|
|
Evaluating Data Attribution for Text-to-Image Models Sheng-Yu Wang, Alexei A. Efros, Jun-Yan Zhu, Richard Zhang ICCV 2023
|
|
Ablating Concepts in Text-to-Image Diffusion Models Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, Jun-Yan Zhu ICCV 2023
|
|
Controllable Visual-Tactile Synthesis Ruihan Gao, Wenzhen Yuan, Jun-Yan Zhu ICCV 2023
|
|
Expressive Text-to-Image Generation with Rich Text Songwei Ge, Taesung Park, Jun-Yan Zhu, Jia-Bin Huang ICCV 2023
|
|
Total-Recon: Deformable Scene Reconstruction for Embodied View Synthesis Chonghyuk Song, Gengshan Yang, Kangle Deng, Jun-Yan Zhu, Deva Ramanan ICCV 2023
|
|
3D-aware Blending with Generative NeRFs Hyunsu Kim, Gayoung Lee, Yunjey Choi, Jin-Hwa Kim, Jun-Yan Zhu ICCV 2023
|
|
Dense Text-to-Image Generation with Attention Modulation Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, Jun-Yan Zhu ICCV 2023
|
|
Holistic Evaluation of Text-To-Image Models Tony Lee*, Michihiro Yasunaga*, Chenlin Meng*, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Benita Teufel, Marco Bellagente, Minguk Kang, Taesung Park, Jure Leskovec, Jun-Yan Zhu, Li Fei-Fei, Jiajun Wu, Stefano Ermon, Percy Liang NeurIPS 2023
|
|
Zero-shot Image-to-Image Translation Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, Jun-Yan Zhu SIGGRAPH 2023
|
|
Scaling up GANs for Text-to-Image Synthesis Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, Taesung Park CVPR 2023
|
|
3D-aware Conditional Image Synthesis Kangle Deng, Gengshan Yang, Deva Ramanan, Jun-Yan Zhu CVPR 2023
|
|
Multi-Concept Customization of Text-to-Image Diffusion Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, Jun-Yan Zhu CVPR 2023
|
|
Generalizing Dataset Distillation via Deep Generative Prior George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, Jun-Yan Zhu CVPR 2023
|
|
Domain Expansion of Image Generators Yotam Nitzan, Michaël Gharbi, Richard Zhang, Jun-Yan Zhu, Daniel Cohen-Or, Eli Shechtman CVPR 2023
|
|
Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models Muyang Li, Ji Lin, Chenlin Meng, Stefano Ermon, Song Han, Jun-Yan Zhu NeurIPS 2022 | TPAMI 2023
|
|
Rewriting Geometric Rules of a GAN Sheng-Yu Wang, David Bau, Jun-Yan Zhu SIGGRAPH 2022
|
|
GAN-Supervised Dense Visual Alignment William Peebles, Jun-Yan Zhu, Richard Zhang, Antonio Torralba, Alexei A. Efros, Eli Shechtman CVPR 2022 (Best Paper Finalist)
|
|
Ensembling Off-the-shelf Models for GAN Training Nupur Kumari, Richard Zhang, Eli Shechtman, Jun-Yan Zhu CVPR 2022 Installation: pip install vision-aided-loss
|
|
On Aliased Resizing and Surprising Subtleties in GAN Evaluation Gaurav Parmar, Richard Zhang, Jun-Yan Zhu CVPR 2022 Installation: pip install clean-fid
|
|
Dataset Distillation by Matching Training Trajectories George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, Jun-Yan Zhu CVPR 2022
|
|
Depth-supervised NeRF: Fewer Views and Faster Training for Free Kangle Deng, Andrew Liu, Jun-Yan Zhu, Deva Ramanan CVPR 2022
|
|
Spatially-Adaptive Multilayer Selection for GAN Inversion and Editing Gaurav Parmar, Yijun Li, Jingwan Lu, Richard Zhang, Jun-Yan Zhu, Krishna Kumar Singh CVPR 2022
|
|
SDEdit: Image Synthesis and Editing with Stochastic Differential Equations Chenlin Meng, Yutong He, Song Yang, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, Stefano Ermon ICLR 2022
|
|
Sketch Your Own GAN Sheng-Yu Wang, David Bau, Jun-Yan Zhu ICCV 2021
|
|
Editing Conditional Radiance Fields Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, Bryan Russell ICCV 2021
|
|
Ensembling with Deep Generative Views Lucy Chai, Jun-Yan Zhu, Eli Shechtman, Phillip Isola, Richard Zhang CVPR 2021
|
|
Anycost GANs for Interactive Image Synthesis and Editing Ji Lin, Richard Zhang, Frieder Ganz, Song Han, Jun-Yan Zhu CVPR 2021
|
|
GAN Compression: Efficient Architectures for Interactive Conditional GANs Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun-Yan Zhu, Song Han TPAMI 2021 | CVPR 2020
|
|
Understanding the Role of Individual Units in a Deep Neural Network David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, Antonio Torralba PNAS 2020
Project | Code |
Paper
|
Arxiv | BibTex |
|
Swapping Autoencoder for Deep Image Manipulation Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei A. Efros, and Richard Zhang NeurIPS 2020
|
|
Differentiable Augmentation for Data-Efficient GAN Training Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, Song Han NeurIPS 2020
|
|
Contrastive Learning for Unpaired Image-to-Image Translation Taesung Park, Alexei A. Efros, and Richard Zhang, Jun-Yan Zhu ECCV 2020
|
|
Rewriting a Deep Generative Model David Bau, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba ECCV 2020
|
|
The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement William Peebles, John Peebles, Jun-Yan Zhu, Alexei A. Efros, Antonio Torralba ECCV 2020
|
|
Transforming and Projecting Images into Class-conditional Generative Networks Minyoung Huh, Richard Zhang, Jun-Yan Zhu, Sylvain Paris, Aaron Hertzmann ECCV 2020
|
|
Diverse Image Generation via Self-Conditioned GANs Steven Liu, Tongzhou Wang, David Bau, Jun-Yan Zhu, Antonio Torralba CVPR 2020
|
|
State of the Art on Neural Rendering Ayush Tewari*, Ohad Fried*, Justus Thies*, Vincent Sitzmann*, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B Goldman, Michael Zollhöfer Eurographics 2020 (STAR Report)
Paper |
Project
|
BibTex |
|
Seeing what a GAN Cannot Generate David Bau, Jun-Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, Antonio Torralba ICCV 2019
|
|
Semantic Photo Manipulation with a Generative Image Prior David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, Antonio Torralba SIGGRAPH 2019
|
|
Learning the Signatures of the Human Grasp Using a Scalable Tactile Glove Subramanian Sundaram, Petr Kellnhofer, Yunzhu Li, Jun-Yan Zhu, Antonio Torralba, and Wojciech Matusik Nature, 569 (7758), 2019
|
|
Connecting Touch and Vision via Cross-Modal Prediction Yunzhu Li, Jun-Yan Zhu, Russ Tedrake, Antonio Torralba CVPR 2019 See CNN News
|
|
Semantic Image Synthesis with Spatially-Adaptive Normalization Taesung Park, Ming-Yu Liu, Ting-Chun Wang, Jun-Yan Zhu CVPR 2019 (Best Paper Finalist) SIGGRAPH 2019 Real-time Live Demo "GauGAN" (with Chris Hebert and Gavriil Klimov) News Won "Best in Show Award" and "Audience Choice Award" in SIGGRAPH 2019 Real-time Live.
Project | Real-time Live | Code | Paper |
|
GAN Dissection: Visualizing and Understanding Generative Adversarial Networks David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, Antonio Torralba ICLR 2019
|
|
Propagation Networks for Model-Based Control Under Partial Observation Yunzhu Li, Jiajun Wu, Jun-Yan Zhu, Joshua B. Tenenbaum, Antonio Torralba, Russ Tedrake ICRA 2019
|
|
Dataset Distillation Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, Alexei A. Efros arXiv 2018
|
|
Visual Object Networks: Image Generation with Disentangled 3D Representation Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum, William T. Freeman NeurIPS 2018
|
|
3D-Aware Scene Manipulation via Inverse Graphics Shunyu Yao*, Tzu-Ming Harry Hsu*, Jun-Yan Zhu, Jiajun Wu, Antonio Torralba, William T. Freeman, Joshua B. Tenenbaum NeurIPS 2018
|
|
Video-to-Video Synthesis Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, Bryan Catanzaro NeurIPS 2018 See our driving game demo.
|
 |
CyCADA: Cycle-Consistent Adversarial Domain Adaptation Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Alexei A. Efros, and Trevor Darrell ICML 2018
|
|
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, Bryan Catanzaro CVPR 2018 Featured in GTC 2018 Keynote.
|
 |
Spatially Transformed Adversarial Examples Chaowei Xiao*, Jun-Yan Zhu*, Bo Li, Mingyan Liu, and Dawn Song ICLR 2018
|
 |
Generating Adversarial Examples with Adversarial Networks Chaowei Xiao, Bo Li, Jun-Yan Zhu, Mingyan Liu, and Dawn Song IJCAI 2018
|
|
Toward Multimodal Image-to-Image Translation Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A. Efros, Oliver Wang, and Eli Shechtman NeurIPS 2017
|
|
Learning to Synthesize and Manipulate Natural Images December, 2017 ACM SIGGRAPH Outstanding Doctoral Dissertation Award. David J. Sakrison Memorial Prize for outstanding doctoral research, by the UC Berkeley EECS Dept.
|
|
Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks Jun-Yan Zhu*, Taesung Park*, Phillip Isola, and Alexei A. Efros ICCV 2017
Project |
PyTorch
|
Torch | Paper |
|
Image-to-Image Translation with Conditional Adversarial Nets Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros CVPR 2017 See Distill blog | Also see neat uses of #pix2pix on Twitter. |
|
Real-Time User-Guided Image Colorization with Learned Deep Priors Richard Zhang*, Jun-Yan Zhu*, Phillip Isola, Xinyang Geng, Angela S. Lin, Tianhe Yu, and Alexei A. Efros SIGGRAPH 2017 Photoshop Element 2020 ColorizePhoto is based on our work
Project | UI Code | PyTorch Training | Youtube | Video |
|
Light Field Video Capture Using a Learning-Based Hybrid Imaging System Ting-Chun Wang, Jun-Yan Zhu, Nima Khademi Kalantari, Alexei A. Efros, and Ravi Ramamoorthi SIGGRAPH 2017
Project
|
GitHub | Youtube | Training code |
|
Generative Visual Manipulation on the Natural Image Manifold Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros ECCV 2016 See Distill blog and article in California Magazine
|
|
A 4D Light-Field Dataset and CNN Architectures for Material Recognition Ting-Chun Wang, Jun-Yan Zhu, Ebi Hiroaki, Manmohan Chandraker, Alexei A. Efros, and Ravi Ramamoorthi ECCV 2016
Paper | Data (thumbnail) | Full data (15.9G) |
|
Learning a Discriminative Model for the Perception of Realism in Composite Images Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros ICCV 2015
|
|
Mirror Mirror: Crowdsourcing Better Portraits Jun-Yan Zhu, Aseem Agarwala, Alexei A. Efros, Eli Shechtman, and Jue Wang SIGGRAPH Asia 2014
Project (code) |
Paper
|
Data
|
Slides
|
Supplement
|
BibTex |
|
AverageExplorer: Interactive Exploration and Alignment of Visual Data Collections Jun-Yan Zhu, Yong Jae Lee and Alexei A. Efros SIGGRAPH 2014
See article in The New Yorker |
|
Unsupervised Object Class Discovery via Saliency-Guided Multiple Class Learning Jun-Yan Zhu, Jiajun Wu, Yan Xu, Eric Chang and Zhuowen Tu TPAMI 2015 | CVPR 2012
Project | Paper | Supplement | Poster | BibTex |
|
Multiple Clustered Instance Learning for Histopathology Cancer Image Classification, Segmentation and Clustering Yan Xu*, Jun-Yan Zhu*, Eric I-Chao Chang and Zhuowen Tu CVPR 2012 | Medical Image Analysis 2014
|
 |
Motion-Aware Gradient Domain Video Composition Tao Chen, Jun-Yan Zhu, Ariel Shamir and Shi-Min Hu TIP 2013
|
Talks
SIGGRAPH Dissertation Award Talk (2018)
Unpaired Image-to-Image Translation
CVPR Tutorial on GANs (2018)
Learning to Synthesize and Manipulate Natural Photos
MIT, HKUST CSE Departmental Seminar, ICCV Tutorial on GANs, O'Reilly AI, AI with the best, Y Conf, DEVIEW, ODSC West (2017)
Stanford, MIT, Facebook, CUHK, SNU (2017)
SIGGRAPH, NVIDIA Innovation Theater, Global AI Hackathon (2017)
Visual Manipulation and Synthesis on the Natural Image Manifold
Facebook, MSR, Berkeley BAIR, THU, ICML workshop "Visualization for Deep Learning" (2016)
Mirror Mirror: Crowdsourcing Better Portraits
SIGGRAPH Asia (2014)
What Makes Big Visual Data Hard?
SIGGRAPH Asia invited course "Data-Driven Visual Computing" (2014)
AverageExplorer: Interactive Exploration and Alignment of Visual Data Collections
SIGGRAPH (2014)
Past Events
MISC
