Course Organization and What is "User Interface Software", Overview of UI Software and Tools

Slides for Lecture 1

Required Readings:

• Brad A. Myers. "Graphical User Interface Programming," chapter 48 of Computer Science Handbook -- Second Edition. Allen B. Tucker, editor in chief. Boca Raton, FL: Chapman & Hall/CRC Press, Inc., 2004. pp. 48-1 - 48-29. pdf  (This is a revision of Brad A. Myers. "User Interface Software Tools," ACM Transactions on Computer-Human Interaction. vol. 2, no. 1, March, 1995. pp. 64-103. ACM DL Reference)
• Brad A. Myers, Scott Hudson and Randy Pausch. "Past Present and Future of User Interface Software Tools," ACM Transactions on Computer Human Interaction, (Mar. 2000). Volume 7, Issue 1. pp. 3-28. ACM DL Reference.

Start on Homework 1 - Editor

Why are user interfaces hard to design and implement?

Slides for Lecture 2

Required Readings:

• Brad A. Myers. "Challenges of HCI Design and Implementation," ACM Interactions. vol. 1, no. 1. January, 1994. pp. 73-83. ACM DL
  Discusses why user interfaces are important, and why they are hard to design and implement.
• Brad A. Myers. "A Brief History of Human Computer Interaction Technology." ACM interactions. Vol. 5, no. 2, March, 1998. pp. 44-54. ACM DL

Amusing Reading:

• Tyson R. Henry, Scott E. Hudson, Andrey K. Yeatts, Brad A. Myers and Steven Feiner; "A nose gesture interface device: extending virtual realities," Proceedings of the fourth annual ACM symposium on User interface software and technology, UIST'1991, Pages 65 - 68. ACM DL or local copy.

Martin Luther King Day; No Classes

Basic Computer Graphics

Slides for Lecture 3

Required Readings:

• New Foley & van Dam, pp. 25-40, 52-60
• old Foley & van Dam, pp. 132-134 (color maps)
• These are combined into one pdf (link only works from CMU or with VPN, but also available on Canvas under Course Content)

Other Output Models: Structured Graphics; Object-Oriented Techniques; Hints for HW2

Slides for Lecture 4

Required Readings:

• Brad A. Myers, Richard G. McDaniel, Robert C. Miller, Alan Ferrency,  Andrew Faulring, Bruce D. Kyle, Andrew Mickish, Alex Klimovitski, and Patrick Doane. "The Amulet Environment: New Models for Effective User Interface Software Development", IEEE Transactions on Software Engineering, Vol. 23, no. 6. June, 1997. pp. 347-365. local pdf or IEEE Explore Ref.
• David S. Kosbie, Brad Vander Zanden, Brad A. Myers, Dario Giuse. "Automatic Graphical Output Management", in Brad A. Myers, editor. The Garnet Compendium: Collected Papers, 1989-1990. Carnegie Mellon University School of Computer Science Technical Report, no. CMU-CS-90-154, August, 1990. pp. 30-43. pdf.

Optional Readings:

• Brad A. Myers, Dario A. Giuse, and Brad Vander Zanden. "Declarative Programming in a Prototype-Instance System: Object-Oriented Programming Without Writing Methods," Proceedings OOPSLA'92: ACM Conference on Object-Oriented Programming Systems, Languages, and Applications. October 18-22, 1992. Vancouver, BC, Canada.  SIGPLAN Notices, vol. 27, no. 10. pp. 184-200. ACM DL Reference.

Course add Deadline.
Turn in Homework 1.
Start on Homework 2 - Output

Design and Evaluation of APIs and UI Tools, API Usability, Cognitive Dimensions

Slides for Lecture 5

Required Readings:

• Brad A. Myers, Andrew J. Ko, Thomas D. LaToza, and YoungSeok Yoon. "Programmers Are Users Too: Human-Centered Methods for Improving Programming Tools," IEEE Computer, Special issue on UI Design, 49, issue 7, July, 2016, pp. 44-52. IEEE DL or local pdf.
• Brad A. Myers and Jeffrey Stylos, "Improving API Usability", Communications of the ACM, vol 59, No. 6, June, 2016, pp. 62-69, Official ACM DL entry; html or local pdf.
• Jakob Nielsen, Heuristic Evaluation. On line in html format. Includes list of 10 heuristics.
• Green, T.R.G. and Petre, M., “Usability Analysis of Visual Programming Environments: A 'Cognitive Dimensions' Framework.” Journal of Visual Languages and Computing, 1996. 7(2): pp. 131-174. ScienceDirect.
  Includes an analysis of HyperCard

Recommended Readings:

• Jeffrey Stylos and Brad Myers, "Mapping the Space of API Design Decisions," 2007 IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC'07. Sept 23-27, 2007, Coeur d'Alene, Idaho. pp. 50-57. ieee or local pdf
• Brian Ellis, Jeffrey Stylos, and Brad Myers. "The Factory Pattern in API Design: A Usability Evaluation". International Conference on Software Engineering (ICSE'2007). May 20-26, 2007. Minneapolis, MN. pp. 302-312. ACM DL or local pdf
• Clarke, Steve, “Measuring API Usability.” Dr. Dobb's Journal; Special Windows/.NET Supplement, May, 2004. pp. S6-S9. html
• Jeffrey Stylos, Brad A. Myers. "The Implications of Method Placement on API Learnability," Sixteenth ACM SIGSOFT Symposium on Foundations of Software Engineering (FSE 2008). Atlanta, GA, November 9-14, 2008. 105-112. ACM DL or  local pdf
• Jeffrey Stylos and Steven Clarke. "Usability Implications of Requiring Parameters in Objects' Constructors". International Conference on Software Engineering (ICSE 2007). May 20-26, 2007. Minneapolis, MN. pp. 529-539. ACM DL or local pdf
• T.R.G. Green, Cognitive Dimensions of Notations. Cognitive Dimensions of Notations Resource Site
• Matthew Conway, et. al., "Alice: lessons learned from building a 3D system for novices", CHI'2000. pp. 486-493. ACM DL or see this html summary.
  Alice was informed by many user studies over the years.

Constraints 1

Slides for Lecture 6

Required Readings:

• Brad Vander Zanden, Brad A. Myers, Dario Giuse and Pedro Szekely. "Integrating Pointer Variables into One-Way Constraint Models," ACM Transactions on Computer-Human Interaction. vol. 1, no. 2, June, 1994. pp. 161-213. ACM DL Reference
• Brad Vander Zanden, "An Incremental Algorithm for Satisfying Hierarchies of Multi-way, Dataflow Constraints", ACM Transactions on Programming Languages and Systems, 18(1), January, 1996. pp. 30-72. ACM DL Reference

Constraints 2 - Guest Lecture by Scott Hudson

Slides for Lecture 7

Required Readings:

• Scott E. Hudson. "Incremental attribute evaluation: a flexible algorithm for lazy update," ACM Transactions on Programming Languages and Systems, Volume 13 , Issue 3 (1991), Pages 315-341.  ACM DL Reference
• Bjorn N. Freeman-Benson, John Maloney and Alan Borning. "An incremental constraint solver," Communications of the ACM . Volume 33 , Issue 1 (1990). Pages 54-63. ACM DL Reference

Optional Readings:

• Scott E. Hudson, Ian Smith, "Ultra-lightweight constraints", UIST'1996. pp. 147 - 155. ACM DL Reference

UI Software Models: Lexical-Syntax-Semantics, Seeheim Model, MVC, Design Patterns

Slides for Lecture 8

Required Readings:

• William Buxton, "Lexical and Pragmatic Considerations of Input Structures," Computer Graphics, January, 1983, (17)1, pp. 31-37. (lexical, syntactic, semantic, etc.). html
• Glenn Krasner and Stephen T. Pope, "A Cookbook for Using the Model-View-Controller User Interface Paradigm in Smalltalk-80", Journal of Object-Oriented Programming (JOOP). August-September, 1988. vol. 1, no. 3. pp. 26-49. pdf scan at UCI

Optional Readings:

• Mark Green, "Report on Dialogue Specification Tools," User Interface Management Systems, G. Pfaff, ed. Berlin: Springer-Verlag, 1985. pp. 9-20. (Seeheim model)
  (Here is a pdf scan of the book chapter, but it is not generally available on the web)

Toolkits Organizations: callbacks, resources, widget hierarchies, geometry management

Slides for Lecture 9

Optional Readings.

• Linton, M.A., Vlissides, J.M., and Calder, P.R., “Composing user interfaces with InterViews.” IEEE Computer, Feb, 1989. 22(2): pp. 8-22. pdf at Georgia Tech
• Funny flash animation about using Java Grid Bag layout (2:42): http://madbean.com/blog/2004/17/totallygridbag.html or YouTube version.

Turn in Homework 2.
Start on Homework 3 - Constraints

Historical Perspectives: Menu trees, Transition Networks, Grammars, Event Languages, HyperTalk, Declarative Languages

Slides for Lecture 10

Required Readings:

• Mark Green, "A Survey of Three Dialogue Models," ACM Transactions on Graphics (5)3, Jul, 1986, pp 244-275. ACM DL Reference
• chapter 10 from Olsen's book: "Editing Dialog Models". User Interface Management Systems: Models and Algorithms.Morgan Kaufmann; (November 1991) (pdf of the chapter, but this is not generally available on the web), or Amazon listing for book.

Optional readings:

• Robert J.K. Jacob,  "A Specification Language for Direct Manipulation Interfaces," ACM Transactions on Graphics, Oct, 1986, vol. 5, no. 4, pp. 283-317. ACM DL Reference.
• Philip J. Hayes, Pedro A. Szekely, and Richard A. Lerner, "Design Alternatives for User Interface Management Systems Based on Experience with COUSIN", Proceedings SIGCHI'85: Human Factors in Computing Systems, San Francisco, CA, Apr, 1985, pp. 169-175. ACM DL Reference. ACM DL Reference

Model-based tools: Creating the UI Automatically

Slides for Lecture 11

Required Readings:

• Dan R. Olsen, Jr., "A Programming Language Basis for User Interface Management," Proceedings SIGCHI'89, Austin, TX, Apr, 1989, pp. 171-176. ACM DL Reference
• Jeffrey Nichols, Brad A. Myers, Michael Higgins, Joe Hughes, Thomas K. Harris, Roni Rosenfeld, Mathilde Pignol. "Generating Remote Control Interfaces for Complex Appliances." CHI Letters: ACM Symposium on User Interface Software and Technology, UIST'02, 27-30 Oct. 2002, Paris, France. pp. 161-170. pdf

Optional Readings:

• Brad Vander Zanden and Brad A. Myers,  "Automatic, Look-and-Feel Independent Dialog Creation for Graphical User Interfaces," Proceedings SIGCHI'90: Human Factors in Computing Systems.  Seattle, WA, April 1-5, 1990. pp. 27-34. ACM DL Reference
• Pedro Szekely, Ping Luo, and Robert Neches, "Beyond Interface Builders: Model-Based Interface Tools, Proceedings INTERCHI'93: Human Factors in Computing Systems. Amsterdam, The Netherlands, April 24-29, 1993. pp. 383-390. ACM DL Reference.
• Paterno, F., Mancini, C. & Meniconi, S. (1997), "ConcurTaskTrees: A Diagrammatic Notation for Specifying Task Models," in Proceedings of Interact-97, Sydney, Australia, pp.362-369. pdf
• S. R. Ponnekanti, B. Lee, A. Fox, P. Hanrahan and T.Winograd. “ICrafter: A service framework for ubiquitous computing environments,” UBICOMP 2001, Atlanta, Georgia, 2001. pp. 56-75. URL
• K. Gajos, Weld, D. “SUPPLE: Automatically Generating User Interfaces,”
  IUI'04: Intelligent User Interfaces, Funchal, Portugal, 2004. pp. 93-100. pdf. See also the video.
• Jeffrey Nichols, Duen Horng Chau, Brad A. Myers, "Demonstrating the Viability of Automatically Generated User Interfaces" Proceedings CHI'2007: Human Factors in Computing Systems. San Jose, CA, April 28 - May 3, 2007. pp. 1283-1292. pdf

Conventional Input Models for Window Managers and Toolkits

Slides for Lecture 12

Declarative input models: Garnet & Amulet "Interactor" and "Command" Objects & Support for Undo

Slides for Lecture 13

Required Readings:

• Brad A. Myers. "A New Model for Handling Input," ACM Transactions on Information Systems, vol. 8, No. 3. July, 1990. pp. 289-320. ACM DL Reference.
• Brad A. Myers and David Kosbie. "Reusable Hierarchical Command Objects," Proceedings CHI'96: Human Factors in Computing Systems. Vancouver, BC, Canada. April 14-18, 1996. ACM DL Reference
• Brad A. Myers. "Scripting Graphical Applications by Demonstration," Proceedings CHI'98: Human Factors in Computing Systems. Los Angeles, CA, April 18-23, 1998. pp. 534-541. ACM DL Reference and local pdf and video. (Topaz)

Interactive Tools: Prototypers (HyperCard, Director, Visual Basic, Balsamiq), Interface Builders, Sketching Tools

Slides for Lecture 14

Required Readings:

• James Landay and Brad A. Myers. "Interactive Sketching for the Early Stages of User Interface Design," Proceedings CHI'95: Human Factors in Computing Systems. Denver, CO. May, 1995. pp. 43-50. (html) or ACM DL Ref
• Brad A. Myers. "Separating Application Code from Toolkits:  Eliminating the Spaghetti of Call-Backs,"  ACM Symposium on User Interface Software and Technology: UIST'91, Hilton Head, SC, Nov. 11-13, 1991. pp. 211-220. ACM DL Reference

Optional Readings:

• James Lin, Michael Thomsen, and James A. Landay, "A Visual Language for Sketching Large and Complex Interactive Designs." In CHI Letters: Human Factors in Computing Systems, CHI 2002, 2002. 4(1): pp. 307-314. ACM DL Ref
  See also the DENIM Video (better version of video in a temporary place), (longer video, no sound)
• Walker, M., Takayama, L., and Landay, J. (2002) High-fidelity or low-fidelity, paper or computer medium? Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting (in press). 
  PDF
• James A. Landay and Brad A. Myers. Just Draw It! Programming by Sketching Storyboards.   Carnegie Mellon University School of Computer Science Technical Report, no. CMU-CS-95-199 and Human Computer Interaction Institute Technical Report CMU-HCII-95-106. November, 1995. html or PDF

Turn in Homework 3.
Start on Homework 4 - Input

Demonstrational Tools

Slides for Lecture 15

Required Readings:

• Brad A. Myers, Richard G. McDaniel, and David S. Kosbie. "Marquise: Creating Complete User Interfaces by Demonstration," Proceedings INTERCHI'93: Human Factors in Computing Systems. Amsterdam, The Netherlands, April 24-29, 1993. pp.  293-300. ACM DL Reference. (pdf)
• Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom Mitchell, Brad Myers. "PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations," ACM Symposium on User Interface Software and Technology, UIST'19, New Orleans, LA, October 20-23, 2019. pp. 577-589. ACM DL and local pdf
• Brad Myers, Richard McDaniel and David Wolber. "Programming by example: Intelligence in Demonstrational Interfaces," Communications of the ACM. March, 2000. vol. 43, no. 3. pp. 82-89. ACM DL Reference. (pdf)

Optional Readings:

• Richard G. McDaniel and Brad A. Myers, "Getting More Out Of Programming-By-Demonstration." Proceedings CHI'99: Human Factors in Computing Systems. Pittsburgh, PA, May 15-20, 1999. pp. 442-449. (Gamut) ACM DL Reference. (pdf)
• Brad A. Myers. "Creating User Interfaces Using Programming-by-Example, Visual Programming, and Constraints," ACM Transactions on Programming Languages and Systems. vol. 12, no. 2, April, 1990. pp. 143-177. (Peridot) ACM DL Reference
• Brad A. Myers, Brad Vander Zanden, and Roger B. Dannenberg. "Creating Graphical Interactive Application Objects by Demonstration," ACM Symposium on User Interface Software and Technology: UIST'89, Williamsburg, VA, Nov. 13-15, 1989. pp. 95-104. (Lapidary) ACM DL Reference. (see also the video excerpt)

Spring Break, no classes

Mid semester grades due.

Spring Break, no classes

CLASS CANCELLED due to Covid-19

Component Techniques: Andrew, OLE, OpenDoc, Java Beans, Web Services, Micro-services

All classes in the second half, starting March 18, will be done using Zoom, through the link shown on Canvas.

Slides for Lecture 16
Link for video of lecture (on Canvas — class use only)

Required Readings:

• Andrew J. Palay, Fred Hansen, Mike Kazar, Mark Sherman, Maria Wadlow, Thomas Neuendorffer, Zalman Stern, Miles Bader, and Thom Peters. "The Andrew Toolkit - An Overview", Published in Proceedings USENIX Technical Conference, Winter 1988. Reproduced as ITC Technical Report 061. pdf

Optional Readings:

• Chad Vawter and Ed Roman, "J2EE vs. Microsoft.NET: A comparison of building XML-based web services," June 2001. http://www.theserverside.com/tt/articles/article.tss?l=J2EE-vs-DOTNET
• JavaBeans Web Pages (http://java.sun.com/products/javabeans/), especially, the official JavaBeans tutorial, and an unofficial "NetBeans JavaBeans Tutorial":
  http://wiki.netbeans.org/NetBeansJavaBeansTutorial

Turn in Homework 4 - extended deadline
Start on Homework 5 - Widgets

ConstraintJS and InterState - Guest Lecture by Stephen Oney

Slides for Lecture 17
Link for video of lecture (on Canvas — class use only)

Stephen Oney is a Assistant Professor in the University of Michigan School of Information. He received his PhD from the HCII at CMU in 2015, advised by Brad Myers. This thesis topics included the ConstraintJS and InterState user interface development tools. His current research focuses on enabling and encouraging more people to write and customize computer programs by creating new programming tools and exploring usability issues in programming environments.

Required Readings:

• Stephen Oney, Brad A. Myers, Joel Brandt, "ConstraintJS: Programming Interactive Behaviors for the Web by Integrating Constraints and States", UIST'2012: ACM Symposium on User Interface Software and Technology, October 7-10, 2012. Cambridge, MA. pp. 229-238. ACM DL or local pdf
• Stephen Oney, Brad A. Myers, and Joel Brandt, "InterState: A Language and Environment for Expressing Interface Behavior", ACM Symposium on User Interface Software and Technology, UIST'14, October 5-8, 2014, Honolulu, Hawaii. pp. 263-272. ACM DL, pdf, video, project Page.

Simple User Interface Toolkits and EUP for UIs

Slides for Lecture 18
Link for video of lecture (on Canvas — class use only)

Required Readings:

• Pausch, R., Conway, M., & DeLine, R. (1992). Lesson Learned from SUIT, the Simple User Interface Toolkit. ACM Transactions on Information Systems, 10(4), 320-344. ACM DL Ref, or citeseer ref
• Conway, M., Audia, S., Burnette, T., Cosgrove, D., Christiansen, K., Deline, R., et al. (2000, Apr 1-6). Alice: Lessons Learned from Building a 3D System For Novices. Paper presented at the Proceedings of CHI 2000, The Hague, The Netherlands. ACM DL Ref, or citeseer ref
• J.F. Pane, B.A. Myers, and L.B. Miller, "Using HCI Techniques to Design a More Usable Programming System," 2002 IEEE Symposia on Human Centric Computing Languages and Environments (HCC'2002). Arlington, VA, September 3-6, 2002. pp. 198-206. local pdf. (Winner Most Influential Paper Award for important influences on VL/HCC research or commerce over the last 10+/-1 years in 2012.)

Optional Readings:

• Repenning, A & Ioannidou, A. (2004) Agent-based end-user development. Communications of the ACM, 47(9), pp. 43-46. ACM DL Ref
• Anders I. Mørch., Stevens, G., et al. (2004) Component-based technologies for end-user development. Communications of the ACM, 47(9), pp. 59-62. ACM DL Ref
• Scratch, Yahoo! pipes, Vegemite
• Brad A. Myers, John F. Pane and Andy Ko, "Natural Programming Languages and Environments". Communications of the ACM. 47(9), pp. 47-52. ACM DL Ref
• Fabio Casati: How End-User Development Will Save Composition Technologies from Their Continuing Failures. IS-EUD 2011: 4-6.  SpringerLink

Turn in Homework 4 - extended deadline

Guest lecture about Flutter: Hans Muller and Dan Field from Google

Hans Muller's bio:
Hans is the manager of the Flutter framework team and has been working on the project since it was just a crazy idea. Before the Flutter project Hans worked on Chrome at Google and Adobe, and before that he was the tech lead for the Java Swing GUI toolkit at Sun. Earlier at Sun, Hans built the LispView GUI toolkit for CommonLisp, which was included in a toolkit survey conducted by one Brad Myers. Hans graduated from CMU in 1982.

Dan Field's bio:
I'm a software engineer at Google working on the Flutter team. I've worked on platform specific embedding and tooling, core Flutter framework areas such as scrolling and image rendering, as well as build infrastructure, testing, and continuous integration. Before working on Flutter, I worked on mobile and desktop application development, web/front-end development, as well as healthcare data integration (OLTP and OLAP).

Slides for Lecture 19 (Google Slides): part 1 and part 2, and local copy (pdf)

Video of the presentation (publically accessible with permission)

Required Readings:

• https://flutter.dev/docs/resources/technical-overview
• https://flutter.dev/docs/resources/inside-flutter

Optional Readings:

• https://flutter.dev/docs/resources/faq
• https://www.youtube.com/watch?v=UUfXWzp0-DU - An excellent overview of Flutter's rendering pipeline by Adam Barth, former Technical Lead on the Flutter team
• https://www.youtube.com/watch?v=dkyY9WCGMi0 - An excellent overview of Flutter's architecture by Ian Hickson, current Technical Lead of the Flutter team

*Sorry the video in the second part only shows me instead of Dan!

3/31: Withdrawal deadline - postponed until April 6.

Animation in Toolkits

Slides for Lecture 20
Link for video of lecture (on Canvas — class use only)

Required Readings:

• Bay-Wei Chang and David Ungar. “Animation: From Cartoons to the User Interface,” UIST'93: ACM SIGGRAPH Symposium on User Interface Software and Technology, Atlanta, GA, Nov, 1993. pp. 45-55. ACM DL Reference
  
• Brad A. Myers, Robert C. Miller, Rich McDaniel, and Alan Ferrency, "Easily Adding Animations to Interfaces Using Constraints." ACM Symposium on User Interface Software and Technology, UIST'96, November 6-8, 1996. Seattle, WA. pp. 119-128. ACM DL Reference

Guest lecture about Unity, a leading 3D toolkit and game engine, by Adam Mechtley and Damian Campeanu: "A tale of three UI frameworks: A decade of evolving developer and designer workflows in a game engine"

Talk Summary:
Over its 15-year lifetime, the Unity game engine has implemented three different UI frameworks. Each framework aimed to address the needs of game developers at different historical moments in time. In this lecture, we will discuss the evolution of these frameworks, highlighting unique advantages and disadvantages, technical details, and requirements that influenced the designs of each one.

Adam Mechtley's bio:
I have worked in the games industry since 2004, and am currently a lead software developer at Unity Technologies focused on physics, animation rigging, and data-oriented design. My professional work has focused on developer tools, usability, first-time user experience, and software development craftsmanship. I am also trained as a learning scientist, and hold a PhD from the University of Wisconsin–Madison focused on design, informal, and creative education. My academic work has focused on design-based research related to epistemic cognition, argumentation, and learning in science- and engineering-based computer and board games. I am based in Copenhagen, Denmark.

Damian Campeanu's bio:
I'm mostly a C++ and C# desktop application and games developer with 10+ years of professional experience, primarily focusing on 3D content creation tools, developer tools, API design, and risky live product demos. I implemented a broad-use C API called Houdini Engine, wrapping the C++ procedural geometry, animation, and special effects core of Houdini, by SideFX. Eating my own dog-food, I developed a feature-rich open-source C# Houdini plugin for Unity using Houdini Engine. At the moment, I am the primary developer on a visual UI Builder for Unity's new UI Toolkit (formally UIElements). Previously at Unity, I worked on the UI Toolkit C# API and I've given several live product demos and talks at Unity's yearly Unite conferences and various events on both the UI Builder and UI Toolkit. I continue my passion for software development at home with an on-going game (engine) project called Synekism since 2010, written in modern C++, with use of CMake, Clang, SDL2, and Vulkan, and with support for Windows, Linux, and macOS.

Google Slides for Lecture 21 and local pdf copy

Video of the presentation (publically accessible with permission)

Required Readings:

• Richard Fine, "Going deep with IMGUI and Editor Customization", December 22, 2015, blog post -- discusses the event loop for Unity's old immediate-mode GUI system
• J. Blow, “Game development: Harder Than You Think”, ACM Queue, ACM, New York, 2004, pp. 29-37. ACM DL REF

Optional readings:

• Unity's modern UI framework, UI Toolkit (formally UIElements).
• Damian Campeanu, "Building UI for games with the new UI Builder" (45 min video), https://youtu.be/t4tfgI1XvGs
  
• B MacIntyre, M Gandy, S Dow, JD Bolter. "DART: a toolkit for rapid design exploration of augmented reality experiences." UIST '04: Proceedings of the 17th annual ACM symposium on User interface software and technology, October 2004, pp. 197–206, ACM DL Ref
• Game Engine Architecture by Jason Gregory (book) "Introduction". pg 1-22 (see Canvas)
• Take a look at standard game engines of today: Unity Game Engine, Unreal Engine, and (non-programmers) GameMaker. There are dozens of other game engines available.
• OpenGL is the graphics library standard, and it is open source. However, it is highly complex. For a gentler introduction and some videos demonstrating its capability, look at a javascript-based wrapper for OpenGL called three.js

4/6: New withdrawal deadline

Guest lecture by Dominik Moritz: Toolkits for Visualization and UIs in Data Science

Slides for Lecture 22 (pdf)
Link for video of lecture (on Canvas — class use only)

Short bio of Dominik Moritz:
I’m a research scientist at Apple, working on data visualization and interactive systems for artificial intelligence. In fall of 2020, I will start as an assistant professor at Carnegie Mellon University. I received my PhD from the Paul G. Allen School at the University of Washington, where I worked with Jeff Heer and Bill Howe in the Interactive Data Lab and the Database Group. I research scalable interactive systems for visualization and analysis. My systems have won awards at premier academic venues and are used by the Python and JavaScript data science communities.

Required Readings:

• Vega-Lite talk: https://www.youtube.com/watch?v=9uaHRWj04D4
• Michael Bostock, Vadim Ogievetsky and Jeffrey Heer. (2011) "D3: Data-Driven Documents", http://vis.stanford.edu/files/2011-D3-InfoVis.pdf. (Paper introducing D3)

Optional Readings:

• Arvind Satyanarayan, Dominik Moritz, Kanit Wongsuphasawat, and Jeffrey Heer, "Vega-Lite: A Grammar of Interactive Graphics", IEEE Transactions on Visualization and Computer Graphics, Volume: 23 , Issue: 1 , Jan. 2017, IEEE DL or local pdf
• Dominik Moritz, Kanit “Ham” Wongsuphasawat, Arvind Satyanarayan, and Jeffrey Heer, Introducing Vega-Lite 2.0, Oct 31, 2017, blog post
• Kanit “Ham” Wongsuphasawat, Dominik Moritz and Jeffrey Heer, Introducing Vega-Lite, Feb 23, 2016, blog post
• Bret Victor's "Up and Down the Ladder of Abstraction" (2011): http://worrydream.com/#!2/LadderOfAbstraction (this one is interactive, and really fun)
• Michael Bostock, "What Makes Software Good?" (2016): https://medium.com/@mbostock/what-makes-software-good-943557f8a488. (This has some 'big picture' thoughts on software, but is also a writeup explaining major changes he made to D3)

Turn in Homework 5.
Start on Final Project

Toolkit support for Gestural Input Techniques, Handwriting Recognition

Slides for Lecture 23
Link for video of lecture (on Canvas — class use only)

Required Readings:

• J.O. Wobbrock, AD Wilson, Y Li. "Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes", In UIST '07, p. 159-168. ACM DL Ref

Optional Readings:

• Stephen Oney, Rebecca Krosnick, Joel Brandt, Brad Myers. "Implementing Multi-Touch Gestures with Touch Groups and Cross Events," Proceedings SIGCHI '2019: Human Factors in Computing Systems. Glaskow, UK. May 4-9, 2019. ACM DL or local pdf. Best Paper Honorable Mention Award.
• The Gesture Recognition Toolkit, http://www.jmlr.org/papers/volume15/gillian14a/gillian14a.pdf
• Hand Gesture Using Kinect, http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6269439

Turn in One Page Proposals for Final Project.

Toolkit support for Assistive and Accessible Interfaces

Slides for Lecture 24
Link for video of lecture (on Canvas — class use only)

Required Readings:

• W3C, "Introduction to Web Accessibility," on-line
• Zimmermann, G., Vanderheiden, G. Accessible design and testing in the application development process: considerations for an integrated approach. Univ Access Inf Soc 7, 117–128 (2008). https://doi.org/10.1007/s10209-007-0108-6

Optional Readings:

• Gemou, M., Montalva Colomer, J.B., Cabrera-Umpierrez, M.F. et al. Validation of Toolkits for developing third-generation Android accessible mobile applications. Univ Access Inf Soc, 15, 101–127 (2016). https://doi.org/10.1007/s10209-014-0377-9
• Google: Accessibility Android Developers, http://developer.android.com/design/patterns/accessibility.html#tools

Special Make up class in lieu of spring carnival.

Survey of Web Toolkits - guest lecture by Joseph Chang and Nathan Hahn

Slides for Lecture 25 (pdf)
Link for video of lecture (on Canvas — class use only)

Short bio of Joseph Chang:
I am a PhD student in the HCII and LTI at CMU advised by Niki Kittur. I will talk about a brief history of how programming for the web has changed throughout the years, how these changes enabled us to build different types of interfaces, and factors that triggered these paradigm shifts. My current research focuses on building interactive systems that help people explore, structure, and make sense of online information in complex decision-making scenarios.

Short bio of Nathan Hahn:
Nathan is a current PhD student in the HCII also advised by Niki Kittur. His current research revolves around building tools and interface to help individuals manage large collections of web information when they are learning about new topics, leveraging computational techniques to support them along the way. In the past decade, he's built UIs in everything from jQuery to Java to React to iOS and will be focusing on how modern toolkits build upon past lessons to provide a better development experience.

Tools for Creating Conversational Agents and ChatBots - guest lecture by Toby Jia-Jun Li

Slides for Lecture 26 (pdf)
Link for video of lecture (on Canvas — class use only)

Short bio of Toby Li:
Toby is a 5th year Ph.D. student at the Human-Computer Interaction Institute, Carnegie Mellon University. He is also a Yahoo! InMind Fellow. His research focuses on designing, implementing, and studying new interaction techniques, multi-modal interfaces, dialog systems, and the underlying AI techniques to empower intelligent agents to interactively learn new tasks and concepts from end users. Toby has published in premier academic venues on interface technologies, groupware, natural language processing, mobile systems, and developer tools (e.g., CHI, UIST, CSCW, ACL, MobiSys, VL/HCC), including two award-winning papers. His work has been supported by NSF, Verizon, and JP Morgan.

• Examples: DialogFlow, Microsoft Bot Network, Voiceflow, ...

Required Readings:

• Dan Jurafsky and James Martin. Dialog Systems and Chatbots. Chapter of Speech and Language Processing (3rd ed.) https://web.stanford.edu/~jurafsky/slp3/24.pdf
• Google: DialogFlow Quickstart Video https://www.youtube.com/watch?v=9aHusGxntPw&list=TLPQMTMwNDIwMjAF_Z9_ZwIbEA&index=1

Optional Readings:

• Daniel G. Bobrow, Ronald M. Kaplan, Martin Kay, Donald A. Norman, Henry Thompson, and Terry Winograd. 1977. GUS, a Frame-driven Dialog System. Artif. Intell. 8, 2 (April 1977), 155–173. https://nlp.stanford.edu/acvogel/gus.pdf
• Toby Jia-Jun Li and Oriana Riva. KITE: Building conversational bots from mobile apps. Proceedings of the 16th ACM International Conference on Mobile Systems, Applications, and Services (MobiSys 2018). http://toby.li/wp-content/uploads/2018/06/TobyLi-MobiSys18-KITE.pdf

Toolkits for SmartPhones & Other Mobile Devices - guest lecture by Mayank Goel

Slides for Lecture 27 (pdf)
Link for video of lecture (on Canvas — class use only)

Short bio of Mayank Goel:
I am an Assistant Professor in ISR and HCII at CMU. I lead the Smash Lab (Smart Sensing for Humans Lab) at CMU. Our group focuses on designing, implementing, and testing new sensing systems for immediate, high-impact problems in various domains, including health sensing, technologies for developing world, activity recognition, and novel interactions. Many times solutions in these domains need to be used and evaluated by the end user in their daily lives. Therefore, we collaborate closely with medical professionals, bio-engineers, and designers to develop end-to-end solutions that can be immediately used and evaluated outside the lab environment. We often rely on adding new capabilities to existing sensors and devices. This approach allows us to add various functionalities to our daily-use devices without making any hardware modifications. This work requires expertise in mobile computing, sensing, signal processing, and machine learning. We actively publish in conference venues of three computer science disciplines:

• Topics: what are some important mobile and cross-platform toolkits to cover? Swift, Xcode, Android Studio, React-Native, ...

CANCELLED: Brad away at CHI

No class, please work on your projects!

Please fill out the class anonymous questionnaire on SurveyMonkey: https://www.surveymonkey.com/r/05830spring2020 and the official CMU evaluation: https://cmu.smartevals.com.

CANCELLED: Brad away at CHI

Project presentations - all 4 groups will present

Link for video of presentations (on Canvas — class use only)

Project Presentations Due today

Everyone please review other groups' presentations using this Google form: https://forms.gle/d7jnh7zWka9nR4oZ7.

Project Written Reports Due on May 4, 2020 at 1:30pm

5/14: grades due. New deadline: May 19, grades due