Project Summary

Standard lessons in Cognitive Tutors (built by Carnegie Mellon's PACT Center) on how to solve algebra equations require students to interact with an interface very different than that of solving equations on paper. Although some alternative interfaces have been explored, typically students use menus to perform manipulations such as adding a quantity to both sides of an equation and then type in the result of such manipulations using the keyboard. There are several weaknesses to this style of online tutoring, especially for mathematics.

1. Keyboard-and-mouse-based interfaces may impose extraneous cognitive load on students during problem-solving and distract them from the math concepts of import, by requiring them to learn and account for idiosyncracies of the type-in interface that are irrelevant to the learning event.
2. Typing mathematics on the computer generally involves some degree of linearization of the input (for instance, "2^x" instead of placing the x as a superscript. A more natural interaction experience using the handwriting modality can more directly support the two-dimensional math notations such as fractions and exponents that more constrained, traditional interfaces do not, allowing students to construct their solutions and express their understanding in an unconstrained way.
3. More generally, current standard interfaces for entering mathematical equations on computers are arguably limited and cumbersome. Mathematical notations have evolved to aid visual thinking and yet text-based interfaces relying on keyboard-and-mouse input do not take advantage of the natural two-dimensional aspects of mathematical equations.
4. Choosing solving steps from a menu rather than requiring the students to generate the next step independently can encourage non-learning-oriented behaviors such as "gaming the system".

Since Cognitive Tutors are not intended to replace teachers or traditional classroom instruction and tests, these lessons could be a weakness in the curriculum with respect to encouraging transfer from the Cognitive Tutor experience to the paper experience. Due to its similarities to paper-based mathematics, pen-based handwriting input may be faster, more efficient, and more preferable for entering mathematics on computers.

Our results have shown that, in addition to more general usability gains for speed and user satisfaction [Anthony et al, 2005], students are able to solve more problems in the same amount of time, while achieving similar learning gains on the post-test [Anthony et al, 2007a]. We were able to enhance the accuracy of the handwriting recognition used in the interface through use of the tutoring system's knowledge of the student learning state as a source of context [Anthony, 2008]. A more natural interaction experience could lead to decreased cognitive load, better performance, and improved usability in the general case, and learning gains and improved transfer to paper tests in the intelligent tutoring case [Anthony et al, 2007a].

Project Status and Findings

This project ran from 2004-2008. We completed several formative user studies and created a prototype system. Major findings include:

1. General users entering calculus-level mathematics equations and expressions on the computer were faster in handwriting than typing (using Microsoft Equation Editor) by a factor of 3! This advantage increased as equation length and complexity increased. They also rated the handwriting modality more highly on a post-sessions Likert scale [Anthony et al, 2005].
2. In the same study, users also input equations in a multimodal handwriting-plus-speech method, which ended up being faster and better liked than the keyboard-and-mouse method and was not much worse than handwriting alone. Also, users’ speech while writing differed from when speaking alone. Finally, user errors in handwriting and speech were non-overlapping, a fact which a multimodal recognition system could use to improve overall performance [Anthony et al, 2006a].
3. Students learning or reviewing algebra equation solving who used a handwriting interface experienced similar learning gains as measured by improvement from pre-test to post-test than those who used a keyboard interface. However, the handwriting students finished the lesson in about half the time of the keyboard students [Anthony et al, 2007a].
4. In the same study, students experienced a greater degree of what we call "transfer to paper." In conditions in which there was a modality switch (e.g., writing on the pre-test to typing in the interface to writing on the post-test), there was a lower correlation in performance during training vs on the tests, indicating that performance iduring tutoring is more predictive of performance during testing when students use the same modality to solve the problems [Anthony et al, 2007a].
5. In general, students and users indicate that handwriting is a more suitable, natural and useful input modality for mathematics on the computer than typing [Anthony et al, 2005; Anthony et al, 2008a].
6. Use of worked examples as part of the tutoring system helped provide a feed-forward instructional intervention to mitigate against the lack of step-targeted feedback which was unfavorable due to inadequate raw handwriting recognition accuracies [Anthony, 2008].
7. Tutor-provided context such as knowledge about student skills, common student errors and the correct answer to the current problem combined with the recognition hypotheses improved raw recognition accuracy by 10% points, decreasing the proportion of time the system would erroneously interrupt the student as s/he worked [Anthony, 2008].

Current project status: This project is no longer active.

Documents

Thesis Proposal

Publications

Presentations

Posters

Selected References

Handwriting, Speech, and Mathematics

• Blostein, D. and Grbavec, A.: Recognition of Mathematical Notation. In Handbook on Optical Character Recognition and Document Analysis, Wang, P.S.P. and Bunke, H. (eds) (1996) 557-582.
• Brown, C.M.L.: Comparison of Typing and Handwriting in "Two-Finger" Typists. Proceedings of the Human Factors Society (1988) 381-385.
• Fateman, R.: How Can We Speak Math? http://www.cs.berkeley.edu/~fateman/papers/speakmath.pdf.
• Kanahori, T., Tabata, K., Cong, W., Tamari, F., and Suzuki, M.: On-Line Recognition of Mathematical Expressions Using Automatic Rewriting Method. Proceedings of the IEEE International Conference on Multimodal Interfaces (ICMI'00) (2000) 394-401.
• Matsakis, N.E.: Recognition of Handwritten Mathematical Expressions. Master's theses, Massachusetts Institute of Technology (1999) Cambridge, MA.
• Miller, E.G., Matsakis, N.E., and Viola, P.A.: Learning from One Example Through Shared Densities on Transforms. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'00) (2000) 464-471.
• Microsoft.: Microsoft Word User's Guide Version 6.0 (1993), Microsoft Press.
• Smithies, S., Novins, K., and Arvo, J.: Equation Entry and Editing via Handwriting and Gesture Recognition. Behaviour and Information Technology 20 (2001) 53-67.
• xThink.: Math Journal (2003) http://www.xthink.com/MathJournal.html.

Multimodal and Recognition Technologies

• Blum, A. and Mitchell, T.: Combining Labeled and Unlabeled Data with Co-Training. Proceedings of the Workshop on Computational Learning Theory (COLT'98) (1998) 92-100.
• Clark, H. and Brennan, S.: Grounding in Communication. In Perspective on Socially Shared Cognition (eds. Resnic, L.B., Levine, J., and Sag, S.). APA, Washington, D.C. (1991) 127-149.
• Oviatt, S.: Mutual Disambiguation of Recognition Errors in a Multimodal Architecture. Proceedings of the CHI Conference (1999) 576-583.
• Oviatt, S., Coulston, R., and Lundsford, R.: When Do We Interact Multimodally? Cognitive Load and Multimodal Communication Patterns. Proceedings of IEEE International Conference on Multimodal Interfaces (ICMI'04) (2004).
• Oviatt, S., DeAngeli, A., and Kuhn, K.: Integration and Synchronization of Input Modes During Multimodal Human-Computer Interaction. Proceedings of the ACM Conference on Human Factors in Computing (CHI'97) (1997) 415-422.

Education, Cognitive Tutors, and Psychology

• Aleven, V.A.W.M.M. and Koedinger, K.R.: An Effective Metacognitive Strategy: Learning by Doing and Explaining with a Computer-Based Cognitive Tutor. Cognitive Science 26 (2002) 147-149.
• Anderson, J.R., Corbett, A.T., Koedinger, K.R., and Pelletier, R.: Cognitive Tutors: Lessons Learned. The Journal of the Learning Sciences 4 (1995) 167-207.
• Hausmann, R.G.M. and Chi, M.T.H.: Can a Computer Interface Support Self-explaining? Cognitive Technology 7 (2002) 4-14.
• Locke, J.L. and Fehr, F.S.: Subvocalization of Heard or Seen Words Prior to Spoken or Written Recall. American Journal of Psychology 85 (1972) 63-68.
• Sweller, J.: Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science 12 (1988) 257-285.

Contact

Main contact for this project is Lisa Anthony (email: ...@lisa-anthony.com).