Symposium Schedule

Intelligent Multimedia Tutoring Systems

An Empirical Evaluation of a Case Argument Tutorial Program

CATO is an artificially intelligent computer program, designed to help first-year law students learn basic skills of analogical argumentation. CATO is based on a computational model of case-based legal argument, which addresses basic argument moves such as drawing analogies, distinguishing opposing cases, and citing counterexamples, and the use of these techniques as building blocks of more elaborate arguments. CATO provides an integrated set of tools, including a Case Database and an Argument Maker, that students can use to study the model in the context of specific problems. The CATO Case Database contains 150 cases in the area of trade secrets law, indexed by features that guide their use in arguments. CATO's Argument Maker produces (written) argument examples using cases from the Case Database, selected by the student.

CATO supports two types of exercises by students: to test theories (or hypotheses) about trade secrets law against cases, and to produce written arguments about trade secrets problems, supported by cases selected from the Database. Students evaluate their arguments by comparing them against arguments produced by CATO.

The main research issue is: How far can a pedagogical method based on the use of query tools and argument examples carry students' learning, even though the program cannot have a detailed understanding of students' natural language arguments? We conducted several evaluation studies to gather data on this issue.

In January and February 1996, we conducted a controlled experiment comparing instruction with CATO against more traditional classroom instruction. The subjects were 30 first-year students from the University of Pittsburgh Law School, randomly assigned to two groups. The experimental group students used CATO for nine 50-minute sessions, working in pairs. The control group students spent the same amount of time preparing for and attending six classroom sessions (in groups of 6-10 students) led by a legal methods instructor, during two of which they engaged in oral argument. All subjects took a written pre-test and post-test comprising argument-making problems. Both groups showed considerable improvement on the post-test. No significant difference was found between the two groups, either at the pre-test or at the post-test. This indicates that instruction with CATO was as effective as small-group classroom instruction.

In addition to the stand-alone methodology, we have also tested CATO in a role as "classroom aid." In the summer of 1995, a law professor used CATO to teach basic argumentation skills to a group of pre-law school students. The CATO output was projected on a large screen using an overhead projector. This lecture seemed to be well-received by students.

Academic Application Support - The Personnel Layer

After many years of working with faculty to incorporate computer-based simulation and communication in their teaching, it is now clear to me that we have failed to meet one of the necessary conditions for realizing the potential of computing for education: the creation of a new personnel layer to support faculty in their pedagogical applications of information technology.

We in academic computing are really suggesting changes in teaching paradigms that will make instruction more like a cinematic performance or publication of a weekly periodical than like the traditional lecture/discussion/textbook model that dominates today. No one would undertake to produce a movie or a periodical without an extensive production support team for the content providers. Yet, this is exactly what we have tried to do with instructional computing. My thesis is that for computer-supported instruction to become a significant feature of our educational system, the personnel structure of that system is going to have to change, at least in the short term, in a far more radical way than we may have ever thought necessary.

Supporting Secondary Science Students' Dynamic Model Building

ScienceWare, a three year project funded by the National Science Foundation, provides an integrated suite of computer-based tools to support learners as they engage in a full range of sustained inquiry activities: planning, investigating, data visualizing, modeling, reporting, collaborating and accessing information. This session will focus on one component of our work: helping students create dynamic models.

One of the hallmarks of science is modeling: constructing and verifying relationships that characterize natural phenomena. Computer tools enable scientists to build dynamic models. The modeling tools used by scientists, however, require much prior knowledge and mathematical ability, making the tools and process inaccessible to secondary science students. The challenge in making modeling accessible to students is to create a modeling environment which requires minimal prior knowledge, which incorporates advanced interface design, which requires no programming experience, which enables rapid generation of simple models, and which facilitates the learnerŐs transition toward more expert-like modeling practices.

The Highly Interactive Computing Group at the University of Michigan has created a modeling tool that can support students creating dynamic models--Model-It. Model-It provides dynamic modeling functionality in a software environment designed for learners. Model-It benefits from the interactivity and speed of computational media by allowing real-time model testing and smooth transitions between building and testing. The close linking of design and testing allows students to make connections between the relationships they designed into their model and the resultant representation of the modelŐs behavior. The photo-realism of Model-It adds to the motivation and ownership of students, as well as providing a concrete context in which to grapple with the abstractions of the relationships.

We have used Model-It with over two hundred science students in ninth and tenth grade and have conducted a series of classroom-based studies of students using Model-It. Our work indicates that when students use Model-It they engage in a range of cognitive strategies for modeling, such as analysis, synthesis, relational reasoning, and testing and debugging. Our work also indicates that the models students build are, for the most part, coherent in structure, realistic in behave, and accurate in content. However, our work also indicates that the instructional support surrounding the use of Model-It is critical to student success. Our work in classroom settings has resulted in a series of redesigning both the instructional components and the software. For instance, our previous work indicates that students did not focus on planning. In our fourth iteration of using Model-It with ninth grade science students, we have incorporated explicit instructional supports to help students focus on planning.

Cognitive Computer Tutors: From Research Lab to Classroom

Cognitive computer tutors for mathematics and programming have been under development at Carnegie Mellon University for over 10 years. Based on John Anderson's ACT-R theory of cognition, these tutors are problem solving environments that provide feedback and advice to students as needed. The tutors have proven to be useful basic research tools and effective classroom learning environments. The Pittsburgh Area Cognitive Tutoring (PACT) Center has recently been established at CMU to build on this work. The PACT Center is intended to support the widespread dissemination of cognitive tutors along with fostering continued research and development. This presentation will include discussions of:

• the role of student modeling in optimizing learning
• tutoring agents that provide intelligent support for work done in separate tool applications.
• classroom evaluations of the PUMP Algebra Tutor at the college and high school level
• PACT Center development and dissemination goals

New Paradigms for Computing, New Paradigms for Thinking

New computational paradigms (such as parallelism and distributed systems) provide not only new ways of solving problems, but (more importantly) new ways of thinking about problems. In this way, new computational paradigms can serve as both tool and metaphor for making sense of the world.

In this talk, I will discuss experiences with StarLogo, a massively-parallel modeling environment designed to help pre-college students explore the workings of decentralized systems (such as ant colonies, traffic jams, and market economies). Most people have great difficulty understanding such systems; they tend to assume that patterns can be formed only through centralized control. As students build models with StarLogo, they move beyond this "centralized mindset," learning how patterns can arise through decentralized interactions among simple components.

I will conclude with a discussion of a new project that enables groups of students to build StarLogo-like models collaboratively on the Internet. The idea is to take advantage of the decentralized architecture of the Internet to help students learn about decentralized systems.

Developing Interactive and Adaptive Instruction on the Web

World Wide Web (WWW) opens new ways of learning for many people. Now, educational programs and learning materials installed and supported in one place can be used by thousands of students from all over the world. However, most of the existing educational WWW applications use the simplest solutions and are much more weak and restricted than existing 'on-site' educational systems and tools (such as Intelligent Tutoring Systems). For many designers, the ideal form of educational WWW material seems to be a static electronic copy of a regular textbook. At the same time, interactivity is very important for any educational application as a prerequisite of active learning. Adaptivity is especially important for educational applications on WWW which are expected to be used by very different classes of users without assistance of a human teacher (who usually can provide adaptivity in a normal classroom).

My presentation will discuss the problems of developing interactive and adaptive "textbooks" on WWW. I present the system ELM-ART which combines the feature of a hypermedia-based electronic textbook and an Intelligent Tutoring Systems (ITS ). ELM-ART is a WWW-based system which supports learning programming in LISP. ELM-ART demonstrates how interactivity and adaptivity can be implemented in WWW electronic textbooks. ELM-ART can also serve as a good example to discuses the problems of implementing and porting ITS systems on WWW. It demonstrates that some ITS technologies, such as intelligent analysis of solutions and example-based problem solving can be ported relatively easily to a WWW context. It shows the importance of on-line course material and the technologies of adaptive hypermedia and curriculum sequencing which help students to navigate through this material.

We consider the problems of implementing ITS and other advanced educational systems on WWW as an important direction of research. WWW can help these advanced systems to move from laboratories to real classrooms. In the WWW context, a complex system can be located on HTTP servers in the research laboratories which have powerful equipment to run powerful intelligent systems and ITS professionals to support and update the systems. At the same time, learners from over the world can access these ITS using any WWW browser. These browsers require relatively cheap hardware and can run on almost any platform. It gives the advanced systems really world-wide audience and unlimited source of data for testing and improving their functionality.

Teaching in an Interactive, Hands-on Computer Classroom

cT: Multimedia Support within an Algorithmic Language (demo)

Computer-implemented "reciprocal teaching" of scientifically important thought processes

Remote Presentation Technologies

This talk will present recent advances in attempts to provide distance learning capability. Previous approaches to distance learning include Stanford Tutored Video Project, canned video lectures, CD Rom based multi-media titles, etc. Many of these have the problem of limiting student/professor interaction. The more recent attempts at video conferencing with telephony based queries have been widely adapted, but suffer from significant infrastructure needs and associated costs. In this presentation, we will briefly describe and demonstrate a Web based remote presentation technology, which appears to provide a number of advantages, such as Chat based mediated student to student and student to faculty interactions in a distributed time and space environment.

Intelligent Coached Practice Environments

The approach to training cognitively complex, technical tasks that we have shaped at the Learning Research and Development Center (LRDC) is one of learning by doing with reflection opportunities after tasks are performed. That is, we attempt to engineer work-like experiences in a simulated environment, providing coaching resources so that trainees are never "stuck" unable to complete a training task, and then provide opportunities to "replay" a record of task performance, critique that performance with the "computer" coach, compare it to expert performances, and ask questions about their own or a simulated expert's actions. We refer to intelligent tutoring systems based on this approach as "intelligent coached practice environments". This approach has been demonstrated to be effective when the tasks trainees undertake represent the hardest parts of the real job. In fact, the newer versions of our approach have achieved not only high levels of training on specific jobs, but also substantial transfer to new but related jobs (Gott et al., in press).

EAGLEKEEPER is an intelligent coached practice environment which was developed to train F15 Air Force flightline avionics technicians to troubleshoot complex electronic circuitry. Following SHERLOCK II, EAGLEKEEPER is the second generation of tutoring systems built upon the "LRDC Tutor Framework" (Katz et al., in press), a system architecture and collection of authoring tools for building coached practice environments developed at the LRDC.

Training systems built upon the LRDC Tutor Framework--e.g., the SHERLOCK tutors, and EAGLEKEEPER--contain several components. First, there is a model of the work environment within which training tasks are assigned (the devices workers use in their jobs). Second, there is a knowledge base sufficient to operate on the device model to derive competent methods for doing the assigned work (the expert model). Third, there are tools for providing a trainee with advice when he is stuck and unable to complete a task. These tools must be sufficiently active during the trainee's work that they can intervene when safety rules are broken or in other circumstances where the lack of safe and adequate progress may not be evident to the trainee. Fourth, there are intelligent graphical schemes for illustrating the structure and function of the simulated devices in ways that match well with expert thinking. Finally, there are schemes for displaying to the trainee a hyperdisplay of his task activities so that he can re-assess them and ask questions about them.

The LRDC Tutor Framework allows these various components to be built up separately and, to a large extent, independently. A relatively small "command processor" sits at the center of the system, coordinating its function. From one view the tutor looks like a specialized language parser. The Command Processor accepts and parses, and then dispatches commands to various other parts of the system for execution. The public interface to the Command Processor correpsonds to a synopsis of the high level behaviors supported by the tutor: changing activity modes (problem identification, problem solution, and reflection are the modes of our current tutors), make measurements, request advice, etc. Funneling all user requests through a central waystation has many benefits, including providing a starting point for organizing the design process.

The demonstration of EagleKeeper will focus on the system's main instructional features: user menus which reify a high-level problem- solving strategy, system coaching (accessible via menus and hypertext), abstract circuit diagrams to model the active circuitry, interactive video, and post-problem review activities. As viewer interest dictates, we will also demonstrate the main authoring tools which facilitated the development of the tutor: a diagram editor, hypertext editor, scripting tools (for expressing and compiling command "macros"), a simulation tool, and various user interface development tools.

REFERENCES

Gott, S.P., Lesgold, A., & Kane, R.S. (in press). Tutoring for Transfer of Technical Competence. AL/HR technical report. Brooks AFB, TX.

Katz, S., Lesgold, A., Hughes, E., Peters, D., Eggan, G., Gordin, M., Greenberg., L. (in press). Sherlock II: An intelligent tutoring system built upon the LRDC Tutor Framework. In C.P. Bloom and R.B. Loftin (Eds.), Facilitating the Development and Use of Interactive Learning Environments. New Jersey: Lawrence Erlbaum Associates.

Lesgold, A., Eggan, G., Katz, S., & Rao, G. (1992). Possibilities for assessment using computer-based apprenticeship environments. W. Regian & V. Shute (Eds.), Cognitive approaches to automated instruction (pp. 49-80). Hillsdale, NJ: Lawrence Erlbaum Associates.

The Advanced Cognitive Tools for Learning Project

The Iowa Electronic Market

Technology Enhanced Learning in an Ethics Course

Adding a probabilistic student model to your edutech application

Student modeling should be viewed as an aid for making pedagogical decisions. In some edutech applications, the software makes decisions, such as whether the student's action warrants a feedback message. In other applications, the teacher makes decisions, such as whether the student has met the instructional objectives of the application.

There is always uncertainty surrounding such pedagogical decisions. The evidence available to the student modeling module is rarely sufficient to unequivocally determine what its students are thinking and how much they know. Thus, there has been substantial interest in using recently developed techniques of probabilistic reasoning to solve the student modeling problem. Algorithms based on Bayesian networks have been especially popular.

This talk first presents general principles and problems with using Bayesian networks for student modeling, and illustrates them with simple generic edutech applications. It then covers a particular application of Bayesian networks being developed for a college physics tutoring system called Andes.

Capital Lab: Financial Decisions and Reporting

Capital Lab is a new computer network-based state-of-the-art software based program developed at Carnegie Mellon Unviersity to teach corporate financial decision-making, reporting, and stock market trading in an integrated environment. Traditional methods of teaching finance and accounting discretely focus on various important topics. Capital Lab is a hands-on approach to learning financial management and reporting decisions that has proved to be an invaluable tool for helping the participants develop an in-depth understanding of their complex interactions.

This goal is achieved through a sequence of sessions in which each member of the class is assigned a decision-making role such as chief executive officer, shareholder, or a member of the board of directors of various artificial firms in the computer laboratory. Participants make decisions appropriate for their respective functional roles. Members' decisions affect the payoff of one another, as they all seek their goals. Starting from simpler scenarios, the class advances to more complex scenarios that require the participants to weigh the pros and cons of various decisions in light of their interactions.

The session is divided into four parts: (1) instruction: learning the structure of the decision-making environment, (2) experiential: making the financial, reporting and trading decisions, (3) theory integration: linking the experience to theoretical concepts, and (4) Empirical assignment: take-home analysis of data.

Aspects of Decision-Making Covered in Capital Lab
Capital budgeting and investments
Dividends
Leverage and capital structure
Executive compensation
Financial reporting
Valuation of stocks
Stock market trading

Incorporating a Digital Video Library into High School Science Instruction

Instructional Systems Design: An Instructional Engineering Alternative

Instructional Systems Design (ISD), is a collection of methods derived from sources as diverse as systems engineering, cognitive and behavioral psychology, and military training. As opposed to exemplifying any one theory of instruction, the systems approach to instructional design is an engineering method. It has fundamental principles derived from its roots in systems engineering.

This analytical and design method has been received with only partial enthusiasm in the educational community, apparently due to its rigorous application of behaviorally observable measures and complexity implied in its implementation. The associated practices, procedures and methods of ISD are of paramount concern in the development of educational media for the digital world. The systems approach to instructional design has at its core one concept: system. For this reason, this approach to creating instruction represents an important milestone in the creation of instructional experiences for the modern era. Without the recognition of the elements influencing instruction and the instructional experience, educators can never fully optimize on the efficacy of instruction. The systems approach provides pathway out of this, often unrecognized, morass. In summary, the systems approach is not a "silver bullet" solution that guarantees instructional success, but establishes a pattern of instructional activities geared towards understanding the educator's intended outcomes in measurable terms, and creating experiences that are aimed at fulfilling the target objectives. The systems approach is a model that can permit educators to explain both the successes and failures of their educational ventures, and to respond to them in a manner consistent with systemic and scientific modes of explanation and empirical experimentation with their curricula.

For intelligent multimedia application development, the systems approach provides a gestalt for the consideration of the appropriateness of media to intended learning outcomes, a method of gathering and scoping requirements, and of ensuring an efficient focus upon the relationship of learning outcomes to the needs of the target audience' needs. This presentation will explicate the principles of instructional systems design and discuss their implementation in a computer-based interaction instructional system called "IDEAS" (Instruction DEsign-based Analysis and Synthesis).

IDEAS is a formal representation for a curriculum that is used for curriculum design as well as guiding intelligent tutors in lesson selection. In IDEAS, a curriculum consists of Lessons and Concepts. A Lesson generally presents information or demonstrates a skill and then evaluates the student's mastery. Units of knowledge and skill are called Concepts. Every lesson has a set of prerequisite concepts and a set of objective concepts. A concept may serve as prerequisite to many different lessons, and many alternate lessons may serve to teach a single concept. Thus, prerequisite and objective relations form a complex graph. IDEAS uses a set of graph analysis algorithms to check that a curriculum is well-formed, helping designers manage curriculum complexity.

A formal curriculum representation is used by IDEAS tutors to select lessons on an individual basis. IDEAS keeps track of the concepts a student has mastered and lessons that have been taken. It then looks for new lessons whose objective concepts make progress toward a knowledge goal. A hypothesis that we are currently investigating is that the syntactical representation of lessons and concepts in IDEAS is on the one hand simple and generic, but on the other hand sufficiently powerful for effective automatic tutoring.

Teaching with the Web: A Case Study in Course Evolution

Over the last few years, two related courses have been transformed from traditional blackboard lectures with handouts into courses using the Web for information delivery and student work.

The content of the courses is similar---using computers for engineering problem solving---but their goals are different. One uses standard productivity tools as the software development environment for engineering applications. The other is a project course; students attack real world design problems in a computer tool rich environment.

Initially both courses relied on traditional lectures and handouts that described assignments, projects and computer lab sessions. Student work was done on the machine, and submitted via diskette or file system. The first step changed the lectures to electronic presentation. In the next step, all assignments, solutions, laboratory instructions and data were put on the Web, and all of the lectures were repurposed for delivery via the Web. As the two courses share a common set of computer tools, a common course Web collection is being developed. To unify the material and Web use, an index and search engine have been added, and communications and student collaboration is via Web-based discussion forums.

In the project course, students form virtual engineering firms, responding to an RFP issued on the Web, finding out about the project via an on-line bidders forum, and submitting the project bid and all subsequent project work via their corporate Web site. In the programming course, student create personal Web sites with their work.

Using the Web to support the courses has required that a number of pedagogical, logistical and technological issues be addressed. Since the target student population is interested in the development and use of computer applications in engineering, the course evolves to track ongoing developments. Supporting all of the course materials and infrastructure has been challenging, both due to the scope and amount (over 800 pages in 40+ lectures and 40+ laboratory sessions) of the materials, the rapid change in the supporting technology, and the lack of good tools to support such teaching. Initial feedback indicates that the students are receptive to the courses in this on-line form.