\begindata{text,538588168} \textdsversion{12} \template{default} Media Technology Syllabus 15-499C/15-820 Roger B. Dannenberg 16 January 1996 Class meets: 10:30 to 11:50, Tuesday and Thursday, in HHB131. \ General information for the class will be posted to academic.cs.15-820d and/or the class web page: http://www.cs.cmu.edu/~rbd/mt96.html (soon, this will be available via my home page or the SCS home page). Instructor: email: dannenberg@cs.cmu.edu phone: 268-3827 office/lab hours: Monday 1:30 - 2:30 or Friday 11 - 12. office/labs: Wean Hall 3214, 3213 (music lab), or 3215 (multimedia lab) \ Please drop in at office hours, or make an appointment. (The best way to make an appointment is send mail at least 24 hours in advance with a few times you can meet. I'll pick one and get back to you within 24 hours. To schedule with less advance notice, it's best to call.) Teaching assistant: Christian Hoffman email: cdhoff+@maps.cs.cmu.edu office and hours TBA Text: Ralf Steinmetz and Klara Nahrstedt. Multimedia: Computing, Communications, & Applications, plus supplemental readings of about 5 articles. (To be announced.) Also, if you miss handouts in class, my secretary, Kay Kowalsky (email: kk00@andrew.cmu.edu, phone: 268-5885, office: Wean Hall 3216) will have copies. Description: This course teaches the fundamentals of media representation, storage, communication, and processing by digital means, with an emphasis on audio, still images, and video media. It includes an introduction to sampling theory and various representation techniques. This is used to describe and explain a variety of real devices, formats, and standards. Students will learn to analyze media technology in terms of critical properties such as resolution, noise, bandwidth, latency, and computation. Topics to be covered include: audio sampling theory and processing, 2D image sampling, color spaces, audio compression, image compression, video compression, storage devices including CD-ROM and RDAT, storage formats including CD-I and HyTime, time codes and synchronization, communication technology including ATM, ISDN, and cellular radio, input technology, and application examples. A premise of this course is that once you understand how media is represented and processed, all the other topics can be viewed in that light. Most of the technology presented can be seen as engineering solutions to problems that arise from fundamental principles. If time runs short, we will cut back on topics and allocate more time to master the fundamentals. Goals: At the end of the semester, you should be able to: 1. Describe the relationships between sampling parameters, quantization noise, and bandwidth. 2. Explain how to convert from one sample rate to another, and be able to analyze the resulting distortion or information loss. 3. Give several examples of data compression used for continuous media and compare the techniques. 4. Determine what issues and parameters are likely to be critical ones in the design and implementation of a digital multimedia system. 5. Describe a variety of devices and techniques used for media storage, communication, and presentation. Evaluation Criteria: There will be a midterm exam (20%) and a final exam (25%). There will be 3 small-to-medium sized projects (10% each). There will be a number of homework assignments (25% total). \begindata{bp,538646776} Version 2 n 0 \enddata{bp,538646776} \view{bpv,538646776,4,0,0} Lecture Schedule and Topics: Notes: \ Page numbers refer to text reading assignments. Topics are subject to reordering. "Just In Time Lecture" means the lecture is on CD-ROM on reserve in the E&S library. Instead of coming to class, watch the lecture. Preliminaries (Jan 16) READ CHAPTERS 1 & 2 Syllabus, Projects, Grading, Reading Topic and Course Overview What is Multimedia? Temporal media properties. Example Systems (Jan 18) READ 17.5 through 17.7 Minitel, Acme/Tactus/Rowe, Maps on CD-ROM, Video Mail, Piano Tutor, Interactive Composition, Computer Graphics, Metamail, Cognitive Coprocessor Languages and Environments (Jan 23) READ CHAPTER 16 Timeline Editors, Flowchart Programs, Hypermedia, Threads, Visual Basic/C++ \ OLE, Apple Events Networks and Information (Jan 25) MIME, World Wide Web, MBone The Frequency Domain (Just In Time Lecture, No Class, Jan 30) Introduction The Fourier Transform Signals and Sampling Theory (Feb 1) Convolution Theorem Sampling and the Nyquist Theorem Foldover/aliasing, Prefiltering Quantization noise - related to number of bits Bandwidth - related to sample rate More Sampling Theory (Feb 6) Dither Oversampling More terms and concepts Representation and Manipulation READ PP. 27-32 Audio (Feb 8, 13) Introduction Mixing, Filtering and Equalization, Sample Rate Conversion Interconnect AES/EBU, S/PDIFF Formats and Standards: AIFF -> (Project 1 due Feb 15) READ PP. 55-61 Image (Feb 15, 20) 2D images as signals \ Dither in images 2D Resampling Color \ Image Processing \ Formats and Standards: TIFF, GIF Exam (Feb 22) More Representation and Manipulation -> (Project 2.1 due Feb 27) Video (Feb 27) READ PP. 81-103 Encoding 2D images in 1D signals CRT technology Scanlines, Vertical Blanking Interval, Vertical Retrace, Interlace Digital Display Systems Video RAM, Frame Buffers Animation Technology \ READ PP. 103-112 Page Flipping, Color Cycling \ Cel Animation, Object Animation Typography \ Formats and Standards, Postscript -> (Project 2.2 due Mar 5) MIDI (Mar 5) READ SECTION 3.2 Formats and Standards, Standard MIDI Files Compression Data (Mar 7) READ 6.1 through 6.4 Huffman Coding, Quantization Music Music Notation, Performance Information Audio (Mar 12) DPCM, ADPCM, Masking, Auditory Transform, Physical Models, LPC Image (Mar 14) READ 6.5, 6.6 CLUT \ Run-length encoding, JPEG, Fractal Animation Object descriptions, Inter-frame differences -> (Project 2.3 due Mar 19) Video (Mar 19) READ 6.7 through 6.9 RGB, Composite, MPEG, Software Codecs Computer Music Synthesis (Mar 21) FM, Sampling, Physical Models -- Spring Break -- Output Technology (Apr 2) LCD Displays, 3D TV, Haptic Displays Storage Technology Magnetic Media and RAID (Apr 4) READ SECTION 9.5 -> (Project 3.1 due Apr 9) Optical Media (Just In Time Lecture, No Class, Apr 9) READ CHAPTER 7 Optical Disk, CD/CD-ROM/CD-I, Videodisc, Writable Optical Input Technology (Apr 11, Apr 16) Mouse, Tablet, Scanners Video Input Touch Sensors Speech, Radio Drum/Polhemus Isotrak VideoHarp/Sensor Frame, Data Glove/Power Glove Eye Tracking, Gesture Recognition, Vision (Project 3.2 due Apr 23) Communication (Apr 18, 23) READ CHAPTER 10 ATM, Isochronous, ISDN, BISDN, FDDI, Cellular Radio Synchronization (Apr 25) READ CHAPTER 15 MIDI Time Representation \ SMPTE \ HyTime, Vertical Blank Interval, Timebase Correction \ Audio Samples, Tactus, Stream-based systems (Project 3.3 due Apr 25) Audio and Video Production (Apr 30) Review for Final (May 2) \begindata{bp,538671944} Version 2 n 0 \enddata{bp,538671944} \view{bpv,538671944,5,0,0} Projects: Warning: Many of you will be taking other courses with project work, and you will be tempted to do one or more big projects to satisfy requirements in each course. You may not turn in a project for this course if you are getting credit for it in another course, unless you work out grading criteria in advance to the satisfaction of all instructors involved. Project 1: Multimedia/Hypermedia Authoring The goal of this project is to gain experience building an application with a high-level authoring environment or language. You will use Authorware to develop a presentation and interactive exercise on some topic of C++ programming. Each student will be given a separate topic with a basic lesson design. (Due Feb 20) Project 2: Quality of Service Experiment Digital media offer great flexibility with regard to media representation. In particular, various sample rates and quantization levels can be used to trade off quality against storage, latency, computation, and/or bandwidth. This project is intended to give you first-hand experience with these tradeoffs. Step 1 (Due Feb 27): Obtain a sound or image and (dis)play it on a machine of your choice. See the instructor if you have problems. Turn in a description of your example and why you chose it. For example, you might have a voice recording because you are interested in quantization applied to telephony. If you write more than a paragraph, you are working too hard. Step 2 (Due Mar 5): Design an experiment to study the effects of quantization. The experiment should be in one of two forms: Form 1: evaluate a tradeoff such as: color resolution vs. image resolution, sampling rate vs. bits/sample, log vs. linear samples, or dithered vs. undithered images. Example: how much extra resolution at 1 bit/pixel gives an image quality equivalent to 2 bits/pixel? Form 2: Measure the amount of added noise (expressed as a fraction of maximum signal amplitude) that gives an equivalent subjective quality to a degraded signal. Example: determine what level of noise must be added to 16 bit audio before it degrades to the quality of 8 bit audio. Hint: In both forms, you make subjective comparisons between two images or sounds and adjust some parameter until you get subjectively equal quality in the two stimuli. Be sure you know what parameter you are varying. ("I turned the Photoshop blur factor to 4" is not a satisfactory explanation.) Turn in a description of the experiment. One or two paragraphs is fine. Step 3 (Due Mar 19): Perform the experiment and write up the results. One page should be sufficient. Be sure to compute and compare the number of bits in the different versions. In most cases, your writeup will include a graph, for example showing the subjective quantization noise as a function of bits/sample, or showing the required image resolution to maintain constant quality as a function of color resolution. Project 3: Digital manipulation of audio, image, or video data. \ In this project you will implement software to manipulate digital media. Examples include: digital reverberation for audio, resampling or special effects for image data, using edge-detection to convert images to line drawings or video to line animation, color quantizing or color correction, pitch detection in audio, pattern recognition in images, motion detection in video, and music synthesis. Since the Project 1 involves reading, processing, and outputing media, you are encouraged to extend that work rather than starting from scratch. \ Step 1 (Due Apr 2): Determine what you wish to implement. Turn in a one paragraph description. Be creative. Step 2 (Due Apr 23): Implement the effect. Turn in (1) an image or a pointer to a sound file, (2) a program listing, and (3) a copy (or revision) of the description from Step 1. \enddata{text,538588168}