Homework 3: Evaluate Designs for Scrolling

Assigned: Thursday, Feb. 17, 2022; Due: Tuesday, Mar 1, 2022 at 1:25pm ET

Overview

In this assignment, each student will evaluate three (3) different designs for scrolling using three (3) different people, and write up a report on the results. Note that this assignment is quite similar to Homework 1, but we have increased the number of techniques you need to do (but it only takes <5 minutes per device). Students will enter the devices they will evaluate into a GoogleDoc, and run the Scrolling Test using our web app.

Scrolling Techniques

Each student will choose three (3) different scrolling techniques. We hope that we can get good coverage of all of the kinds of scrolling techniques that are currently in use, with approximately the same number of people doing each one. If you have an “interesting” way of doing scrolling, please use that as one of your choices. You should enter the devices you will do into this GoogleDoc. (Note that you need to be using your CMU Andrew Google account to access this.)

For example, here are some scrolling techniques we thought of that you can pick from:

• A regular scroll bar on a desktop or laptop, using only the arrow buttons at each end
• A regular scroll bar on a desktop or laptop, only by dragging the “thumb”
• A regular scroll bar on a desktop or laptop, letting the user choose how to use it
• A mouse wheel with “notches”—the user feels each scrolling increment
  • You can borrow one of my mice if you don't have one
• A mouse wheel without “notches”—scrolling is smooth
• A mouse wheel or the pointing stick on a desktop or laptop—the user first press down and then drags up and down to scroll (a “rate controlled” scroll)
• A two-finger scroll on a laptop touchpad
• Scrolling on a desktop or laptop using the keyboard arrow keys (and optionally also with the “PgUp” and “PgDn” keys—at least for Windows)
• Drag scrolling (only) on a phone or tablet
• A “flick-scroll” on a phone or tablet (which needs to include drag scrolling for fine-tuning)
• Scrolling on a non-touchscreen tablet like an eReader that is not touch sensitive, using whatever kind of scrolling it supports
• other—what other scrolling techniques can you get access to?
  • One or more students can borrow my Kensington trackball with a scroll wheel, or my Contour "RollerMouse Red plus" which also has a scrolling mechanism
  • Game controller connected to a PC to control the scrolling?

Extra points (up to +3) on this assignment for doing a fourth or more techniques! Also, extra credit (up to +2) if you test a really interesting scrolling technique, like a VR controller, game controller, etc.

Everyone enter the techniques you will do here.

Participants

Every student should test three (3) people: (it is OK to use the same people you used for Homework 1):

• Yourself—you should use the three scrolling techniques yourself..
• Someone who is like you - for example, another student, but not anyone in the class.
• Someone who is not like you—for example, someone your parent’s age, a child, someone with little or no computer skills, etc. (Note that a person who uses a PC if you use a Macintosh is not sufficiently different!) Be sure to note in your report which person you are counting is not-like-you, and why.

Extra points (up to +3) on this assignment for testing a fourth or more people!

Test Software

The professor and TA are creating a version of scrolling test software for the web. This software is brand new this year! It is based on the final project from IxT 2019 by Cem Ergin, Emily Porat, Sijia Li, and Chun Wang.

You will be able to run the software from any web-enabled device. The software by default will display pages full of content, and users will need to scroll down to find the appropriate line, and get that line inside the grey box at the top. The page will then record the times. For your three people, use different orders for the scrolling techniques. Be sure the mention the orders in your report. (If you do extra techniques or extra people, then still have everyone do all the techniques, with different orders for the different people.) Note that it only takes about 5 minutes to run each person for one technique through the whole test, so it should be less than 15 minutes per person.

Make sure the participants read the instructions on the first page, so they know what to do.

The test uses Google Sheets to record the answers. Happily, we created the scrolling test app a different way from homework 1, and now Google doesn't require you to log in or to confirm that you are ok sharing your data, but it is still saved behind the scenes.

Click here to run the scrolling experiment software
(Please let the Professor and TA know of any bugs or issues with the software.)

Design of the Test

We have tried to make sure the content is always about 10 screenfuls, so lots of scrolling will be needed. The active area is square. For regular computers, please make the window full-height and width of the screen. For phones and tablets, please use portrait mode, so the longer dimension is vertical, since this is the normal way that these devices are used.

For most kinds of scrolling on a regular computer, the pointer cursor must be inside the active square scrolling region. For example, to use 2-finger scrolling on the touchpad, or the mouse scroll wheel, the user must move the cursor into the active area after clicking on the start button. It does not work to leave the cursor on the start button, or in the black or grey areas.

The "formal test" does 20 different scrolls in each of 2 phases (40 total scrolls). The target lines are the same for the 2 phases, and are the same for all participants: 11,11,20,20,30,30,40,40,50,50,60,60,70,70.80,80,90,90,99,99. However, the order is randomized for each run, so users should not be able to guess which one they are doing or what is next, etc. We start with line 11 so the target is never visible before starting, and we end with 99 (the last row). Each value is used twice.

The two phases of the test are:

1. Somewhere on the page are a row of stars. These should be scrolled to be in the target area. This tests for the situation when the user needs to locate something by visually scanning the document.
2. The user is told the specific line to jump to. This tests for the situation where the user knows (at least approximately) where the content they are looking for is. Examples might be scrolling an alphabetical list for a known last name, or remembering that the content you are looking for is at the top or bottom of the document. Note that the scrolling pegs at the end, so 99 should be especially easy to get to.

The target area is designed to be about 1.5 lines tall, so it should be possible to get the target line into the area. Note that if using a mouse wheel or equivalent scrolling devices, the increment may need to be adjusted to be "1 line". (My scroll wheel was originally set to 3 lines, and it kept skipping over the target area, both going up and then back down!)

Preference and Demographic Questionnaire

Same as for homework 1, you should create a simple questionnaire to give each of your subjects (including yourself!) to be filled out at the end after doing all the tasks. It should ask basic demographic information (see report below), experience with computers and experience with the scrolling techniques you are using. Then, you should have question(s) about their preferences among the scrolling techniques they used and why. You need to include a blank copy of the questionnaire with your report so we can see what you asked.

Report

The deliverable for this homework is a written report. We expect it to be maybe about 4-5 pages (not including figures or the copy of the questionnaire), single-spaced in TimesRoman 12 point font. Create a pdf of this report, and upload it to Canvas before class on the due date.

Be sure your name is on the first page of your report and in the file name! Your pdf file should be named: lastname_firstname_ixt22_hw3.pdf. For example, mine would be: myers_brad_ixt22_hw3.pdf.

The report should have the following sections. (Note that you should use this as a template using these section names). Please put your information into the appropriate section as described below, so it is easier to grade.

• Description: A detailed description of your three scrolling techniques. This should be sufficient to allow the reader to reproduce your test exactly. For example, if doing a scroll bar with a mouse on a PC, what is the screen size (in inches) and what kind of mouse were you using? What mouse settings did you use on the computer? What browser did you use to run the test? If doing a Smartphone, what exact screen resolution and screen size (diagonal)? (For example, you can often find these by searching Google for "screen resolution" -- e.g., Apple says the "Display" on the iPhone 13 pro max is: "6.7-inch (diagonal)" and has "2778-by-1284-pixel resolution at 458 ppi"). Also measure the size of the scrolling area (width and height in inches) on the screen. It would also be great to include screenshots or photos of your scrolling techniques.
• Users: A detailed description of your users. (Note: not their names!) Describe all the relevant demographic characteristics, like their experience with computers, their experience with scrolling techniques you used, whether they are left-handed or right-handed (and if left-handed, did they use their left hands—many left-handed people still use computers right-handed), etc. This section of the report should include anything of interest from your questionnaire, but also your observations about the user.
• Environment: A detailed description of the environment in which you did the test. In particular, were the subjects seated at a desk, standing, at a table? If a phone or tablet, was it on the table or in their hands? (You could even test both if you want and see if there is a difference.) If testing a smartphone, how was it held and scrolled? Also include the order of the techniques used for each user and explain why you chose that order. If your participants are remote, then ask them to show you their setup and describe the environment so you can write these into the report.
• Results: Then present your results (note that you should address each of the following):
  1. Present the actual measured results for the total time required for each of your scrolling techniques for your users and the averages for each technique across the users, etc. (Note that the individual scroll times are not that interesting—use the total time that the person took with that technique.) Which technique was fastest?
  2. You should check if there are differences for short scrolls vs. long scrolls—that is, do your comparison of techniques separately for the total time for all short scrolls, for all long scrolls and for all scrolls (total). For example, when the target line is 90 or 99 might be considered long scrolls (there are 8 of these), versus target line being 11 or 20 being short scrolls (there are 8 of these as well). Does the time differ between the first phase (target unknown) and second phase (target known)? Note that the target lines are presented in a randomized order, so you might need to sort or filter your data based on the target-line in the data files.
  3. You should see if there are differences for unknown scrolls vs. known scrolls—that is, do your comparison of techniques separately for the total time for the first phase, when the user did not know how far to go, versus the second phase when the user did know how far they would have to go. Are there any differences? Did users try different strategies for longer scrolls vs. shorter scrolls when they knew how far they had to go?
  4. With pointing, there is a relation between the distance between targets and the time, which is logarithmic. Can you find any relationship between scrolling distance (target line) and time? Is it different for the first 20 (unknown) versus the second 20 (known)?
  5. Which scrolling technique seemed the most accurate? The only accuracy measure we have is the number of overshoots - when users scrolled past the target and had to backtrack, which can happen multiple times in both directions. This is named "switchbacks" in the data, since it counts overshoots in both directions. How did the number of switchbacks vary by scrolling technique, or by long vs. short scrolls, or by unknown vs. known target locations?
  6. Which scrolling technique did the subjects prefer and why? (Note that this should be included in your questionnaire.)
  7. Was there any effect of the order in which people used the techniques (did it seem like there was a learning effect happening)? Since participants will be doing the test multiple times, they are likely to get better at it.
• Discussion: Next in your report, have a discussion section that shows that you have command of the following topics from the class syllabus, with respect to the scrolling techniques you studied:
  • Comparative Performance: How well did your scrolling techniques perform compared to each other? Would you recommend one kind of scrolling technique over all others in all situations? If not, what does it depend on?
    • The literature I am aware of combines scrolling and then pointing, so it isn't a fair comparison between the literature and our results here, but is there anything you can say about speed or accuracy of the techniques you tested, compared to other scrolling techniques in the literature? For example, in the following 2 articles. (Extra credit -- up to +5 -- for finding any other articles about scrolling that discuss speed and/or accuracy and summarizing how they compare to your findings.)
      • William Buxton and Brad Myers. “A Study in Two-Handed Input,” Proceedings SIGCHI '86: Human Factors in Computing Systems. Boston, MA. April 13–17, 1986. pp. 321–326. pdf or html and video. (discussed by Bill Buxton in his guest lecture)
      • Brad Myers, Kin Pou (“Leo”) Lie and Bo-Chieh (“Jerry”) Yang, “Two-Handed Input Using a PDA and a Mouse”, Proceedings CHI'2000: Human Factors in Computing Systems. April 1–6, 2000. The Hague, The Netherlands. pp. 41–48. local pdf.
  • Design Issues: The design issues regarding the different scrolling techniques, especially the trade-offs among them. For example, discuss whether some scrolling techniques are better for short scrolls compared to longer scrolls. Or whether one method is better or worse when the target is in a known vs. unknown place.
  • Appropriate Evaluation: How well did the evaluation of the scrolling methods work? Were you using the appropriate tests for performance? Were the tests you used the right ones? What did the tests miss (not measure) relevant to scrolling? Note that this is a brand new scrolling test, so there are likely to be a number of issues and limitations with it.
• Future Work: Finally, you should have section of future work, which should discuss new studies or improved ways of doing scrolling.
• Appendix: Questionnaire: Include a blank copy of your questionnaire (just like it was given to the participants) in an appendix

Grading

The various parts of this homework will be calculated as follows, so you know how much each part is worth:

• 9 points: Overall following instructions
• 15 points: Description, Users, Environment sections of report
• 50 points: Results
• 20 points: Discussion
• 3 points: Future Work
• 3 points: Appendix
• Extra Credit (up to these values)
• +3 points: if test an extra person or people (total, no matter how many extra people)
• +3 points: if test an extra scrolling technique or scrolling techniques (total, no matter how many extra devices)
• +2 points: if test a really interesting device, like a VR controller, game controller, etc. (total no matter if multiple devices were interesting)
• +5 points: discuss other articles (beyond the 2 listed) about scrolling that discuss speed and/or accuracy and summarizing how they compare to your findings