Questions to be Answered
about the Implementation
of a Benchmark
These are the questions that we would like to know the answers to
about your implementation of one of the benchmark
tasks. Please send the answers to
bam@cs.cmu.edu.
There are many things you will need to keep track of while you are
learning the tool and while implementing the benchmark.
Be sure to read over the questions listed below to see what you need to pay
attention to. In particular,
be sure to keep track of:
- Time spent reading the documentation for the tool.
- What specific references you used to learn about the tool (names of books
or manuals, on-line documentation, etc.). Be sure to include complete
references.
- Exact amount of time spent doing the implementation.
- Please try to be quite accurate in timing how long this takes.
- During the implementation, keep track of how much time is spent:
- Thinking about the design.
- Actually typing in the code.
- Using interactive layout tools to draw parts of the interface.
- Going back to the documentation to look up things and figure out how
to do things.
- Debugging code to make it work.
- The amount of code written to make this benchmark work:
- For interactive tools, please count the number of lines of code
in the scripting language you had to type in.
- For example, the number of lines of Visual Basic code, or the number
of lines of Lingo.
- It might be helpful to print out the project and count them on the hardcopy.
- It would also be useful to have a count of the number of objects
created interactively.
- For example, how many objects were drawn with the editor?
Note that it may be impossible or very difficult to implement
the entire benchmark in the tool. In this case,
don't. Be sure to discuss what part you implemented in your
report.
Questions to be answered
In addition to the answers to the questions below, it would be fun
to receive a screen shot of the implementation of your application. GIF
format is preferred.
The questions are:
- Which Benchmark are you implementing
- What is the date that you performed this experiment?
- What is your name, phone, e-mail address, and physical mail address?
(optional)
- Do you want your name listed in the acknowledgements of reports of
this study, or do you want to be anonymous?
- What hardware did you run on?
- What tool and operating system, etc. did you use? Which versions? (e.g.,
"I used Visual Basic
Version 4.0 on Microsoft Windows 95 using a PC with a 150mhz Pentium").
- What kinds of programs do you think the tool you used are best suited
for?
(I.e., are they best for graphical editors, text editors, form-filling programs,
multi-media games, etc.?)
- If the specification of the benchmark leaves anything unspecified,
what is your final design? (Please specify fully.)
- Did you implement anything not in the specification? (presumably,
because your tools gave you this capability for free -- don't spend time
adding extras that are not specified.)
- Were there any features that you could not implement because they
were too difficult? Which ones and why?
- How familiar were you with the tool before starting the
implementation? (Expert, familiar with it, never used it before). If
experienced, how much have you used the tool before?
- What books or documentation did you use to learn the system? (Give
complete
references if books.)
- What was the total time spent doing this benchmark?
Please
break the time down into:
- Learning time before the implemention began?
- Time to do the implementation?
If possible, please further break the
implementation time down into:
- How long did it take to design the implementation (thinking time)?
- How long did it take to use interactive layout tools to draw parts of
the interface?
- How much time did you spend going back to the documentation to look up
things and figure out how
to do things?
- How long did it take to type in the code of the program?
- How long did it take to debug the program?
- How many lines of code is the resulting program (excluding
comments)?
- For interactive tools, please count the number of lines of code
in the scripting language you had to type in.
- For example, the number of lines of Visual Basic code, or the number
of lines of Lingo.
- It might be helpful to print out the project and count them on the hardcopy.
- How many objects were created interactively?
- For example, how many objects were drawn with the editor?
- How many bytes long is the source code, i.e.,
what is the size of the source file in bytes on the disk? If the source is
in multiple files, please add them all together.
- How many bytes is the resulting binary (compiled) code?
- Please rate and discuss the quality of the tool you used along
the
following dimensions:
- Ease of learning how to use the tool.
- Appropriateness of the tool for creating applications like the benchmark.
- Ease of creating the benchmark.
- Quality of the implementation resulting from using the tool.
- Discuss the good and bad points and features of the tool in general.
- How would you rate the run-time performance of the
resulting implementation? (Does it run fast enough to be usable?)
- How would you rate the quality of the implementation created while
using the tool ? (e.g., is it well-structured, would it be easy to modify
and maintain, etc.)
- What features of the benchmark were easiest to create using the tool?
- What features of the benchmark were most difficult to create using the
tool?
- What changes to the tool would have made it easier to handle
the difficult parts?
- What was the most significant source of bugs in the code? (What were
the hardest things to debug?)
- Please evaluate the benchmark task itself along the following
dimensions:
- Is the benchmark sufficiently specified that you knew what to
implement?
- What is too vague or unclear in the benchmark specification?
- Is the benchmark a good test of this tool?
- Any other comments on the quality of the toolkit you were using
or on this benchmark task?
Please send the results back to
bam@cs.cmu.edu
Thank you for your participation.
Back to Benchmarks main page