Methodology

To try to expose any performance effects that could be due to machine architecture rather than to the code being tested, we used two different machines for benchmarking: a Sun SPARCstation 5/85 with an 85 MHz MicroSPARC2 processor and 32 MB of RAM, running Solaris 2.4; and a PC with a 120 MHz AMD 486 processor and 24 MB of RAM, running Windows 95. For compilation we normally used the GNU C Compiler 2.7.0 (gcc -O2) and the Java compiler from Sun's Java Development Kit (JDK) 1.0.2 (javac -O). However, for the PC native code benchmarks we used a third-party port of JDK 1.0.1, running on Linux 1.2.13. For the just-in-time compiler on the PC we used the JIT in Microsoft Internet Explorer 3.0 Beta 2. We also tried Netscape 3.0b5, but its JIT proved to have a much higher overhead and a slightly higher per-element cost on all of the benchmarks; Internet Explorer was 18-38% faster for the largest benchmark sizes. No JIT compiler was available for the SPARCstation.

We compiled the NESL source code into VCODE using version 3.1 of the NESL compiler [3], combined with an additional optimization phase that inlines VCODE functions and removes unnecessary stack operations. All benchmarks were performed on idle machines to minimize outside effects. This was particularly important for the Java benchmarks, because Java provides only a time-of-day clock (java.lang.System.currentTimeMillis()), rather than a per-process timer. The poor resolution of the PC clock also created problems. To obtain accurate timings of the benchmarks at small problem sizes, we timed multiple iterations of each benchmark, adjusting iteration counts so that each run took at least a second.

We ran the Java virtual machines with their default heap sizes, which resulted in some garbage collection taking place for all but the smallest of runs. To reduce these nondeterministic memory effects, we forced a Java garbage collection before the beginning of each timing run. This reduced the variance but did not eliminate it.