\section*{Experience and Conclusions}

The events that led to the development of Archimedes are interesting.
In late 1991, civil engineers at Carnegie Mellon began exploring the
use of parallel computers for modeling earthquake--induced ground
motion. They manually implemented an explicit solver on the Thinking
Machines CM-2 using an unstructured mesh.  The code ran slowly because
the processors spent almost all of their time communicating with each
other over the relatively slow global communication router.  So they
rewrote the code, this time creating a block--structured mesh and
painstakingly partitioning it over the processors to try to minimize
the use of the relatively slow router.  The fast nearest--neighbor
communication system was used for the regular communication patterns
within each block, while the global communication system was invoked
to ``stitch'' the blocks together.  The resulting code exhibited an
order-of-magnitude improvement in Mflop/s, due to the substantial
reduction in global communication.  Unfortunately, because the
gradation of the block-structured mesh was insufficiently flexible,
smaller elements than desired were generated in the rock surrounding
the valley. These elements were critical with respect to the stability
condition. Thus, the solver was forced to take many more iterations
than the physics required. Switching to an implicit solver was
problematic, for it entailed a complete rewrite of the code. Parallel
computers, rather than a tool, were becoming an impediment to doing
science!

Frustrated by the painstaking level of detail required to create
reasonably efficient programs, the engineers sought out computer
scientists at Carnegie Mellon for assistance. During these
discussions, we realized that by exploiting exploiting prior work in
mesh generation, VLSI placement and routing, and our own ongoing work
in graph partitioning and parallelizing compilers, it would be
possible to fully automate the irregular finite element modeling
process. Thus was born the collaboration that produced Archimedes.

Since its original implementation for the iWarp and CM-5 systems, we
have retargeted Archimedes to the Intel Paragon and DEC Alpha
workstations.  As other parallel systems become available we will
retarget Archimedes to these as well. It generally requires a week or
two of effort to retarget the system. 

We have used Archimedes to duplicate the original 3D seismic ground
motion study, and the engineers are currently using Archimedes on a
production basis to model earthquake--induced ground motion in
moderate--sized 2D and 3D basins~\cite{bielak95}. Ultimately,
Archimedes will be used to model the entire Los Angeles basin. 

Our experience with Archimedes and the algorithms described in this
paper has taught us a number of things.  We have learned that it is
feasible to construct automatic tools for solving unstructured PDE
problems on parallel systems and that it is possible to for automatic
tools to generate efficient parallel code.  For example, the 3D
seismic ground motion simulation produced by Archimedes for the iWarp
system spends only 20 percent of its time transferring data between
the processors; the remaining 80 percent of the time is spent doing
useful work. However, important issues remain, including automatic
generation of quality 3D meshes, parallelization of the mesh
generation and partitioning algorithms, run--time methods for
dynamically changing meshes, and techniques for debugging very large
finite element models.


