Civil engineers at Carnegie Mellon are using Archimedes to build
finite element simulations that model the ground motion during
earthquakes in the San Fernando Valley of Southern California
~\cite{superquake96}.  The running time of each simulation is
dominated by global sparse matrix-vector product operations. Each global sparse
matrix-vector operation consists of a computation phase, where each
processor performs a local matrix-vector product, followed by a
communication phase, where processors that share mesh nodes exchange
data over a network.

The largest earthquake simulation constructed to date is based on a
mesh with 13.4 million nodes and 76.7 million 3D linear tetrahedral
elements.  The mesh was generated automatically from a geological
database of the Los Angeles Basin~\cite{magistrale96}. The simulation
runs on 256 processors of a Cray T3D, with a sustained rate of 35
MFLOPS/processor during the global sparse matrix-vector product
operations, and an overall sustained rate over the entire execution
time of 14 MFLOPS/processor.

Figure~\ref{fig:scale} shows the scalability of one of the smaller San
Fernando earthquake simulations, which is based on a 3D mesh with
378,747 nodes and 2,067,739 elements.  In general, computation time
decreases linearly, and communication time decreases steadily,
although not quite linearly, with the result that total running time
scales almost linearly.  In each case, communication time represents a
small fraction of the total running time, even though Archimedes uses
a simple subset of MPI (Message Passing Interface) and makes no
attempt to use optimal communication schedules. The sustained
computation rate on each processor during global matrix-vector product
operations is around 30 MFLOPS, which is quite good, so the small
communication fraction is not due to artificially slow computation.
The main point is that communication overhead does not need to be a
big issue for finite element simulations that run on modern parallel
systems.

%
% Here is how I included the figure 
% (/iwarp/member/droh/papers/cacm95/scale.epsf)
% when I tested including it with Latex:
%
\begin{figure}[htb]
\centerline{
\epsfbox{scale.epsf}
}
\caption{Scalability of the San Fernando Earthquake Simulation on the Cray T3D}
\label{fig:scale}
\end{figure}


%
% Here are the bibtex entries for the quake project and the LA database
%

@inproceedings{superquake96,
author= "Hesheng Bao and Jacobo Bielak and Omar Ghattas and David R. O'Hallaron and Loukas F. Kallivokas and Jonathan R. Shewchuk and Jifeng Xu",
title = "Earthquake Ground Motion Modeling on Parallel Computers",
booktitle = "Proceedings of Supercomputing '96",
year = 1996,
month = nov,
address = "Pittsburgh, PA"
}

@article	(magistrale96,
author	=	"H. Magistrale and K. McLaughlin and S. Day",
title	=	"A geology-based 3-{D} velocity model of the {L}os {A}ngeles basin sediments",
journal	=	"Submitted to the Bulletin of the Seismological Society of America",
month	=	"" ,
year	=	"1996" ,
volume	=	"" ,
number	=	"" ,
pages	=	"" ,
note	=	"" ,
)