We use our mesh generator to create a mesh of the San Fernando Basin with a 220 m/s shear-wave velocity in the softest soil, assuming a seismic scenario with frequencies up to 1.6 Hz. Figure 3 depicts the nodes generated by the balanced octree. The octree produces 13 million nodes, but many fewer are shown for clarity. As can be seen from the figure, the density of nodes is highest in the softest soil, where the shortest wavelengths occur. A regular grid for this material model would have resulted in 200 times the number of grid points. A Delaunay tetrahedralization of the set of 13 million nodes produces a mesh of 77 million tetrahedra. The mesh is generated in 13 hours on one processor of a DEC 8400 and requires 7.7 Gb of memory. It has a maximum aspect ratio of 5.5 and exhibits a spatial resolution variability of over an order of magnitude. Figure 4 shows the resulting mesh of tetrahedra, again coarsened for visualization purposes. Next, the 77 million element mesh is partitioned into 256 subdomains in about 3.8 hours on one processor of the DEC 8400, and requires 7.9 Gb of memory. The resulting partition (for the coarser mesh and for 64 subdomains) is shown in Fig. 5. The figure shows the circular cuts characteristic of the partitioner. Despite the high spatial variability of the mesh, the partitions appear to be well-shaped.
Figure 3: Nodal distribution for the San Fernando Valley. Node
generation is based on an octree method that locally resolves the
elastic wavelength. The node distribution shown here is a factor of 12
coarser in each direction than the real one used for simulation, which
is too fine to be shown, and appears solid black when displayed.
However, the relative resolution between soft soil regions and rock
illustrated here is similar to that of the 13 million node model we
use for simulations.
Figure 4: Tetrahedral element mesh of the San Fernando Valley. Maximum
tetrahedral aspect ratio is 5.5. Again, for illustration purposes,
the mesh shown is much coarser than those used for simulation.
Figure 5: Mesh partitioned for 64 subdomains.
The last step of the pre-simulation sequential phase is parceling, i.e. generating the communication schedule, the global-to-local mapping, and the global matrix nonzero structure. On the DEC 8400, parceling requires about 2.3 hours and 7.7 Gb memory for the 77 million element San Fernando Basin mesh. The communication graph generated by the parceler is shown in Fig. 6. Each vertex represents a subdomain and corresponding processor; each edge represents communication between two processors.
Figure 6: Communication graph for the partitioned element mesh
depicted in Fig. 5.
