Contemporary High Performance Computing

From Petascale toward Exascale, Volume Two

Jeffrey S. Vetter

• Publisher: Chapman and Hall/CRC
• Year: 2017
• eBook ISBN: 9781351103923
• Topic: Computer Science
• Subtopic: Computer Engineering
• Index: Computer Science

HPC is used to solve a number of complex questions in computational and data-intensive sciences. These questions include the simulation and modeling of physical phenomena, such as climate change, energy production, drug design, global security, and materials design; the analysis of large data sets such as those in genome sequencing, astronomical observation, and cybersecurity; and the intricate design of engineered products, such as airplanes and automobiles.

This second volume of Contemporary High Performance Computing: From Petascale toward Exascale continues to document international HPC ecosystems, including the sponsors and sites that host them. Each chapter is punctuated with a site's flagship system and:

• Presents highlights of applications, workloads, and benchmarks



• Describes hardware architectures, system software, and programming systems



• Explores storage, visualization, and analytics



• Examines the data center/facility as well as system statistics

Featuring pictures of buildings and systems in production, floorplans, and many block diagrams and charts to illustrate system design and performance, Contemporary High Performance Computing: From Petascale toward Exascale, Volume Two delivers a detailed snapshot of the rich history of practice in modern HPC. This book provides a valuable reference for researchers in HPC and computational science.

Chapter 1

Terascale to Petascale: The Past 17 Years in High Performance Computing

Jeffrey S. Vetter

Oak Ridge National Laboratory and Georgia Institute of Technology

Bibliography

High Performance Computing (HPC) is used to solve a number of complex questions in computational and data-intensive sciences. These questions include the simulation and modeling of physical phenomena, such as climate change, energy production, drug design, global security, and materials design; the analysis of large data sets, such as those in genome sequencing, astronomical observation, and cybersecurity; and, the intricate design of engineered products, such as airplanes and automobiles.

It is clear and well-documented that HPC is used to generate insight that would not otherwise be possible. Simulations can augment or replace expensive, hazardous, or impossible experiments. Furthermore, in the realm of simulation, HPC has the potential to suggest new experiments that escape the parameters of observability.

Although much of the excitement about HPC focuses on the largest architectures and on specific benchmarks, such as TOP500, there is a much deeper and broader commitment from the international scientific and engineering community than is first apparent. In fact, it is easy to lose track of history in terms of the broad uses of HPC and the communities that design, deploy, and operate HPC systems and facilities. Many of these sponsors and organizations have spent decades developing scientific simulation methods and software which serves as the foundation of HPC today. This community has worked closely with countless vendors to foster the sustained development and deployment of HPC systems internationally.

In this second volume of Contemporary High Performance Computing [4], we continue to document international HPC ecosystems including the sponsors and sites that host them. We have selected contributions from international HPC sites, which represent a combination of sites, systems, vendors, applications, and sponsors. Rather than focus on simply the architecture or the application, we focus on HPC ecosystems that have made this dramatic progress possible. Though the very word ecosystem can be a broad, all-encompassing term, it aptly describes HPC. That is, HPC is far more than one sponsor, one site, one application, one software system, or one architecture. Indeed, it is a community of interacting entities in this environment that sustains the community over time. Contributors were asked to address the following topics in their chapters:

1. Sponsor and site history

2. Highlights of applications, workloads, and benchmarks

3. Systems overview

4. Hardware architecture

5. System software

6. Programming systems

7. Storage, visualization, and analytics

8. Data center/facility

9. Site HPC statistics

By any measure, the past 17 years have witnessed dramatic increases in both the use and scale of HPC. Thinking back, it was 1997, only 17 years ago, when the ASCI Red system at Sandia National Laboratories in Albuquerque, New Mexico, broke the 1 Terascale barrier on TOP500 Linpack using 7,264 Pentium P6 processors over a proprietary interconnection network. In 2013, China’s Tihane-2 system accelerated with Xeon Phi surpassed 33.8 Petaflops on the same benchmark: an increase of about 33,000 times in 17 years! It is impressive, indeed, when considering the well-known fact that the performance of commodity microprocessors has slowed due to power and thermal constraints [1, 2, 3].

Although much of the focus is on the #1 system on TOP500, and its architecture, it is important to promote the fact that there are dozens, in fact, hundreds of systems around the world in daily use for solving HPC problems. As Table 1.1 illustrates, we have seven chapters from contributing authors around the world describing their ecosystem. The architectures of these systems span the range from a fully customized processor and interconnection network to a complete commodity solution. A number of new sites have installed the new Cray XC30. Two sites have commodity clusters, where one is accelerated by Xeon Phi. Finally, Japan’s K system is a specialized version of the SPARC64 processor, and the Tofu interconnection network.

In contrast to the hardware diversity we see in these systems, they share a tremendous amount of software as listed in Table 1.2. In fact, much of this software is open-source software that organizations can download, port, and install on any system. In fact, although HPC systems share many hardware components with servers in enterprise and data centers, the HPC software stack is a distinguishing component of HPC. Generally speaking, the HPC software stack has multiple levels: system software, development environments, system management software, and scientific data management and visualization systems. Nearest to the hardware, system software typically includes operating systems, runtime systems, and low-level I/O software, like file systems. Next, the category of programming systems is a broad area that facilitates design and development of applications. In our framework, it includes programming models, compilers, scientific frameworks and libraries, and correctness and performance tools. Then, system management software coordinates, schedules, and monitors the system and the applications running on that system. Finally, scientific data management and visualization software provides users with domain specific tools for generating, managing, and exploring data for their science. This data may include output from simulations per se or empirically measured data from sensors in the real world, which can also be used to validate and calibrate simulation software.

TABLE 1.1: Significant systems in HPC.

TABLE 1.2: HPC software summary.

Category	Item
Operating Systems	Linux (Multiple versions)
Languages	C, C++, FORTRAN (various)
Compilers	Cray Compiler Environment, GNU, Intel, PathScale, PGI
Scripting Languages	Java, Perl, Python (incl. mpipy, Numpy, Matplotlib, Anaconda), Ruby, Tcl/Tk
Distributed Memory Programming Models	Cray CCE (incl. Co-Array Fortran, UPC), Cray MPT, Global Arrays, MPI (OpenMPI, MVAIPICH, Intel MPI, Cray MPI, MPICH, IBM MPI), SHMEM
Shared Memory Programming Models	OpenMP, Pthreads
Heterogeneous Programming Models	Intel MIC Offload, OpenACC, PGI Accelerator
Performance Tools	Allinea MAP, Cray PAT, Cray Reveal, Darshan, Extrae/-Paraver, IPM, Intel Trace Analyzer, Vtune, ITAC MPIP, PAPI, Perftools, Perftools-lite, Scalasca, TAU, Vampir
Correctness Tools	Allinea DDT, Cray ATP, GNU GDB, LGDB, Totalview, Valgrind
Scientific Libraries and Frameworks	ACML, BLAS, Boost, Cray LibSci, FFTW, GNU GSL, GSL, HDF5, Hugetables, Intel MLK, LAPACK, NetCDF, PETSc, PETSc, ParMetis, SLEPc SPRNG, ScaLAPACK, SuiteSparse, Trilinos
Parallel Filesystems and Storage	IBM GPFS, Lustre, GridFtp
I/O Libraries and Software	ADIOS, CGNS, HDF5, Hercule, IOBUF, Silo, pnetCDF
Job Schedulers and Resource Managers	Cray ALPS, IBM LSF, PBSPro, Torque/Moab
System Management	Ganglia, HP Cluster Management Utility, xCAT
Visualization	Ferret, GNUPlot, GrADS SPlash, IDL, NcView R, VisIt
Integrated Problem Solving Environnments	MATLAB, Octave, R

Nevertheless, over the past 17 years, HPC software has had to adapt and respond to several challenges. First, the concurrency in applications and systems has grown over three orders of magnitude. The primary programming model, MPI, has had to grow and change to allow this scale. Second, the increase in concurrency has per core driven lower the memory and I/O capacity, and the memory, I/O, and interconnect bandwidth. Third, in the last five years, heterogeneity and architectural diversity have placed a new emphasis on application and software port...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Table of Contents
• List of Figures
• List of Tables
• Preface
• 1 Terascale to Petascale: The Past 17 Years in High Performance Computing
• 2 ARCHER
• 3 Edison: Enabling Extreme Scale Science at NERSC
• 4 HLRN-III at Zuse Institute Berlin
• 5 The K Computer
• 6 Lindgren—The Swedish Tier-1 System
• 7 Peregrine at the National Renewable Energy Laboratory
• 8 Yellowstone: A Dedicated Resource for Earth System Science
• Index