Content provided in partnership with

Technology Industry

Clustered storage: improved utility for production computer clusters

Computer Technology Review, Dec, 2004 by Bruce Moxon

With origins in the DARPA Strategic Computing program of the mid-1980s, Beowulf-class cluster computing has evolved as the dominant approach for developing high performance computing systems. In recent years, once prevalent vector computers from the likes of Cray, Convex, Alliant, Fujitsu, and others, have given way to cluster computing configurations for even the most aggressive computing applications. Top500.org, which tracks the most powerful computing systems in the world, recently reported that nearly 70% of the fastest computer systems in the world are now labeled as cluster configurations. This trend has been steadily increasing over the last few years.

Most Popular Articles in Technology: An overview of continuous data protection; Why all those current ratings?; Many countries now have a mobile penetration rate above 100%, report says; The Tata Group's big telecom gamble: VSNL's recent acquisition of Tyco ...; MEASURING BANK BRANCH EFFICIENCY USING DATA ENVELOPMENT ANALYSIS: MANAGERIAL ...; More »

Once the esoteric tool of PhD computer scientists, cluster computing technology has matured to become an integral part of many organizations' production computing environments. Approaches refined in traditional scientific computing applications (such as high energy physics research, weather prediction, and seismic data analysis) are the core of a new wave of production-oriented applications in areas such as drug discovery, circuit design and simulation, aerospace and automotive design and simulation, financial analytics, and digital media.

[FIGURE 1 OMITTED]

Concurrent with this evolution, more capable instrumentation, more powerful processors, and higher fidelity computer models serve to continually increase the data throughput required of these clusters. This trend applies pressure to the storage systems used to support these I/O-hungry applications, and has prompted a wave of new storage solutions based on the same scale-out approach as cluster computing.

Anatomy of Production High Throughput Computing Applications

Most of these high throughput applications can be classified as one of two processing scenarios: data reduction, or data generation. In the former, large input datasets--often taken from some scientific instrument--are processed to identify patterns and/or produce aggregated descriptions of the input. This is the most common scenario for seismic processing, as well as similarly structured analysis applications such as micro array data processing, or remote sensing. In the latter scenario, small input datasets (parameters) are used to drive simulations that generate large output datasets--often time sequenced--that can be further analyzed or visualized. Examples here include crash analysis, combustion models, weather prediction, and computer graphics rendering applications used to generate special effects and full-feature animated films. These two data-intensive scenarios are depicted in Figure 1.

Divide and Conquer

To address these problems, today's cluster computing approaches utilize what is commonly called a scale-out or shared nothing approach to parallel computing. In the scale-out model, applications are developed using a divide-and-conquer approach; the problem is decomposed into hundreds, thousands or even millions of tasks, each of which is executed independently (or nearly independently). The most common decomposition approach exploits a problem's inherent data parallelism-breaking the problem into pieces by identifying the data subsets, or partitions, that comprise the individual tasks, then distributing those tasks and the corresponding data partitions to the compute nodes for processing.

These scale-out approaches typically employ single or dual processor compute nodes in a 1U configuration that facilitates rack-based implementations. Hundreds (or even thousands) of these nodes are connected to one another with high-speed, low latency proprietary interconnects such as Myricom's Myrinet, Infiniband, or commodity Gigabit Ethernet switches. Each compute node may process one or more application data partitions, depending on node configuration and the application's computation, memory, and I/O requirements. These partitioned applications are often developed using the Message Passing Interface (MPI) program development and execution environment.

The scale-out environment allows accomplished programmers to exploit a common set of libraries to control overall program execution and to support the processor-to-processor communication required of distributed high-performance computing applications. Scale-out approaches provide effective solutions for many problems addressed by high-performance computing.

It's All About Data Management

However, scalability and performance come at a cost--namely, the additional complexity required to break a problem into pieces (data partitions), exchange or replicate information across the pieces when necessary, then put the partial result sets back together into the final answer. This data parallel approach requires the creation and management of data partitions and replicas that are used by the compute nodes. Management of these partitions and replicas poses a number of operational challenges, especially in large cluster and grid computing environments shared amongst a number of projects or organizations, and in environments where core datasets change regularly. This is typically one of the most time consuming and complex development problems facing organizations adopting cluster computing.

Technology Industry

Clustered storage: improved utility for production computer clusters

Computer Technology Review, Dec, 2004 by Bruce Moxon

Find Research Guides for:

Find Featured Titles for: Health

Most Popular Articles in Technology

Most Popular Publications in Technology