Computer Science
Parallel Architectures
Parallel architectures refer to computer systems designed to carry out multiple operations simultaneously. They are characterized by the use of multiple processing units that work together to execute tasks in parallel, leading to improved performance and efficiency. Parallel architectures are commonly used in high-performance computing, scientific simulations, and data-intensive applications.
Written by Perlego with AI-assistance
Related key terms
1 of 5
12 Key excerpts on "Parallel Architectures"
eBook - PDFDistributed Computer Control Systems 1981
Proceedings of the Third IFAC Workshop, Beijing, China, 15-17 August 1981
- William E. Miller(Author)
- 2014(Publication Date)
- Pergamon(Publisher)
1. Relationships between parallelism and development of computer architecture Architectural Considerations 43 based on the principle of task distribution. In order for such a distributed computer sys-tem to be used effectively, it is necessary to be able to detect a high degree of paralle-lism in the program and uniformly distribute its tasks among the functionally identical processors of the system. The meaning of parallelism may be generalized by introducing time parameter into its concept, thus leading to the idea of time-overlapping,, that means overlapping in time successive stages of execution of an operation or a pro-gram. The necessary condition for realizing this special form of parallelism is the sepa-ration and specialization of functional units of a computer system. However, this condition is not sufficient. Significant speed-up can be achieved only under the condition that the works of these functional units are overlapped in time in a proper manner such that their bandwidths are balanced or matched one another. In uniprocessors the idea of time-overlapping has led to look-ahead control, instruction and data buffering, etc. Great success has been achieved in various pipeline architectures, such as STAR-100, TI ASC and CRAY-1 at opera-tion level, and BSP at vector-form instruction level. In the limits of multiple computers, the ten-dency of functional specialization opens a wide perspective of developing new multipro-cessor systems on the basis of functional distribution. Such a system may have a loosely coupled structure, comprising various kinds of special-purpose processors used to handle, for example, I/O operations, array computations, data-base management, high-level language com-pilation, etc. Typical MIMD computer systems developed in this direction are the heteroge-neous or asymmetrical multiprocessors, which are characterized by high throughput and excellent cost-performance.
No longer available |Learn more- (Author)
- 2014(Publication Date)
- Learning Press(Publisher)
________________________ WORLD TECHNOLOGIES ________________________ Chapter 2 Parallel Computing Parallel computing is a form of computation in which many calculations are carried out simultaneously, operating on the principle that large problems can often be divided into smaller ones, which are then solved concurrently (in parallel). There are several different forms of parallel computing: bit-level, instruction level, data, and task parallelism. Parallelism has been employed for many years, mainly in high-performance computing, but interest in it has grown lately due to the physical constraints preventing frequency scaling. As power consumption (and consequently heat generation) by computers has become a concern in recent years, parallel computing has become the dominant paradigm in computer architecture, mainly in the form of multicore processors. Parallel computers can be roughly classified according to the level at which the hardware supports parallelism—with multi-core and multi-processor computers having multiple processing elements within a single machine, while clusters, MPPs, and grids use multiple computers to work on the same task. Specialized parallel computer architectures are sometimes used alongside traditional processors, for accelerating specific tasks. Parallel computer programs are more difficult to write than sequential ones, because concurrency introduces several new classes of potential software bugs, of which race conditions are the most common. Communication and synchronization between the different subtasks are typically one of the greatest obstacles to getting good parallel program performance. The speed-up of a program as a result of parallelization is observed as Amdahl's law. Background Traditionally, computer software has been written for serial computation. To solve a problem, an algorithm is constructed and implemented as a serial stream of instructions. These instructions are executed on a central processing unit on one computer.
No longer available |Learn more- (Author)
- 2014(Publication Date)
- College Publishing House(Publisher)
________________________ WORLD TECHNOLOGIES ________________________ Chapter 1 Parallel Computing Parallel computing is a form of computation in which many calculations are carried out simultaneously, operating on the principle that large problems can often be divided into smaller ones, which are then solved concurrently (in parallel). There are several different forms of parallel computing: bit-level, instruction level, data, and task parall-elism. Parallelism has been employed for many years, mainly in high-performance com-puting, but interest in it has grown lately due to the physical constraints preventing frequency scaling. As power consumption (and consequently heat generation) by com-puters has become a concern in recent years, parallel computing has become the dominant paradigm in computer architecture, mainly in the form of multicore processors. Parallel computers can be roughly classified according to the level at which the hard-ware supports parallelism, with multi-core and multi-processor computers having multi-ple processing elements within a single machine, while clusters, MPPs, and grids use multiple computers to work on the same task. Specialized parallel computer architect-ures are sometimes used alongside traditional processors, for accelerating specific tasks. Parallel computer programs are more difficult to write than sequential ones, beca- use concurrency introduces several new classes of potential software bugs, of which race conditions are the most common. Communication and synchronization between the different subtasks are typically some of the greatest obstacles to getting good parallel program performance. The maximum possible speed-up of a program as a result of parallelization is observed as Amdahl's law. Background Traditionally, computer software has been written for serial computation. To solve a problem, an algorithm is constructed and implemented as a serial stream of instructions.
No longer available |Learn more- (Author)
- 2014(Publication Date)
- Learning Press(Publisher)
________________________ WORLD TECHNOLOGIES ________________________ Chapter 6 Parallel Computing Parallel computing is a form of computation in which many calculations are carried out simultaneously, operating on the principle that large problems can often be divided into smaller ones, which are then solved concurrently (in parallel). There are several different forms of parallel computing: bit-level, instruction level, data, and task para-llelism. Parallelism has been employed for many years, mainly in high-performance computing, but interest in it has grown lately due to the physical constraints preventing frequency scaling. As power consumption (and consequently heat generation) by com-puters has become a concern in recent years, parallel computing has become the domi-nant paradigm in computer architecture, mainly in the form of multicore processors. Parallel computers can be roughly classified according to the level at which the hardware supports parallelism—with multi-core and multi-processor computers having multiple processing elements within a single machine, while clusters, MPPs, and grids use multi-ple computers to work on the same task. Specialized parallel computer architectures are sometimes used alongside traditional processors, for accelerating specific tasks. Parallel computer programs are more difficult to write than sequential ones, because concurrency introduces several new classes of potential software bugs, of which race conditions are the most common. Communication and synchronization between the different subtasks are typically one of the greatest obstacles to getting good parallel program performance. The speed-up of a program as a result of parallelization is observed as Amdahl's law. Background Traditionally, computer software has been written for serial computation. To solve a problem, an algorithm is constructed and implemented as a serial stream of instructions.
No longer available |Learn more- (Author)
- 2014(Publication Date)
- Learning Press(Publisher)
________________________ WORLD TECHNOLOGIES ________________________ Chapter- 1 Parallel Computing Parallel computing is a form of computation in which many calculations are carried out simultaneously, operating on the principle that large problems can often be divided into smaller ones, which are then solved concurrently (in parallel). There are several different forms of parallel computing: bit-level, instruction level, data, and task parallelism. Parallelism has been employed for many years, mainly in high-performance computing, but interest in it has grown lately due to the physical constraints preventing frequency scaling. As power consumption (and consequently heat generation) by computers has become a concern in recent years, parallel computing has become the dominant paradigm in computer architecture, mainly in the form of multicore processors. Parallel computers can be roughly classified according to the level at which the hardware supports parallelism, with multi-core and multi-processor computers having multiple processing elements within a single machine, while clusters, MPPs, and grids use multiple computers to work on the same task. Specialized parallel computer architectures are sometimes used alongside traditional processors, for accelerating specific tasks. Parallel computer programs are more difficult to write than sequential ones, because concurrency introduces several new classes of potential software bugs, of which race conditions are the most common. Communication and synchronization between the different subtasks are typically some of the greatest obstacles to getting good parallel program performance. The maximum possible speed-up of a program as a result of parallelization is observed as Amdahl's law. Background Traditionally, computer software has been written for serial computation. To solve a problem, an algorithm is constructed and implemented as a serial stream of instructions.
No longer available |Learn more- (Author)
- 2014(Publication Date)
- Research World(Publisher)
________________________ WORLD TECHNOLOGIES ________________________ Chapter 14 Parallel Computing Parallel computing is a form of computation in which many calculations are carried out simultaneously, operating on the principle that large problems can often be divided into smaller ones, which are then solved concurrently (in parallel). There are several different forms of parallel computing: bit-level, instruction level, data, and task parallelism. Parallelism has been employed for many years, mainly in high-performance computing, but interest in it has grown lately due to the physical constraints preventing frequency scaling. As power consumption (and consequently heat generation) by computers has become a concern in recent years, parallel computing has become the dominant paradigm in computer architecture, mainly in the form of multicore processors. Parallel computers can be roughly classified according to the level at which the hardware supports parallelism—with multi-core and multi-processor computers having multiple processing elements within a single machine, while clusters, MPPs, and grids use multiple computers to work on the same task. Specialized parallel computer architectures are sometimes used alongside traditional processors, for accelerating specific tasks. Parallel computer programs are more difficult to write than sequential ones, because concurrency introduces several new classes of potential software bugs, of which race conditions are the most common. Communication and synchronization between the different subtasks are typically one of the greatest obstacles to getting good parallel program performance. The maximum possible speed-up of a program as a result of parallelization is observed as Amdahl's law. Background Traditionally, computer software has been written for serial computation. To solve a problem, an algorithm is constructed and implemented as a serial stream of instructions.
eBook - PDF- Ivan Stanimirovic(Author)
- 2019(Publication Date)
- Arcler Press(Publisher)
Parallel computing is a programming technique in which many instructions are executed simultaneously. It is based on the principle that large problems can be divided into smaller parts that can be solved concurrently (“in parallel”). Several types of parallel computing: bit-level parallelism, instruction level parallelism, data parallelism, and task parallelism. For many years, parallel computing has been implemented in high-performance computing (HPC), but interest in it has increased in recent years due to the physical constraints preventing frequency scaling. Parallel computing has become the dominant paradigm in computer architecture mainly in multicore processors. But recently, the power consumption of parallel computers has become a concern. Parallel computers can be classified according to the level of parallelism that supports your hardware: multicore and multiprocessing computers have multiple processing elements on a single machine, while clusters and MPP grids use multiple computers to work on the same task. The parallel computer programs are more difficult to write than sequential because concurrency introduces new types of software errors. Communication and synchronization between the different subtasks are typically the greatest Parallel Computer Systems 53 barriers to achieve good performance of parallel programs. The increase in speed achieved as a result of a program parallelization is given by Amdahl law. 4.1. HISTORY The software has traditionally oriented computing series. To solve a problem, it builds an algorithm and is implemented on a serial instruction stream. These instructions are executed on the central processing unit of a computer. At the time when an instruction is completed, the next run. Parallel computing uses multiple processing elements simultaneously to solve a problem.
eBook - PDFParallel Iterative Algorithms
From Sequential to Grid Computing
- Jacques Mohcine Bahi, Sylvain Contassot-Vivier, Raphael Couturier(Authors)
- 2007(Publication Date)
- Chapman and Hall/CRC(Publisher)
Obviously, that definition is very broad and thus holds for many systems which are not in the scope of this book. However, it is exactly that concept which is used in computer science and it encompasses all the computing parallel systems going from parallel machines to distributed clusters while also including pipelines. 49 50 Parallel Iterative Algorithms Since the beginning of parallel computing in the mid fifties, the evolution of parallel systems has been influenced by several factors: the progress of the interconnection networks and of the Integrated Circuits (IC) technology, but also the decrease in the production costs of processing units. In fact, the improvements of the networks have tended to increase the distance between the processing units whereas the advances related to ICs and to the financial aspect have influenced their nature. In this way, parallelism initially invaded computers at the processor level under several aspects. The first one took place during the era of scalar proces-sors, in the development of coprocessors taking in charge some specific tasks of the working unit (mathematical operations, communications, graphics,...) and relieving the Central Processing Unit (CPU). Another aspect has resided in the processor itself. The development of the Complementary Metal-Oxide-Semiconductor (CMOS) technology since 1963 and of the Very-Large-Scale Integration (VLSI) since the 1980s have allowed the inclusion of more and more complex components in the processors such as pipelines and multiple computation units. More recently, as the CMOS technology is nearer and nearer its physical limits, that intrinsic parallelization of the processors has logically been followed by the emergence of multiple-core processors. Except for that last development, those forms of parallelism are hidden to the programmer of the machine; this is why we call it intrinsic parallelism.
eBook - ePub- Alan D. George(Author)
- 2018(Publication Date)
- CRC Press(Publisher)
Parallel ComputingThis chapter provides a background overview of parallel computing, including such issues as parallel processing overview and basic definitions, parallel architecture models, and related background information with regard to operating systems and real-time computing systems.3.1 Parallel Processing Overview and DefinitionsIt has been predicted that the decade of the 1990s will usher in the era of true parallel and distributed computing with calculations partitioned into sub-calculations and then given to distinct processors to execute within a multiprocessor system or on a network of processors [DEIT90]. As microprocessors and workstations become more powerful, and the corresponding prices fall, it becomes economically more feasible to incorporate parallel rather than sequential computing and distributed rather than centralized computing. The degree of physical separation of the processors tends to be small due to the fact that communication over long distances is based on telephone lines, which were originally established for highly redundant voice messages at slow speeds. It is instead usually much more efficient to cluster processors at geographically close locations in order to send data and instructions between processors quickly.As the research areas of parallel processing and parallel computing systems have developed, a number of basic concepts and terms have been introduced. Of these, the following represents some of the fundamental definitions that will be referenced.The need for parallel processing in future computing systems is becoming increasingly more evident. Non-parallel or sequential computers are quickly approaching the upper limit of their computational potential. This potential performance envelope is dictated by the speed of light, which restricts the maximum signal transmission speed in silicon to 3 × 107 meters per second. Thus, a chip which is 3 centimeters in diameter requires at least 10-9 seconds to propagate a signal, thereby restricting such a chip to at most 109 floating-point operations per second (i.e. 1 GFLOPS). Since existing supercomputer processors are quickly approaching this limit, the future of sequential processors appears limited, as illustrated in Figure 3.1
- David H. Bailey, Robert F. Lucas, Samuel Williams(Authors)
- 2010(Publication Date)
- CRC Press(Publisher)
Unfortunately, the performance of individual processing elements is limited to the frequency at which they may be clocked. For this reason, the performance of a scientific computation (even setting aside the question of whether there is enough memory to hold the necessary data) is limited to basically the reciprocal of the clock period. However, since virtually all computations exhibit at least a modest degree of parallelism, this means that at a given level of technology, the only means of increasing performance is to employ parallel processors. There are many styles of Parallel Architectures, but all present-day designs consist of processing elements, memory, and an interprocessor network. The primary differentiating factor is whether computation is instruction-driven or data-driven. In this chapter, we will restrict ourselves to the former, namely architectures whose processing elements are based, in a fundamental sense, on a classical von Neumann design, originally sketched by John von Neu-mann. Within this class, there are two principle types of Parallel Architectures: shared-memory and distributed-memory. These two styles are illustrated in Figure 2.1. We will discuss processor architecture, memory architecture, and network architecture in the following sections. As mentioned earlier, we will keep this discussion at a rather high-level, since parallel processing may be realized across a wide variety of scales, from multicore chips to supercomputers. For additional details, see [104, 164, 268], or any of many articles on the topic, many of which are accessible via the Internet. 2.2.1 Shared-Memory Parallel Architectures Figure 2.1(a) illustrates a shared-memory parallel architecture.In this ar-chitecture, the von Neuman architecture is extended by adding additional processing elements (each with their own program counters) yet keeping one (conceptually) memory that is now shared among processing elements.
eBook - ePubParallel Computing Architectures and APIs
IoT Big Data Stream Processing
- Vivek Kale(Author)
- 2019(Publication Date)
- Chapman and Hall/CRC(Publisher)
It then described benchmarks for performance evaluation. Benchmarks are a measure of the efficient execution by the computer architecture of those features of a programming language that are most frequently used in actual programs. Benchmarks are considered to be representative of classes of applications envisioned for the architecture. Benchmarks are useful in evaluating hardware as well as software and single processor as well as multiprocessor systems. The chapter then looked at the degree of parallelism of an application, which is simply the index of the number of computations that can be executed concurrently. Modeling and measurement of application characteristics can be used to optimize the design of a cluster, for example, in terms of relative resources dedicated to computation, memory, and interconnect. There are three main categories of Parallel Architectures: shared memory parallel architecture, distributed memory parallel architecture, and parallel accelerator architecture. We study these in Chapters 10 through 12, respectively. 10 Shared Memory Architecture Shared memory systems form a major category of multiprocessors. In this category, all processors share a global memory. Communication between tasks running on different processors is performed through writing to and reading from the global memory. All interprocessor coordination and synchronization are also accomplished via the global memory. A shared memory computer system consists of a set of independent processors, a set of memory modules, and an interconnection network, as shown in Figure 6.2. Two main problems that need to be addressed when designing a shared memory system are • Performance degradation due to contention : Performance degradation might happen when multiple processors are trying to access the shared memory simultaneously. A typical design might use caches to solve the contention problem
eBook - ePubHigh Performance Computing and the Art of Parallel Programming
An Introduction for Geographers, Social Scientists and Engineers
- Stan Openshaw, Ian Turton(Authors)
- 2005(Publication Date)
- Routledge(Publisher)
M separate CPUs all busily working on the same problem – just think of all that computing power – but how does it work? How does a simple-minded geographer come to grips with the challenges of parallel processing and parallel programming? One way is to dash out and read several ‘Parallel Processing’ books written by computer scientists but, unless you are extremely lucky, the effect will almost certainly be the equivalent to a cold shower of sufficient intensity to totally deter waverers and to greatly dampen potential enthusiasts possessed of anything less than a steely determination to persevere. Of course there is a catch. Writing software and designing algorithms that work well on machines with multiple processors inside them need not be trivial or easy. However, the problems can be explained without undue complexity, so read on – you have little to lose!3.1.1 What do the words mean?Perhaps a useful starting point is to consider what the words mean. There are various definitions of the term ‘parallel computing’, which all have more or less the same meaning. Parallel processing is self-evident. It is ‘only’ a matter of getting multiple processors to work simultaneously on your program, which previously had to make do with only one. More formal computer scientist-like definitions are as follows.Baker and Smith (1996) would agree:‘Parallel computing is the use of more than one Central Processing Unit (CPU) at the same time to solve a single problem’ (p. 1 ).Chalmers and Tidmus (1996) write:‘Parallel processing is the solution of a single problem by dividing it into a number of sub-problems, each of which may be solved by a separate agent’ (p. 2 ).Similarly, Chandy and Taylor (1992) explain:‘A parallel program is simply a collection of co-operating programs that together satisfy a given specification’ (p. 3 ).Almasi and Gottlieb (1989) explain that a parallel processor is‘A large collection of processing elements that can communicate and cooperate to solve large problems fast’ (p. 5 ).Hwang and Briggs (1984) write‘Parallel processing is an efficient form of information processing which emphasises the exploitation of concurrent events in the computing process. Concurrency implies parallelism, simultaneity, and pipelining’ (p. 6 );while Krishnamurthy (1989) writes‘A simple-minded approach to gain speed, as well as power, in computing is through parallelism; here many computers would work together, all simultaneously executing some portions of a procedure used for solving a problem’ (p. 1
Index pages curate the most relevant extracts from our library of academic textbooks. They’ve been created using an in-house natural language model (NLM), each adding context and meaning to key research topics.
