parallel computing ibms blue genep massively parallel supercomputer parallel computing is a type of computation in which many calculations or the execution of processes are carried out simultaneously parallel computing is closely related to concurrent computingthey are frequently used together and often conflated though the two are distinct it is possible to have parallelism without concurrency such as bitlevel parallelism and concurrency without parallelism such as multitasking by timesharing on a singlecore cpu in parallel computing a computational task is typically broken down into several often many very similar subtasks that can be processed independently and whose results are combined afterwards upon completion in contrast in concurrent computing the various processes often do not address related tasks when they do as is typical in distributed computing the separate tasks may have a varied nature and often require some interprocess communication during execution parallel computers can be roughly classified according to the level at which the hardware supports parallelism with multicore and multiprocessor computers having multiple processing elements within a single machine while clusters mpps and grids use multiple computers to work on the same task specialized parallel computer architectures are sometimes used alongside traditional processors for accelerating specific tasks in some cases parallelism is transparent to the programmer such as in bitlevel or instructionlevel parallelism but explicitly parallel algorithms particularly those that use concurrency are more difficult to write than sequential ones because concurrency introduces several new classes of potential software bugs of which race conditions are the most common communication and synchronization between the different subtasks are typically some of the greatest obstacles to getting good parallel program performance a theoretical upper bound on the speedup of a single program as a result of parallelization is given by amdahls law background traditionally computer software has been written for serial computation to solve a problem an algorithm is constructed and implemented as a serial stream of instructions these instructions are executed on a central processing unit on one computer only one instruction may execute at a timeafter that instruction is finished the next one is executed parallel computing on the other hand uses multiple processing elements simultaneously to solve a problem this is accomplished by breaking the problem into independent parts so that each processing element can execute its part of the algorithm simultaneously with the others the processing elements can be diverse and include resources such as a single computer with multiple processors several networked computers specialized hardware or any combination of the above frequency scaling was the dominant reason for improvements in computer performance from the mid1980s until 2004 the runtime of a program is equal to the number of instructions multiplied by the average time per instruction maintaining everything else constant increasing the clock frequency decreases the average time it takes to execute an instruction an increase in frequency thus decreases runtime for all computebound programs to deal with the problem of power consumption and overheating the major central processing unit cpu or processor manufacturers started to produce power efficient processors with multiple cores the core is the computing unit of the processor and in multicore processors each core is independent and can access the same memory concurrently multicore processors have brought parallel computing to desktop computers thus parallelisation of serial programmes has become a mainstream programming task in 2012 quadcore processors became standard for desktop computers while servers have 10 and 12 core processors from moores law it can be predicted that the number of cores per processor will double every 1824 months this could mean that after 2020 a typical processor will have dozens or hundreds of cores an operating system can ensure that different tasks and user programmes are run in parallel on the available cores however for a serial software programme to take full advantage of the multicore architecture the programmer needs to restructure and parallelise the code a speedup of application software runtime will no longer be achieved through frequency scaling instead programmers will need to parallelise their software code to take advantage of the increasing computing power of multicore architectures amdahls law and gustafsons law a graphical representation of amdahls law the speedup of a program from parallelization is limited by how much of the program can be parallelized for example if 90 of the program can be parallelized the theoretical maximum speedup using parallel computing would be 10 times no matter how many processors are used assume that a task has two independent parts a and b part b takes roughly 25 of the time of the whole computation by working very hard one may be able to make this part 5 times faster but this only reduces the time for the whole computation by a little in contrast one may need to perform less work to make part a be twice as fast this will make the computation much faster than by optimizing part b even though part bs speedup is greater by ratio 5 times versus 2 times optimally the speedup from parallelization would be lineardoubling the number of processing elements should halve the runtime and doubling it a second time should again halve the runtime however very few parallel algorithms achieve optimal speedup most of them have a nearlinear speedup for small numbers of processing elements which flattens out into a constant value for large numbers of processing elements the potential speedup of an algorithm on a parallel computing platform is given by amdahls law s latency s 1 1 p p s displaystyle stextlatencysfrac 11pfrac ps where slatency is the potential speedup in latency of the execution of the whole task s is the speedup in latency of the execution of the parallelizable part of the task p is the percentage of the execution time of the whole task concerning the parallelizable part of the task before parallelization since slatency 11 p it shows that a small part of the program which cannot be parallelized will limit the overall speedup available from parallelization a program solving a large mathematical or engineering problem will typically consist of several parallelizable parts and several nonparallelizable serial parts if the nonparallelizable part of a program accounts for 10 of the runtime p 09 we can get no more than a 10 times speedup regardless of how many processors are added this puts an upper limit on the usefulness of adding more parallel execution units when a task cannot be partitioned because of sequential constraints the application of more effort has no effect on the schedule the bearing of a child takes nine months no matter how many women are assigned a graphical representation of gustafsons law amdahls law only applies to cases where the problem size is fixed in practice as more computing resources become available they tend to get used on larger problems larger datasets and the time spent in the parallelizable part often grows much faster than the inherently serial work s latency s 1 p s p displaystyle stextlatencys1psp both amdahls law and gustafsons law assume that the running time of the serial part of the program is independent of the number of processors amdahls law assumes that the entire problem is of fixed size so that the total amount of work to be done in parallel is also independent of the number of processors whereas gustafsons law assumes that the total amount of work to be done in parallel varies linearly with the number of processors dependencies understanding data dependencies is fundamental in implementing parallel algorithms no program can run more quickly than the longest chain of dependent calculations known as the critical path since calculations that depend upon prior calculations in the chain must be executed in order however most algorithms do not consist of just a long chain of dependent calculations there are usually opportunities to execute independent calculations in parallel let pi and pj be two program segments bernsteins conditions describe when the two are independent and can be executed in parallel for pi let ii be all of the input variables and oi the output variables and likewise for pj pi and pj are independent if they satisfy i j o i displaystyle ijcap oivarnothing i i o j displaystyle iicap ojvarnothing o i o j displaystyle oicap ojvarnothing violation of the first condition introduces a flow dependency corresponding to the first segment producing a result used by the second segment the second condition represents an antidependency when the second segment produces a variable needed by the first segment the third and final condition represents an output dependency when two segments write to the same location the result comes from the logically last executed segment consider the following functions which demonstrate several kinds of dependencies 1 function depa b 2 c a b 3 d 3 c 4 end function in this example instruction 3 cannot be executed before or even in parallel with instruction 2 because instruction 3 uses a result from instruction 2 it violates condition 1 and thus introduces a flow dependency 1 function nodepa b 2 c a b 3 d 3 b 4 e a b 5 end function in this example there are no dependencies between the instructions so they can all be run in parallel bernsteins conditions do not allow memory to be shared between different processes for that some means of enforcing an ordering between accesses is necessary such as semaphores barriers or some other synchronization method race conditions mutual exclusion synchronization and parallel slowdown subtasks in a parallel program are often called threads some parallel computer architectures use smaller lightweight versions of threads known as fibers while others use bigger versions known as processes however threads is generally accepted as a generic term for subtasks threads will often need synchronized access to an object or other resource for example when they must update a variable that is shared between them without synchronization the instructions between the two threads may be interleaved in any order for example consider the following program thread a thread b 1a read variable v 1b read variable v 2a add 1 to variable v 2b add 1 to variable v 3a write back to variable v 3b write back to variable v if instruction 1b is executed between 1a and 3a or if instruction 1a is executed between 1b and 3b the program will produce incorrect data this is known as a race condition the programmer must use a lock to provide mutual exclusion a lock is a programming language construct that allows one thread to take control of a variable and prevent other threads from reading or writing it until that variable is unlocked the thread holding the lock is free to execute its critical section the section of a program that requires exclusive access to some variable and to unlock the data when it is finished therefore to guarantee correct program execution the above program can be rewritten to use locks thread a thread b 1a lock variable v 1b lock variable v 2a read variable v 2b read variable v 3a add 1 to variable v 3b add 1 to variable v 4a write back to variable v 4b write back to variable v 5a unlock variable v 5b unlock variable v one thread will successfully lock variable v while the other thread will be locked outunable to proceed until v is unlocked again this guarantees correct execution of the program locks may be necessary to ensure correct program execution when threads must serialize access to resources but their use can greatly slow a program and may affect its reliability locking multiple variables using nonatomic locks introduces the possibility of program deadlock an atomic lock locks multiple variables all at once if it cannot lock all of them it does not lock any of them if two threads each need to lock the same two variables using nonatomic locks it is possible that one thread will lock one of them and the second thread will lock the second variable in such a case neither thread can complete and deadlock results many parallel programs require that their subtasks act in synchrony this requires the use of a barrier barriers are typically implemented using a lock or a semaphore not all parallelization results in speedup generally as a task is split up into more and more threads those threads spend an everincreasing portion of their time communicating with each other or waiting on each other for access to resources finegrained coarsegrained and embarrassing parallelism applications are often classified according to how often their subtasks need to synchronize or communicate with each other an application exhibits finegrained parallelism if its subtasks must communicate many times per second it exhibits coarsegrained parallelism if they do not communicate many times per second and it exhibits embarrassing parallelism if they rarely or never have to communicate embarrassingly parallel applications are considered the easiest to parallelize consistency models consistency model parallel programming languages and parallel computers must have a consistency model also known as a memory model the consistency model defines rules for how operations on computer memory occur and how results are produced one of the first consistency models was leslie lamports sequential consistency model sequential consistency is the property of a parallel program that its parallel execution produces the same results as a sequential program specifically a program is sequentially consistent if the results of any execution is the same as if the operations of all the processors were executed in some sequential order and the operations of each individual processor appear in this sequence in the order specified by its program software transactional memory is a common type of consistency model software transactional memory borrows from database theory the concept of atomic transactions and applies them to memory accesses mathematically these models can be represented in several ways introduced in 1962 petri nets were an early attempt to codify the rules of consistency models dataflow theory later built upon these and dataflow architectures were created to physically implement the ideas of dataflow theory beginning in the late 1970s process calculi such as calculus of communicating systems and communicating sequential processes were developed to permit algebraic reasoning about systems composed of interacting components more recent additions to the process calculus family such as the calculus have added the capability for reasoning about dynamic topologies logics such as lamports tla and mathematical models such as traces and actor event diagrams have also been developed to describe the behavior of concurrent systems see also relaxed sequential flynns taxonomy michael j flynn created one of the earliest classification systems for parallel and sequential computers and programs now known as flynns taxonomy flynn classified programs and computers by whether they were operating using a single set or multiple sets of instructions and whether or not those instructions were using a single set or multiple sets of data flynns taxonomy single data stream sisd misd multiple data streams simd mimd spmd mpmd the singleinstructionsingledata sisd classification is equivalent to an entirely sequential program the singleinstructionmultipledata simd classification is analogous to doing the same operation repeatedly over a large data set this is commonly done in signal processing applications multipleinstructionsingledata misd is a rarely used classification while computer architectures to deal with this were devised such as systolic arrays few applications that fit this class materialized multipleinstructionmultipledata mimd programs are by far the most common type of parallel programs according to david a patterson and john l hennessy some machines are hybrids of these categories of course but this classic model has survived because it is simple easy to understand and gives a good first approximation it is alsoperhaps because of its understandabilitythe most widely used scheme types of parallelism bitlevel parallelism bitlevel parallelism from the advent of verylargescale integration vlsi computerchip fabrication technology in the 1970s until about 1986 speedup in computer architecture was driven by doubling computer word sizethe amount of information the processor can manipulate per cycle increasing the word size reduces the number of instructions the processor must execute to perform an operation on variables whose sizes are greater than the length of the word for example where an 8bit processor must add two 16bit integers the processor must first add the 8lowerorder bits from each integer using the standard addition instruction then add the 8higherorder bits using an addwithcarry instruction and the carry bit from the lower order addition thus an 8bit processor requires two instructions to complete a single operation where a 16bit processor would be able to complete the operation with a single instruction historically 4bit microprocessors were replaced with 8bit then 16bit then 32bit microprocessors this trend generally came to an end with the introduction of 32bit processors which has been a standard in generalpurpose computing for two decades not until the early 2000s with the advent of x8664 architectures did 64bit processors become commonplace instructionlevel parallelism instructionlevel parallelism a canonical processor without pipeline it takes five clock cycles to complete one instruction and thus the processor can issue subscalar performance ipc 02 1 a canonical fivestage pipelined processor in the best case scenario it takes one clock cycle to complete one instruction and thus the processor can issue scalar performance ipc 1 a computer program is in essence a stream of instructions executed by a processor without instructionlevel parallelism a processor can only issue less than one instruction per clock cycle ipc 1 these processors are known as subscalar processors these instructions can be reordered and combined into groups which are then executed in parallel without changing the result of the program this is known as instructionlevel parallelism advances in instructionlevel parallelism dominated computer architecture from the mid1980s until the mid1990s all modern processors have multistage instruction pipelines each stage in the pipeline corresponds to a different action the processor performs on that instruction in that stage a processor with an nstage pipeline can have up to n different instructions at different stages of completion and thus can issue one instruction per clock cycle ipc 1 these processors are known as scalar processors the canonical example of a pipelined processor is a risc processor with five stages instruction fetch if instruction decode id execute ex memory access mem and register write back wb the pentium 4 processor had a 35stage pipeline a canonical fivestage pipelined superscalar processor in the best case scenario it takes one clock cycle to complete two instructions and thus the processor can issue superscalar performance ipc 2 1 most modern processors also have multiple execution units they usually combine this feature with pipelining and thus can issue more than one instruction per clock cycle ipc 1 these processors are known as superscalar processors instructions can be grouped together only if there is no data dependency between them scoreboarding and the tomasulo algorithm which is similar to scoreboarding but makes use of register renaming are two of the most common techniques for implementing outoforder execution and instructionlevel parallelism task parallelism task parallelism task parallelisms is the characteristic of a parallel program that entirely different calculations can be performed on either the same or different sets of data hardware memory and communication main memory in a parallel computer is either shared memory shared between all processing elements in a single address space or distributed memory in which each processing element has its own local address space distributed memory refers to the fact that the memory is logically distributed but often implies that it is physically distributed as well distributed shared memory and memory virtualization combine the two approaches where the processing element has its own local memory and access to the memory on nonlocal processors accesses to local memory are typically faster than accesses to nonlocal memory a logical view of a nonuniform memory access numa architecture processors in one directory can access that directorys memory with less latency than they can access memory in the other directorys memory computer architectures in which each element of main memory can be accessed with equal latency and bandwidth are known as uniform memory access uma systems typically that can be achieved only by a shared memory system in which the memory is not physically distributed a system that does not have this property is known as a nonuniform memory access numa architecture distributed memory systems have nonuniform memory access computer systems make use of cachessmall and fast memories located close to the processor which store temporary copies of memory values nearby in both the physical and logical sense parallel computer systems have difficulties with caches that may store the same value in more than one location with the possibility of incorrect program execution these computers require a cache coherency system which keeps track of cached values and strategically purges them thus ensuring correct program execution bus snooping is one of the most common methods for keeping track of which values are being accessed and thus should be purged designing large highperformance cache coherence systems is a very difficult problem in computer architecture as a result shared memory computer architectures do not scale as well as distributed memory systems do processorprocessor and processormemory communication can be implemented in hardware in several ways including via shared either multiported or multiplexed memory a crossbar switch a shared bus or an interconnect network of a myriad of topologies including star ring tree hypercube fat hypercube a hypercube with more than one processor at a node or ndimensional mesh parallel computers based on interconnected networks need to have some kind of routing to enable the passing of messages between nodes that are not directly connected the medium used for communication between the processors is likely to be hierarchical in large multiprocessor machines classes of parallel computers parallel computers can be roughly classified according to the level at which the hardware supports parallelism this classification is broadly analogous to the distance between basic computing nodes these are not mutually exclusive for example clusters of symmetric multiprocessors are relatively common multicore computing multicore processor a multicore processor is a processor that includes multiple processing units called cores on the same chip this processor differs from a superscalar processor which includes multiple execution units and can issue multiple instructions per clock cycle from one instruction stream thread in contrast a multicore processor can issue multiple instructions per clock cycle from multiple instruction streams ibms cell microprocessor designed for use in the sony playstation 3 is a prominent multicore processor each core in a multicore processor can potentially be superscalar as wellthat is on every clock cycle each core can issue multiple instructions from one thread concurrent multithreading of which intels hyperthreading is the best known was an early form of pseudomulticoreism a processor capable of concurrent multithreading includes multiple execution units in the same processing unitthat is it has a superscalar architectureand can issue multiple instructions per clock cycle from multiple threads temporal multithreading on the other hand includes a single execution unit in the same processing unit and can issue one instruction at a time from multiple threads symmetric multiprocessing symmetric multiprocessing a symmetric multiprocessor smp is a computer system with multiple identical processors that share memory and connect via a bus distributed computing distributed computing a distributed computer also known as a distributed memory multiprocessor is a distributed memory computer system in which the processing elements are connected by a network distributed computers are highly scalable the terms concurrent computing parallel computing and distributed computing have a lot of overlap and no clear distinction exists between them cluster computing computer cluster a beowulf cluster a cluster is a group of loosely coupled computers that work together closely so that in some respects they can be regarded as a single computer the remaining are massively parallel processors explained below because grid computing systems described below can easily handle embarrassingly parallel problems modern clusters are typically designed to handle more difficult problemsproblems that require nodes to share intermediate results with each other more often this requires a high bandwidth and more importantly a lowlatency interconnection network many historic and current supercomputers use customized highperformance network hardware specifically designed for cluster computing such as the cray gemini network as of 2014 most current supercomputers use some offtheshelf standard network hardware often myrinet infiniband or gigabit ethernet massively parallel computing massively parallel computing a cabinet from ibms blue genel massively parallel supercomputer a massively parallel processor mpp is a single computer with many networked processors mpps have many of the same characteristics as clusters but mpps have specialized interconnect networks whereas clusters use commodity hardware for networking mpps also tend to be larger than clusters typically having far more than 100processors ibms blue genel the fifth fastest supercomputer in the world according to the june 2009 top500 ranking is an mpp grid computing grid computing grid computing is the most distributed form of parallel computing it makes use of computers communicating over the internet to work on a given problem because of the low bandwidth and extremely high latency available on the internet distributed computing typically deals only with embarrassingly parallel problems many distributed computing applications have been created of which setihome and foldinghome are the bestknown examples most grid computing applications use middleware software that sits between the operating system and the application to manage network resources and standardize the software interface the most common distributed computing middleware is the berkeley open infrastructure for network computing boinc often distributed computing software makes use of spare cycles performing computations at times when a computer is idling specialized parallel computers within parallel computing there are specialized parallel devices that remain niche areas of interest while not domainspecific they tend to be applicable to only a few classes of parallel problems reconfigurable computing with fieldprogrammable gate arrays reconfigurable computing is the use of a fieldprogrammable gate array fpga as a coprocessor to a generalpurpose computer an fpga is in essence a computer chip that can rewire itself for a given task fpgas can be programmed with hardware description languages such as vhdl or verilog however programming in these languages can be tedious several vendors have created c to hdl languages that attempt to emulate the syntax and semantics of the c programming language with which most programmers are familiar the best known c to hdl languages are mitrionc impulse c dimec and handelc specific subsets of systemc based on c can also be used for this purpose amds decision to open its hypertransport technology to thirdparty vendors has become the enabling technology for highperformance reconfigurable computing generalpurpose computing on graphics processing units gpgpu gpgpu nvidias tesla gpgpu card generalpurpose computing on graphics processing units gpgpu is a fairly recent trend in computer engineering research gpus are coprocessors that have been heavily optimized for computer graphics processing computer graphics processing is a field dominated by data parallel operationsparticularly linear algebra matrix operations in the early days gpgpu programs used the normal graphics apis for executing programs however several new programming languages and platforms have been built to do general purpose computation on gpus with both nvidia and amd releasing programming environments with cuda and stream sdk respectively other gpu programming languages include brookgpu peakstream and rapidmind nvidia has also released specific products for computation in their tesla series the technology consortium khronos group has released the opencl specification which is a framework for writing programs that execute across platforms consisting of cpus and gpus amd apple intel nvidia and others are supporting opencl applicationspecific integrated circuits applicationspecific integrated circuit several applicationspecific integrated circuit asic approaches have been devised for dealing with parallel applications because an asic is by definition specific to a given application it can be fully optimized for that application as a result for a given application an asic tends to outperform a generalpurpose computer however asics are created by uv photolithography this process requires a mask set which can be extremely expensive a mask set can cost over a million us dollars high initial cost and the tendency to be overtaken by mooreslawdriven generalpurpose computing has rendered asics unfeasible for most parallel computing applications however some have been built one example is the pflops riken mdgrape3 machine which uses custom asics for molecular dynamics simulation vector processors vector processor the cray1 is a vector processor a vector processor is a cpu or computer system that can execute the same instruction on large sets of data vector processors have highlevel operations that work on linear arrays of numbers or vectors an example vector operation is a b c where a b and c are each 64element vectors of 64bit floatingpoint numbers cray computers became famous for their vectorprocessing computers in the 1970s and 1980s however vector processorsboth as cpus and as full computer systemshave generally disappeared modern processor instruction sets do include some vector processing instructions such as with freescale semiconductors altivec and intels streaming simd extensions sse software parallel programming languages list of concurrent and parallel programming languages concurrent programming languages libraries apis and parallel programming models such as algorithmic skeletons have been created for programming parallel computers these can generally be divided into classes based on the assumptions they make about the underlying memory architectureshared memory distributed memory or shared distributed memory shared memory programming languages communicate by manipulating shared memory variables distributed memory uses message passing posix threads and openmp are two of the most widely used shared memory apis whereas message passing interface mpi is the most widely used messagepassing system api one concept used in programming parallel programs is the future concept where one part of a program promises to deliver a required datum to another part of a program at some future time caps entreprise and pathscale are also coordinating their effort to make hybrid multicore parallel programming hmpp directives an open standard called openhmpp the openhmpp directivebased programming model offers a syntax to efficiently offload computations on hardware accelerators and to optimize data movement tofrom the hardware memory openhmpp directives describe remote procedure call rpc on an accelerator device eg gpu or more generally a set of cores the directives annotate c or fortran codes to describe two sets of functionalities the offloading of procedures denoted codelets onto a remote device and the optimization of data transfers between the cpu main memory and the accelerator memory the rise of consumer gpus has led to support for compute kernels either in graphics apis referred to as compute shaders in dedicated apis such as opencl or in other language extensions automatic parallelization automatic parallelization automatic parallelization of a sequential program by a compiler is the holy grail of parallel computing despite decades of work by compiler researchers automatic parallelization has had only limited success mainstream parallel programming languages remain either explicitly parallel or at best partially implicit in which a programmer gives the compiler directives for parallelization a few fully implicit parallel programming languages existsisal parallel haskell sequencel system c for fpgas mitrionc vhdl and verilog application checkpointing application checkpointing as a computer system grows in complexity the mean time between failures usually decreases application checkpointing is a technique whereby the computer system takes a snapshot of the applicationa record of all current resource allocations and variable states akin to a core dump this information can be used to restore the program if the computer should fail application checkpointing means that the program has to restart from only its last checkpoint rather than the beginning while checkpointing provides benefits in a variety of situations it is especially useful in highly parallel systems with a large number of processors used in high performance computing algorithmic methods as parallel computers become larger and faster we are now able to solve problems that had previously taken too long to run fields as varied as bioinformatics for protein folding and sequence analysis and economics for mathematical finance have taken advantage of parallel computing common types of problems in parallel computing applications include dense linear algebra sparse linear algebra spectral methods such as cooleytukey fast fourier transform nbody problems such as barneshut simulation structured grid problems such as lattice boltzmann methods unstructured grid problems such as found in finite element analysis monte carlo method combinational logic such as bruteforce cryptographic techniques graph traversal such as sorting algorithms dynamic programming branch and bound methods graphical models such as detecting hidden markov models and constructing bayesian networks finitestate machine simulation fault tolerance further information faulttolerant computer system parallel computing can also be applied to the design of faulttolerant computer systems particularly via lockstep systems performing the same operation in parallel this provides redundancy in case one component fails and also allows automatic error detection and error correction if the results differ these methods can be used to help prevent singleevent upsets caused by transient errors although additional measures may be required in embedded or specialized systems this method can provide a cost effective approach to achieve nmodular redundancy in commercial offtheshelf systems history history of computing illiac iv the most infamous of supercomputers the origins of true mimd parallelism go back to luigi federico menabrea and his sketch of the analytic engine invented by charles babbage in april 1958 s gill ferranti discussed parallel programming and the need for branching and waiting it was during this debate that amdahls law was coined to define the limit of speedup due to parallelism in 1969 honeywell introduced its first multics system a symmetric multiprocessor system capable of running up to eight processors in parallel simd parallel computers can be traced back to the 1970s the motivation behind early simd computers was to amortize the gate delay of the processors control unit over multiple instructions when it was finally ready to run its first real application in 1976 it was outperformed by existing commercial supercomputers such as the cray1 biological brain as massively parallel computer in the early 1970s at the mit computer science and artificial intelligence laboratory marvin minsky and seymour papert started developing the society of mind theory which views the biological brain as massively parallel computer in 1986 minsky published the society of mind which claims that mind is formed from many little agents each mindless by itself similar models which also view biological brain as massively parallel computer ie the brain is made up of a constellation of independent or semiindependent agents were also described by thomas r blakeslee michael s gazzaniga robert e ornstein ernest hilgard michio kaku george ivanovich gurdjieff neurocluster brain model parallel computing portal concurrency computer science content addressable parallel processor list of distributed computing conferences list of important publications in concurrent parallel and distributed computing manycore multi tasking parallel programming model serializability synchronous programming transputer vector processing rodriguez c villagra m baran b 29 august 2008 asynchronous team algorithms for boolean satisfiability bioinspired models of network information and computing systems 2007 bionetics 2007 2nd 6669 doi101109bimnics20074610083 sechin a parallel computing in photogrammetry gim international 1 2016 pp2123 listen to this article infodl this audio file was created from a revision of the article parallel computing dated 20130821 and does not reflect subsequent edits to the article audio help more spoken articles wikibooks has a book on the topic of distributed systems wikiversity has learning resources about parallel computing instructional videos on caf in the fortran standard by john reid see appendix b parallel computing at curlie lawrence livermore national laboratory introduction to parallel computing designing and building parallel programs by ian foster internet parallel computing archive parallel processing topic area at ieee distributed computing online parallel computing works free online book frontiers of supercomputing free online book covering topics like algorithms and industrial applications universal parallel computing research center course in parallel programming at columbia university in collaboration with ibm tj watson x10 project parallel and distributed grbner bases computation in jas see also grbner basis course in parallel computing at university of wisconsinmadison berkeley par lab progress in the parallel computing landscape editors david patterson dennis gannon and michael wrinn august 23 2013 the trouble with multicore by david patterson posted 30 jun 2010 the landscape of parallel computing research a view from berkeley one too many dead link at this site introduction to parallel computing coursera parallel programming vteparallel computinggeneral distributed computing parallel computing massively parallel cloud computing highperformance computing multiprocessing manycore processor gpgpu computer network systolic array levels bit instruction thread task data memory loop pipeline multithreading temporal simultaneous smt speculative spmt preemptive cooperative clustered multithread cmt hardware scout theory pram model analysis of parallel algorithms amdahls law gustafsons law cost efficiency karpflatt metric slowdown speedup elements process thread fiber instruction window array data structure coordination multiprocessing memory coherency cache coherency cache invalidation barrier synchronization application checkpointing programming stream processing dataflow programming models implicit parallelism explicit parallelism concurrency nonblocking algorithm hardware flynns taxonomy sisd simd simt misd mimd dataflow architecture pipelined processor superscalar processor vector processor multiprocessor symmetric asymmetric memory shared distributed distributed shared uma numa coma massively parallel computer computer cluster grid computer hardware acceleration apis ateji px boostthread chapel charm cilk coarray fortran cuda dryad c amp global arrays mpi openmp opencl openhmpp openacc tpl plinq pvm posix threads raftlib upc tbb zpl problems deadlock livelock deterministic algorithm embarrassingly parallel parallel slowdown race condition software lockout scalability starvation category parallel computing media related to parallel computing at wikimedia commons 