Difference between revisions of "Parallelism vs. Concurrency"

From HaskellWiki
Jump to navigation Jump to search
(heading for the anecdote)
(Clarifying quote added; various other changes)
 
(4 intermediate revisions by 2 users not shown)
Line 1: Line 1:
  +
== What the difference is ==
−
The term '''Parallelism''' refers to techniques to make programs faster by performing several computations in parallel.
  
⚫
This requires hardware with multiple processing units.
  
⚫
In many cases the sub-computations are of the same structure, but this is not necessary.
  
⚫
Graphic computations on a GPU are parallelism.
  
   
  +
<blockquote>
−
The term '''Concurrency''' refers to techniques that make program more usable.
  
  +
I make a sharp distinction between ''parallelism'' and ''concurrency'':
⚫
Concurrency can be implemented and is used a lot on single processing units,
  
  +
  +
* A ''parallel'' functional program uses multiple processors to gain performance. For example, it may be faster to evaluate ''e''<sub>1</sub> + ''e''<sub>2</sub> by evaluating ''e''<sub>1</sub> and ''e''<sub>2</sub> in parallel, and then add the results. Parallelism has no semantic impact at all: the meaning of a program is unchanged whether it is executed sequentially or in parallel. Furthermore, the results are deterministic; there is no possibility that a parallel program will give one result in one run and a different result in a different run.
  +
  +
* In contrast, ''concurrent'' program has concurrency as part of its specification. The program must run concurrent threads, each of which can independently perform input/output. The program may be run on many processors, or on one — that is an implementation choice. The behaviour of the program is, necessarily and by design, non-deterministic. Hence, unlike parallelism, concurrency has a substantial semantic impact.
  +
  +
:<small>[https://www.cs.tufts.edu/comp/150PLD/Papers/awkward.pdf Tackling the Awkward Squad;] Simon Peyton Jones (page 28 of 60)</small>
  +
</blockquote>
  +
  ⚫
[[Parallelism]] requires hardware with multiple processing units.
  ⚫
While the sub-computations can be of the same structure, this is not necessary.
  ⚫
Graphic computations on a GPU are an example of parallelism.
  +
A key problem of parallelism is to reduce data dependencies
  +
in order to be able to perform computations on independent computation units
  +
with minimal communication between them.
  +
To this end, it can even be an advantage to do the same computation twice on different units.
  +
  ⚫
[[Concurrency]] can be implemented and is used a lot on single processing units,
 
nonetheless it may benefit from multiple processing units with respect to speed.
  
nonetheless it may benefit from multiple processing units with respect to speed.
 
If an operating system is called a multi-tasking operating system, this is a synonym for supporting concurrency.
  
If an operating system is called a multi-tasking operating system, this is a synonym for supporting concurrency.
 
If you can load multiple documents simultaneously in the tabs of your browser and
  
If you can load multiple documents simultaneously in the tabs of your browser and
−
you can still open menus and perform more actions, this is concurrency.
 +
you can still open menus and perform other actions, this is an example of concurrency.
⚫
If you run distributed-net computations in the background, that is concurrency.
  
   
  ⚫
== How to distinguish between parallelism and concurrency ==
⚫
== An anecdote from good old Amiga days ==
  
  +
  +
* dividing a task into packets that can be computed via distributed-net clients: ''parallelism''.
  +
  ⚫
* running distributed-net computations in the background while working with interactive applications in the foreground: ''concurrency''.
  +
  ⚫
* if you need <hask>getNumCapabilities</hask> in your program: ''parallelism''.
  +
  ⚫
* if those parallelising efforts make sense on a single-processor machine, too: ''concurrency''.
  +
  +
If you're not sure, the [[Parallel/Glossary|glossary]] of terms specific to parallelism and concurrency could also help.
  +
  ⚫
== An anecdote from the good old Amiga days ==
  +
  +
How concurrency by itself can be useful:
   
−
Let me tell an anecdote to further sharpen the difference:
  
 
Amiga computers were always advertised for their multi-tasking operating system.
  
Amiga computers were always advertised for their multi-tasking operating system.
 
However DOS/Windows-3.1 users were never attracted by this advertisement
  
However DOS/Windows-3.1 users were never attracted by this advertisement
 
since they argued that a single CPU cannot be made faster by performing several tasks in an interleaved way.
  
since they argued that a single CPU cannot be made faster by performing several tasks in an interleaved way.
  +
−
They were right, but this was not the point: Multitasking allows to avoid that the computer gets bored.
 +
They were right, but this was not the point: multitasking avoids the computer and that one CPU "sitting idle".
⚫
Indeed in the eighties Amiga computers were considered great for raytracing.
  
  +
  ⚫
Indeed in the 1980s Amiga computers were considered great for raytracing.
 
However the special graphics and sound hardware in Amiga computers could not help with raytracing.
  
However the special graphics and sound hardware in Amiga computers could not help with raytracing.
 
The important advantage was, that you could perform the graphics rendering concurrently to your daily work (office applications) without noticing the computation load of the raytracing.
  
The important advantage was, that you could perform the graphics rendering concurrently to your daily work (office applications) without noticing the computation load of the raytracing.
 
Multitasking just assigns the time between your keystrokes to the raytracer.
  
Multitasking just assigns the time between your keystrokes to the raytracer.
⚫
However multitasking was not possible with most games, office software that eats all the memory or simply crashing applications.
  
−
This leads to another confusing area: [[Error vs. Exception]].
  
   
  ⚫
However multitasking was not possible with most games, office software that eats all the memory or simply crashing applications. (which leads to another confusing area: [[Error vs. Exception]]).
⚫
== How to distinguish between Parallelism and Concurrency ==
  
   
  +
== A note of caution ==
⚫
* If you need <hask>getNumCapabilities</hask> in your program, then your are certainly programming parallelism.
  
⚫
* If your parallelising efforts make sense on a single processor machine, too, then you are certainly programming concurrency.
  
−  
−
== Warning ==
  
   
 
Not all programmers agree on the meaning of the terms 'parallelism' and 'concurrency'.
  
Not all programmers agree on the meaning of the terms 'parallelism' and 'concurrency'.
−
They may define them in different ways or do not distinguish them at all.
 +
They may define them in different ways or do not distinguish them at all. Hence:
  +
  +
* [[Applications and libraries/Concurrency and parallelism]]
  +
* [[Research papers/Parallelism and concurrency]]
  +
* [http://chimera.labs.oreilly.com/books/1230000000929/index.html Parallel and Concurrent Programming in Haskell]
  +
  +
...amongst others!
   
 
== See also ==
  
== See also ==
Line 40: Line 69:
 
* Note of Simon Marlow in [http://www.haskell.org/pipermail/haskell-cafe/2008-September/047312.html Haskell-Cafe] about the distinction of parallelism and concurrency
  
* Note of Simon Marlow in [http://www.haskell.org/pipermail/haskell-cafe/2008-September/047312.html Haskell-Cafe] about the distinction of parallelism and concurrency
 
* GHC mutterings on [http://ghcmutterings.wordpress.com/2009/10/06/parallelism-concurrency/ Parallelism /= Concurrency]
  
* GHC mutterings on [http://ghcmutterings.wordpress.com/2009/10/06/parallelism-concurrency/ Parallelism /= Concurrency]
  +
   
 
[[Category:Idioms]]
  
[[Category:Idioms]]

Latest revision as of 12:02, 9 May 2024

What the difference is

I make a sharp distinction between parallelism and concurrency:

  • A parallel functional program uses multiple processors to gain performance. For example, it may be faster to evaluate e1 + e2 by evaluating e1 and e2 in parallel, and then add the results. Parallelism has no semantic impact at all: the meaning of a program is unchanged whether it is executed sequentially or in parallel. Furthermore, the results are deterministic; there is no possibility that a parallel program will give one result in one run and a different result in a different run.
  • In contrast, concurrent program has concurrency as part of its specification. The program must run concurrent threads, each of which can independently perform input/output. The program may be run on many processors, or on one — that is an implementation choice. The behaviour of the program is, necessarily and by design, non-deterministic. Hence, unlike parallelism, concurrency has a substantial semantic impact.
Tackling the Awkward Squad; Simon Peyton Jones (page 28 of 60)

Parallelism requires hardware with multiple processing units. While the sub-computations can be of the same structure, this is not necessary. Graphic computations on a GPU are an example of parallelism. A key problem of parallelism is to reduce data dependencies in order to be able to perform computations on independent computation units with minimal communication between them. To this end, it can even be an advantage to do the same computation twice on different units.

Concurrency can be implemented and is used a lot on single processing units, nonetheless it may benefit from multiple processing units with respect to speed. If an operating system is called a multi-tasking operating system, this is a synonym for supporting concurrency. If you can load multiple documents simultaneously in the tabs of your browser and you can still open menus and perform other actions, this is an example of concurrency.

How to distinguish between parallelism and concurrency

  • dividing a task into packets that can be computed via distributed-net clients: parallelism.
  • running distributed-net computations in the background while working with interactive applications in the foreground: concurrency.
  • if you need getNumCapabilities in your program: parallelism.
  • if those parallelising efforts make sense on a single-processor machine, too: concurrency.

If you're not sure, the glossary of terms specific to parallelism and concurrency could also help.

An anecdote from the good old Amiga days

How concurrency by itself can be useful:

Amiga computers were always advertised for their multi-tasking operating system. However DOS/Windows-3.1 users were never attracted by this advertisement since they argued that a single CPU cannot be made faster by performing several tasks in an interleaved way.

They were right, but this was not the point: multitasking avoids the computer and that one CPU "sitting idle".

Indeed in the 1980s Amiga computers were considered great for raytracing. However the special graphics and sound hardware in Amiga computers could not help with raytracing. The important advantage was, that you could perform the graphics rendering concurrently to your daily work (office applications) without noticing the computation load of the raytracing. Multitasking just assigns the time between your keystrokes to the raytracer.

However multitasking was not possible with most games, office software that eats all the memory or simply crashing applications. (which leads to another confusing area: Error vs. Exception).

A note of caution

Not all programmers agree on the meaning of the terms 'parallelism' and 'concurrency'. They may define them in different ways or do not distinguish them at all. Hence:

...amongst others!

See also

Retrieved from "https://wiki.haskell.org/index.php?title=Parallelism_vs._Concurrency&oldid=66639"