This came up in a question in the post where I started to talk about π-calculus, but I thought it was an interesting enough topic to promote it up to a top-level post. If you listen to anyone talking about computers or software, there are three worlds you’ll constantly hear: parallel, concurrent, and distributed. At first glance, it sounds like they mean the same thing, but in fact, they’re three different things, and the differences are important.
The connection between them is that they’re all terms that describe systems made up of computers and software that are doing more than one thing at a time. The difference is are in why and how they do it.
Parallelism, or parallel code, or parallel systems talks about how to take a given system, and make it run faster by breaking into pieces that can run simultaneously. So suppose you want to do something really complicated. It’s got three steps, A, B, and C. A and B each prepare things for C, but they don’t interact with each other at all. Then you can run that by running A, waiting until it’s done, then running B, waiting until it’s done, and then running C. Or, if you’ve got a spare processor, you could run A and B simultaneously, and then when they’re done, run C. When you’re making a program run parts at the same time in order to make it
run faster, then you’re doing parallelism.
Concurrency talks about systems that have multiple parts that are designed with the explicit goal of taking place simultaneously, not because it makes it faster, but because it’s a necessary part of the functionality of the system. The backend system that we use for ScienceBlogs handles lots of concurrency, because it’s designed to simultaneously support thousands of readers viewing pages at the same time, as well as allowing us to write and publish new posts without interfering with the operation of the system. If things happening at the same time is a necessary part of the semantics of your system, then you’re doing concurrency.
Distribution is talking about systems that are made up of multiple physical parts
connected by a communication network. It’s fundamentally a study of how to build systems where
the system itself is broken into physical pieces, which may be located in different places, have a variety of constraints on communication, etc. If your system is is specifically designed to be run as multiple programs running simultaneously on on many different pieces of hardware, but behaving in some sense as a single system, then you’re doing a distributed system.
So, an example of each:
- Weather forecasting software is usually parallel code. Doing the computational fluid
dynamics work to generate accurate predictions requires an enormous amount of computation,
and dividing it up among many processors makes it run at a (more) reasonable rate.
- Database systems are often built for concurrency. The idea is that there’s a huge database
somewhere, and it’s being pounded with lots and lots of queries. When one user starts
a query, the system doesn’t shut down and stopping doing anything else until that query
is done. It allows multiple users to be running queries all at the same time. And most
databases even guarantee that if one user is performing an update query, other users that are performing concurrent queries while that update is in progress, the queries will return consistent results representing the state of the database either before or after the update, but not a combination of the two.
- An example of a distributed system would be a piece of software like writely, which is
a word processor that runs inside of a web browser. In writely, you can edit a document
in your web-browser, and you can share editing with multiple people – so you can have three or four web browsers all editing the same document. In terms of the system, the web browsers are
each running little Java applications that talk to the server and each other by messaging; and they have absolutely none of the code for actually changing the document. The server has no code for doing things like rendering the UI, but it communicates with the clients to get
receive and process edit commands, and send out updates so that all of the UIs are rendering
the same thing. The entire design of the system is built around the idea that there are
these multiple pieces, each running on different machines.