Massively parallel computing is something I've been interested in for a while and have blogged about a few times in the past. This blog entry by an Intel Researcher made me think about it again. He continues to proclaim that the future isn't that far away and we had better start designing our software so that it can run on machines with thousands of cores. He worries that we're aren't ready yet and we need to start getting ready. And he's right.
Being a tools guy, I think this is the next big paradigm that the tooling industry needs to address. Object-oriented programming and design was a godsend when machines started scaling up in the size of memory and storage and our programs began filling that with data. We built a lot of tools to help with that. Programming languages and compilers are obvious examples. But so is the JDT and CDT, with their code analysis to show type hierarchies and help you easily find classes. Not to mention all the object modeling tools for drawing pictures of your classes.
Coming up with the languages and compilers and other tools necessary to deal with thousands of concurrently running threads is our next great challenge. This is why I keep one eye on the Parallel Tools Project at Eclipse. They're already in this world dealing with the thousands of processors that run the super computers they work with. This effort is a research project in itself (quite literally if you notice who participates in this project :).
But as the Intel researcher warns, this stuff is going to hit the mainstream soon. We're starting to see that with OpenMP parallel language extensions supported in almost all recent compiler releases, including gcc. And I'm convinced it's an area where modeling can help since you really need to think of your program in multiple dimensions, which is something modeling is good at.
I think it's a matter of time before we're at the head of a new paradigm. I remember the fun we had when object-oriented programming hit the mainstream. I think this one will be just as fun.
The trend with programming platforms and tools has been towards improving programmer productivity. That's why we love garbage collection, it may make your program slower, but it allows you to write programs a lot faster. Programmer time is often more important than processing time.
ReplyDeleteThe problem with parallel programming is that it can be hard, really hard. Deadlocks and race situations abound. And its hard to debug. Basically multicore programming has the potential to kill the productivity we have come to enjoy.
What we need is a paradigm that makes parallel programming easy, kind of like how garbage collection makes memory management easy. That way your average joe-blow programmer can take advantage of it. Because if you need a PHD to code for 1000 cores, well... expect programmer productivity to come to a screeching halt.
And what if this miraculous new paradigm doesn't arrive in time. Are we going to have 1000 core desktops with 990 cores always idle.
Donald Knuth has an interesting (and skeptical) view of the multicore revolution (about 1/3 way down the page), very interesting.
Erlang.
ReplyDeleteAnd, on the JVM, Scala (which copies Erlang's concurrency model).
I'm pretty confident we'll find an answer. There's too much investment by the big silicon vendors not to.
ReplyDeleteAnd I do know Intel is investing in something called Ct (the t is for throughput). Erlang might be a better answer. At least it has some legs under it.
But I still think simple extensions to textual programming languages won't get us there. It's going to take something bigger to get us out of our 2D thinking.
Scala is the answer with its fancy pants Actors for JVM-based things.
ReplyDeleteHowever, it still takes a certain mindset to do concurrent programming... without fantastic tools to help people... we will be stuck with our current sequential mindset.
Great points here.
ReplyDeleteMy sense is that the multicore programming challenge will need to be solved well before we have 1,000 core processors. Already, compute-bound apps - such as CAD/EDA, image processing, etc. - are slower on dual-core machines.
And functional decomposition, manually separating out the various pieces of the app, will only get you so far; and worse, not scale the next time the number of cores double.
The bet we are making (at Cilk Arts) is that to get adopted, the programming paradigm has to be evolutionary, and should preserve the serial semantics of the legacy app.
Erlang has been able to achieve 9 9's of reliability in massively-scaled, *distributed* telephone switches.
ReplyDeleteAs I mentioned earlier, Scala borrows from Erlang.
The following blog post is an excellent introduction to some of the ideas behind this:
http://jonasboner.com/2008/06/16/erlang-style-supervisor-module-for-scala-actors/
"Erlang has been able to achieve 9 9's of reliability in massively-scaled, *distributed* telephone switches."
ReplyDeleteNo, Erlang didn't achieve 9 9's, the programs written with it did. Lots of programming languages facilitated 9 9's.