buying new hardware; want fast validate

[Simon Marlow]

Duncan Coutts wrote:
> On Wed, 2008-04-30 at 21:01 -0400, Norman Ramsey wrote:
>> Dear Fellow GHC Hackers,
>>
>> I'll be buying new computers soon, and right now the only thing I do
>> for which computer performance is a problem is GHC hacking---especially
>> running 'validate'.  I've got a good performance boost on my current
>> machine by using two SATA drives striped in a RAID-0 configuration.
>> I'm wondering what other tricks may be warranted.  Specifically
>> I'm interested in answers to the following questions:
>>
>>   1. How much benefit does one get from four cores versus two?
>>      I know that the build process runs in parallel thanks to 
>>      make -j, but I don't know how much parallelism is available
>>      in the testing part.   Maybe somebody who already has
>>      quad core can compare and let us know the results?
> 
> I recently got a quad core amd64 and I run linux. I noticed from the
> system monitor graphs that not that much of the build process can use
> many cores. I was using make -j5.
> 
> The one bit that really does benefit is building the ghc compiler
> itself. That can make full use of all 4 cores for an extended period.
> 
> The ./configure step is painfully serial and building the early tools
> and the libraries is rather serial too. There are lots of little changes
> here that we could do to make it more parallel. One example is that we
> link 20 copies of Setup.hs rather than reusing one copy. Linking is very
> much serial.
> 
> In principle the test suite should be possible to parallelise but I
> don't know much about python and parallelism.

We're pretty close to being able to run the testsuite in parallel, but 
we ran up against a Python bug:

http://hackage.haskell.org/trac/ghc/ticket/1558

perhaps we could work around that bug with a big lock or something. 
Unfortunately it only happens a handful of times over a testsuite run, 
so it's hard to work on.

This is the low-hanging fruit for speeding up validate right now. 
Anyone feel like having a crack at it?

Cheers,
	Simon