parallelizing ghc

[Simon Marlow]

On 26/01/2012 23:37, Evan Laforge wrote:
>> I'm slightly surprised by this - in my experience parallel builds beat
>> --make as long as the parallelism is a factor of 2 or more.  Is your
>> dependency graph very narrow, or do you have lots of very small modules?
>
> I get full parallelism, 4 threads at once on a 2 core machine * 2
> hyperthread/whatever core i5, and SSD.  Maybe I should try with just 2
> threads.  I only ever get 200% CPU at most, so it seems like the
> hyperthreads are not really much like a whole core.
>
> The modules are usually around 150-250 lines.  Here are the timings
> for an older run:
>
>      from scratch (191 modules):
>      runghc Shake/Shakefile.hs build/debug/seq  128.43s user 20.04s system 178%
> cpu 1:23.01 total
>      no link: runghc Shake/Shakefile.hs build/debug/seq  118.92s user 19.21s sys
> tem 249% cpu 55.383 total
>      make -j3 build/seq  68.81s user 9.98s system 98% cpu 1:19.60 total

This looks a bit suspicious.  The Shake build is doing nearly twice as 
much work as the --make build, in terms of CPU time, but because it is 
getting nearly 2x parallelism it comes in a close second.  How many 
processes is the Shake build using?

I'd investigate this further.  Are you sure there's no swapping going 
on?  How many processes is the Shake build creating - perhaps too many?

Cheers,
	Simon