parallelizing ghc

[Evan Laforge]

> I like the idea!  And it should be possible to build this without modifying
> GHC at all, on top of the GHC API.  As you say, you'll need a server
> process, which accepts command lines, executes them, and sends back the
> results.  A local socket should be fine (and will work on both Unix and
> Windows).

I took a whack at this, but I'm having to backtrack a bit now because
I don't fully understand the GHC API, so I thought I should explain my
understanding to make sure I'm on the right track.

It appears the cached information I want to preserve between compiles
is in HscEnv.  At first I thought I could just do what --make does,
but what it does is call 'GHC.load', which maintains the HscEnv (which
mostly means loading already compiled modules into the
HomePackageTable, since the other cache entries are apparently loaded
on demand by DriverPipeline.compileFile).  But actually it does a lot
of things, such as detecting that a module doesn't need recompilation
and directly loading the interface in that case.  So I thought it
would be quickest to just use it: add a new target to the set of
targets and call load again.

However, there are problems with that.  The first is it doesn't pay
attention to DynFlags.outputFile, which makes sense because it's
expecting to compile multiple files.  The bigger problem is that it
apparently wants to reload the whole set each time, so it winds up
being slower rather than faster.  I guess 'load' is really set up to
figure out dependencies on its own and compile a set of modules, so
I'm talking at the wrong level.

So I think I need to rewrite the HPT-maintaining parts of GHC.load and
write my own compileFile that *does* maintain the HPT.  And also
figure out what other parts of the HscEnv should be updated, if any.
Sound about right?


Along the way I ran into the problem that it's impossible to re-parse
GHC flags to compare them to previous runs, because static flags only
export a parsing function that mutates global variables and can only
be called once.  So I parse out the dynamic flags, strip out the *.hs
args, and assume the rest are static flags.  I noticed comments about
converting them all to dynamic, I guess that might make a nice
housekeeping project some day.