Head broken again

[Simon Marlow]

I've just been chatting with Simon about this, and basically we're willing to 
try a more rigorous testing regime for changes to the HEAD.

Yesterday Ian and I uncovered a problem with the "known buildable" repositories 
idea: the build slave has a partial repository, so it can't necessarily push to 
the buildable repository because it doesn't necessarily have the patches it 
needs to push.  We could solve this by giving the builders full repositories, 
but it's yet more fiddling around with the buildbot setup.

So we're thinking of putting this idea to one side for now, and trying the 
suggestions from this thread instead.  The choices aren't mutually exclusive, so 
we can still do the "known buildable" repositories later (when darcs' lazy 
partial repositories are available it'll be much easier).

Here's our draft game plan.  There would be a minimal testing requirement for 
patches to HEAD, consisting of

  - a 2-stage bootstrap, probably with the "quick" settings, core libs only
  - a 'make fast' in testsuite, with zero failures

in addition, if you make changes that might break something that isn't tested by 
the above (multiple platforms, older versions of GHC, profiling, unregisterised, 
GHCi, binary distributions, HPC etc.) then it is your responsibility to make a 
reasonable attempt to test it.  We'll only hand out the pointy hats to people 
who fail to do minimal testing, though.

To make testing easier, we'll add an automated 'validate' script which does the 
complete build:

  - make distclean
  - sh boot
  - configure
  - put an appropriate build.mk in place
  - make
  - make -C testsuite fast

and if it gets all the way through and reports zero failures, you're ok to push.

The standard workflow would be to keep two trees: one for developing in, and one 
for testing.  You record patches in developing, and occasionally pull into 
testing and run 'validate'.  When it succeeds, you push from testing.  Of course 
you can have multiple developing trees and a single testing tree.  Your 
developing trees can lag behind HEAD, but the testing tree should be up to date 
with HEAD before running validate.

Right now the fast testsuite doesn't have zero failures (it has about 20), we'll 
need to get that down to zero first.  The fast testsuite currently runs in about 
8 minutes on a fast Linux machine; it was about 5 minutes when I first did it, 
so we probably need some tuning there.  I have no idea why the build is 
currently slower than it was (indeed I wasn't aware that it was), so we should 
look into that too.

Comments?

Cheers,
	Simon x 2