Head broken again

[Simon Marlow]

The HEAD now has a 'validate' script at the top level, which you can use for 
testing that patches in your tree pass minimal testing.  It won't work on 
Windows yet (no args are passed to configure), and parallelism is fixed at -j2.

'make fast' in the testsuite now runs with zero failures on x86-64/Linux.  We 
should make this the case on other platforms as a high priority.  When we hit 
zero failures on most platforms, we can start to announce the new testing regime 
for patches to the HEAD.

In the meantime I'd urge people to start using the validate script and switch to 
using separate "developing" and "testing" source trees.  I've been doing that 
for a couple of days now, and it seems to be working quite well.

I did some tests on build times, here's a quick summary.  These times are on a 
1.2GHz Core2 using 1 CPU only, bootstrapping using GHC 6.6.1, with the "quick" 
settings, core packages only:

  - 6.6.1: 17.7 minutes
  - HEAD:  19.0 minutes

Looking in more detail it seems that the difference is mostly in the libraries, 
although I'd like to do some more analysis.  It's not clear cut because we used 
to do a single configure step at the top, and now we configure the libraries 
later in the build.  Also HEAD has fewer libraries in core-packages than 6.6.1.

A full validate on a fast 2-core machine should take about 20 minutes.  We can 
get that down to 15 minutes or so using THREADS=2 for the testsuite, but 
currently a few tests fail when I do that (due to bogus ordering requirements, 
presumably).

Cheers,
	Simon

Simon Marlow wrote:
> I've just been chatting with Simon about this, and basically we're 
> willing to try a more rigorous testing regime for changes to the HEAD.
> 
> Yesterday Ian and I uncovered a problem with the "known buildable" 
> repositories idea: the build slave has a partial repository, so it can't 
> necessarily push to the buildable repository because it doesn't 
> necessarily have the patches it needs to push.  We could solve this by 
> giving the builders full repositories, but it's yet more fiddling around 
> with the buildbot setup.
> 
> So we're thinking of putting this idea to one side for now, and trying 
> the suggestions from this thread instead.  The choices aren't mutually 
> exclusive, so we can still do the "known buildable" repositories later 
> (when darcs' lazy partial repositories are available it'll be much easier).
> 
> Here's our draft game plan.  There would be a minimal testing 
> requirement for patches to HEAD, consisting of
> 
>  - a 2-stage bootstrap, probably with the "quick" settings, core libs only
>  - a 'make fast' in testsuite, with zero failures
> 
> in addition, if you make changes that might break something that isn't 
> tested by the above (multiple platforms, older versions of GHC, 
> profiling, unregisterised, GHCi, binary distributions, HPC etc.) then it 
> is your responsibility to make a reasonable attempt to test it.  We'll 
> only hand out the pointy hats to people who fail to do minimal testing, 
> though.
> 
> To make testing easier, we'll add an automated 'validate' script which 
> does the complete build:
> 
>  - make distclean
>  - sh boot
>  - configure
>  - put an appropriate build.mk in place
>  - make
>  - make -C testsuite fast
> 
> and if it gets all the way through and reports zero failures, you're ok 
> to push.
> 
> The standard workflow would be to keep two trees: one for developing in, 
> and one for testing.  You record patches in developing, and occasionally 
> pull into testing and run 'validate'.  When it succeeds, you push from 
> testing.  Of course you can have multiple developing trees and a single 
> testing tree.  Your developing trees can lag behind HEAD, but the 
> testing tree should be up to date with HEAD before running validate.
> 
> Right now the fast testsuite doesn't have zero failures (it has about 
> 20), we'll need to get that down to zero first.  The fast testsuite 
> currently runs in about 8 minutes on a fast Linux machine; it was about 
> 5 minutes when I first did it, so we probably need some tuning there.  I 
> have no idea why the build is currently slower than it was (indeed I 
> wasn't aware that it was), so we should look into that too.
> 
> Comments?
> 
> Cheers,
>     Simon x 2
> 
>