Suggestion re altering the build system

[Manuel M T Chakravarty]

Simon Marlow wrote,
> Manuel M T Chakravarty wrote:
>> In the last couple of weeks I spent a lot of time just trying to 
>> compile GHC.  With the constantly changing build system, incremental 
>> builds fail more often than not after new patches are pulled.  And 
>> sometimes even clean builds don't work.  For example, today (that is 
>> Monday morning in .au) I could neither build the ghc-ndp branch nor 
>> the head pulling the ghc sources and packages straight of darcs.
>>
>> I am maybe in an especially bad situation by working on the head and a 
>> branch (ghc-ndp), but we sync the branch fairly often (although that's 
>> been discouraged by the constant build problems).  I am also building 
>> on MacOS, which once in a while presents its on kinks (eg, readline 
>> breaking with the change to the Cabal build system until Roman fixed it).
>>
>> Nevertheless, I don't think I am alone.  For example, not long ago I 
>> met Andy Gill in #ghc trying to find a version of the head that's 
>> fairly recent and still builds.
>>
>> So, let me make a suggestion.  Could the changes to the build system 
>> maybe done on a branch of the repos and be integrated into the live 
>> tree in larger intervals?  Then, we'd have the pain in bursts, but 
>> less often.  I appreciate that this is not entirely trivial as you 
>> have to branch the packages along with the ghc repo, but the current 
>> situation is extremely frustrating.
> 
> I feel your frustration, stability of the HEAD does seem to have been a 
> problem of late.  Looking through the logs though, there was only one 
> day in the last two weeks in which there was no successful build at 
> all.  So, problems arise and are generally fixed quickly.  We're still 
> improving our nightly build infrastructure, so I expect this situation 
> to improve in the future - e.g. we don't currently get notifications as 
> soon as one of the builds fails, but Ian plans to add this to the IRC 
> bot, which should mean we're able to respond more quickly.
> 
> We should try to use the "FIX BUILD" convention more consistently.  
> Hopefully that would allow people to get from a broken build to a 
> working one more quickly and reliably.
> 
> I'm not sure that working on a branch would help here.  In a sense the 
> build system work *is* being done in a branch - a private branch, and 
> making the branch public wouldn't really help, because nobody else would 
> be testing it.  I suspect we'd have a similar batch of failures when the 
> code was merged into the HEAD and tested in a wider variety of 
> configurations than Ian can reasonably test on his own.  Add to that the 
> extra overhead of developing on a branch (with multiple repos) and I'd 
> say this isn't the way forward.

Maybe the work is on a branch, but it is pushed into the head frequently 
enough that it doesn't make a difference.  And I don't agree that a 
public branch wouldn't help.  A public branch with a call for wider 
testing at points where you think it actually works would get it at 
least some testing before it hits everybody.

> Incremental builds after pulling patches can fail for many reasons, 
> which is why we recommend doing a distclean and building from scratch 
> after pulling.  There are simply too many dependencies for the build 
> system to track them all - for example, we can't tell when the interface 
> format changes and the libraries need to be rebuilt.

I understand that incremental builds are more tricky, but during 
teaching session I often have only 2 hours to hack on a day.  If I have 
to download a fresh repo and build from scratch before I start, I can as 
well leave it.  So, I of course try to work from where I am.  If that 
doesn't work then I clean components, and if that doesn't work than I 
start from scratch.  If, like this Monday, that doesn't work either, I 
am *very* annoyed.

(And I am not worried about changing interface file formats, I can 
figure that out fine.  Its the build system changes, which screwed me up 
most.)

> We definitely need more nightly builds, in particular we don't have a 
> current MacOS X nightly build - why is that?  There are several 
> registered, but they all seem to be offline.

I'd like to offer a host, but our Macs are all laptops.

Manuel