Daily report for head

[Simon Marlow]

Thorkil Naur wrote:

> With the tnaur PPC OSX buildbot slave back in business,

yay!  thanks for getting it going again.

> I have finally spent 
> the effort to figure out in detail what these reports actually tell me. As I 
> understand the buildbot-collator code, an element in the lists of unexpected 
> passes and failures gives the test name, the number of builders that report 
> the unexpected result, and the name of the "smallest" of these builders. And 
> "smallest" is defined in Data.hs as the most preferred builder for debugging 
> the problem according to suitable criteria. So, for example, the line
> 
>     GMapAssoc               3  x86-64 Linux head
> 
> among the "Old unexpected test failures" tells me that 3 builders got this 
> unexpected failure and the smallest of these were "x86-64 Linux head".
> 
> I would suggest extending this scheme by listing, for each test, all the 
> builders, sorted, rather than just the smallest. Perhaps simply on a single 
> line, although more elaborate formatting could be imagined. But in such a way 
> that it was easy to see, for a particular test, which builders actually 
> reported the unexpected result.

It also doesn't tell you which "ways" failed for this test, so in practice you 
probably have to go and look at the actual log.  If you're interested in 
failures for one particular build (yours), then this is the best place to look 
in any case.  e.g.

http://darcs.haskell.org/buildbot/head/tnaur%20PPC%20OSX%20head/builds/66/step-runtestsuite/2

It's pretty clear that we have some kind of problem with the "threaded1" way on 
PPC/OSX.


FWIW, I think listing all the builds that failed a particular test would help as 
long as it didn't overflow 80 columns, since then it would be a lot less readable.

Perhaps the easiest thing to do is attach the test summaries from each build?

Another thing we could do is distinguish between tests that are failing on all 
the platform we have seen, and tests that have succeeded on some platforms and 
failed on others.  Unfortunately it's hard to know which tests were only *run* 
on certain platforms, so this is a bit tricky.

Cheers,
	Simon