Mon, 17 Dec 2001 13:08:50 +0000
> I think two modes, at the most, would be useful. The nofib suite is
> generally used in two ways (but perhaps you can think of more): =
> (a) to run in batch mode to compare the full results against
> a previous compiler. For this purpose we archive the
> results from previous compilers (for a given machine, of
> (b) interactively, when bug-hunting or performance tuning.
I would add the following:
(c) Running highly augmented code in order to gather some kind of informa=
Eg, making a log of usage patterns of thunks - generating masses of profi=
information. Nofib can be a useful example set for this. Such tests need =
run very quickly when compiled normally, or they are likely to take too l=
when running with masses of crud hanging off them.
(d) Comparing the performance of one compilation approach against another=
(essentially "a", but without the same archiving requirements). This requ=
tests to run for long enough to be able to give statistically useful timi=
If we don't particularly care about being able to compare with "old nofib=
then I would think that two modes would be sensible.
Fast: Very quick, good for interactive bug hunting, and augmented code.
Slow: Around 10-20seconds for all tests.
But, seeing as leaving the existing "normal" mode in there as well seems =
to be =
very little effort, and it might possibly be useful, I see no reason not =
to do =
> Given that we're archiving results, we're only going to run the suite i=
> one of the "modes" when doing a batch run, so we don't have to archive
> results for each mode. For interactive use, a faster mode is going to
> be more useful (but not for bug hunting, where you want
"Slow" mode would probably be the most sensible mode for this, as it woul=
give the most useful numbers.
Or use "normal" if you want to compare to the archived old results from =
> Alternatively, you might want to consider normalising each test so that=
> all the tests run in roughly the same time (say 1 second), and then
> having a global "iterations" parameter that can be tweaked to alter the=
> runtime of the whole suite. But do you want the program itself to
> handle the extra iterations, or the build system? Probably the program=
> if possible, since it reduces the interference from external factors.
I suspect that that might fall victim to "suprise optimisations" where GH=
noticed that the iterations were doing the same thing, and cleverly only =
the work once.
Probably easier to just make sure every test has some kind of variable in=
Or have I got the wrong end of the stick here?