nofib

[Robert Ennals]

> I think two modes, at the most, would be useful.  The nofib suite is
> generally used in two ways (but perhaps you can think of more):  =

> =

>   (a) to run in batch mode to compare the full results against
>       a previous compiler.  For this purpose we archive the
>       results from previous compilers (for a given machine, of
>       course).
> =

>   (b) interactively, when bug-hunting or performance tuning.

I would add the following:

(c) Running highly augmented code in order to gather some kind of informa=
tion. =

Eg, making a log of usage patterns of thunks - generating masses of profi=
ling =

information. Nofib can be a useful example set for this. Such tests need =
to =

run very quickly when compiled normally, or they are likely to take too l=
ong =

when running with masses of crud hanging off them.

(d) Comparing the performance of one compilation approach against another=
=2E =

(essentially "a", but without the same archiving requirements). This requ=
ires =

tests to run for long enough to be able to give statistically useful timi=
ng =

results.


If we don't particularly care about being able to compare with "old nofib=
", =

then I would think that two modes would be sensible.

Fast: Very quick, good for interactive bug hunting, and augmented code.

Slow: Around 10-20seconds for all tests.


But, seeing as leaving the existing "normal" mode in there as well seems =
to be =

very little effort, and it might possibly be useful, I see no reason not =
to do =

so.

> Given that we're archiving results, we're only going to run the suite i=
n
> one of the "modes" when doing a batch run, so we don't have to archive
> results for each mode.  For interactive use, a faster mode is going to
> be more useful (but not for bug hunting, where you want
> reproduceability).

"Slow" mode would probably be the most sensible mode for this, as it woul=
d =

give the most useful numbers.

Or use "normal" if you want to compare to the archived old results from =

earlier NoFib.
 =

> Alternatively, you might want to consider normalising each test so that=

> all the tests run in roughly the same time (say 1 second), and then
> having a global "iterations" parameter that can be tweaked to alter the=

> runtime of the whole suite.  But do you want the program itself to
> handle the extra iterations, or the build system?  Probably the program=

> if possible, since it reduces the interference from external factors.

I suspect that that might fall victim to "suprise optimisations" where GH=
C =

noticed that the iterations were doing the same thing, and cleverly only =
did =

the work once.

Probably easier to just make sure every test has some kind of variable in=
put.


Or have I got the wrong end of the stick here?



-Rob