NoFib for benchmarking vs testing

Robert Ennals Robert.Ennals@cl.cam.ac.uk
Mon, 17 Dec 2001 16:11:50 +0000


Looking through NoFib some more, I am increasingly getting the impression that 
quite a few of the tests aren't particularly useful benchmarks.


Currently NoFib is both a benchmark suite and a test suite.

I think it might be useful to separate those roles out, giving us separate 
"test" and "benchmark" modes.

When run in "benchmark" mode, we would ignore tests that don't make good 
benchmarks.


While this wouldn't involve any significant change to the tests themselves, it 
would make it clearer which tests were thought to be useful benchmarks for 
people to test optimisations against.


[I don't think it would be particularly useful for me to work out a way for 
"awards" to take 10 seconds]


-Rob