how to check for performance regressions caused by ghc hacking?

[Simon Marlow]

Stefan O'Rear wrote:
> On Sun, Jun 03, 2007 at 05:59:34PM -0400, Isaac Dupree wrote:
>> Hash: SHA1
>>
>> While working on making GHC more compilable by other compilers, and
>> seeing some code that would be nicer-looking refactored anyway: I want
>> to change the code, but I'm afraid I'll make GHC's speed worse (these
>> are critical things like FastString and FiniteMap).  I could send my
>> patches to the list, but would the reviewers know any better than me
>> just by looking at it? (I'm much better at mental
>> strictness/sharing/etc. analysis than I used to be, at least) ... so,
>> what's a good way to observe the overall speed that a GHC compiles
>> things at? (Is or should this be somewhere in the GHC Commentary?  I
>> didn't find anything relevant there)
> 
> We already have a Haskell runtime performance testing suite:
> 
> http://www.cse.unsw.edu.au/~dons/code/nobench
> 
> Extending it to measure compile times shouldn't be hard.  Note that
> you'll have to set up your own instance, because the public instance
> won't have ghc-isaacd.
> 
> (Of course, compile time measurement patches should be sent upstream)

Right, extending nobench to do this would be great.

You can also use the existing nofib suite, together with utils/nofib-analyse, to 
get compile times compared across two separate runs.  This is pretty unreliable 
for measuring small changes, though.

When I make a small change to the compiler, as a performance sanity check I 
often go and compile nofib/anna with and without the change.  Something like this:

   $ cd nofib/anna
   $ ghc -O --make -cpp Main +RTS -sstderr
   $ ghc-new -O --make -cpp Main +RTS -sstderr

note that results from this test shouldn't be considered definitive - probably 
compiling GHC with itself would be a better test.

Cheers,
	Simon