My current speed-testing validating technique

[Isaac Dupree]

Simon Marlow wrote:
> This is certainly a useful test, but I use a slightly different method 
> to check for performance regressions when I'm trying small modifications 
> to the compiler.  This isn't as foolproof, but in some ways it's more 
> sensitive and it's deterministic.

Okay, allocations are a reasonable indication of performance for 
GHC-compiled programs, I will keep that in mind...

>   cd nofib/anna
>   ../../compiler/stage2/ghc-inplace --make Main +RTS -sstderr

seems it's nofib/real/anna now

> The important thing here is that you get to see the amount of 
> allocation, which doesn't vary much from run to run of GHC.

True. `rm *.o *.hi Main` seems to reset the test perfectly.

>  If the 
> allocations don't change much, then you have a pretty good indication 
> that you haven't made GHC any slower (though not foolproof, of course).  
> Isaac, I recommend doing this test with your FastString changes.

Second includes FastString and FiniteMap changes above the first (that's 
what builds I have convenient now)... and I'm just using the stage1, 
which will certainly have some effect on the results (similar effect on 
both)
first:
      40,328 bytes allocated in the heap
       1,500 bytes copied during GC (scavenged)
       4,108 bytes copied during GC (not scavenged)
      30,176 bytes maximum residency (1 sample(s))
second:
      40,632 bytes allocated in the heap
       1,500 bytes copied during GC (scavenged)
       4,108 bytes copied during GC (not scavenged)
      30,176 bytes maximum residency (1 sample(s))

Is that the main interesting data that is output? How should I interpret 
that?

Of course I can test other patch combinations at approximately 15 
minutes each (to rebuild GHC+libs... maybe that can be shortened by 
taking some shortcuts. Certainly if the changes aren't supposed to 
change ghc's behavior _at all_, I can probably skip rebuilding the libs)

Isaac