My current speed-testing validating technique

[Isaac Dupree]

Isaac Dupree wrote:
> Second includes FastString and FiniteMap changes above the first (that's 
> what builds I have convenient now)... and I'm just using the stage1, 
> which will certainly have some effect on the results (similar effect on 
> both)
> first:
>      40,328 bytes allocated in the heap
>       1,500 bytes copied during GC (scavenged)
>       4,108 bytes copied during GC (not scavenged)
>      30,176 bytes maximum residency (1 sample(s))
> second:
>      40,632 bytes allocated in the heap
>       1,500 bytes copied during GC (scavenged)
>       4,108 bytes copied during GC (not scavenged)
>      30,176 bytes maximum residency (1 sample(s))
> 
> Is that the main interesting data that is output? How should I interpret 
> that?

FWIW here's those 'time'd.

first (unpatched)
../../../compiler/ghc-inplace --make Main +RTS -sstderr  6.36s user 
0.23s system 99% cpu 6.621 total

second (patched)
../../../compiler/ghc-inplace --make Main +RTS -sstderr  6.34s user 
0.24s system 99% cpu 6.588 total


I suspect that allocations don't always indicate slowness (maybe 
computation can happen without allocations...), although if allocations 
_don't_ change, that probably is an indication that it's exactly the 
same as before.... I could be imagining it, but it seems like according 
to 'time', the versions _with_ those patches have tended to be slightly 
faster (surprisingly?)  Repeating the same thing three times tends to 
give me quite consistent 'time' results. I don't know how much I should 
trust that, or how much I should be worrying about small performance 
changes... But if I should be worrying about small performance 
differences, I will make sure to add comments explaining why particular 
bits of code don't look so nice if they perform worse when I try to 
"improve" them ^_^

One odd (irrelevent) effect I've noticed is that I sometimes get 
somewhat different CPU percentages >100% (possible because of dual core 
and parallel make) - like, one day it's 135% and another day it's 127%, 
but it's the same for two different successive GHC builds! The real time 
goes up in proportion to the CPU% decreasing - the "user" time tends to 
be more consistent from day to day.

Isaac