My current speed-testing validating technique

[Simon Peyton-Jones]

I can't comment about the build process (others will) but if you find performance regressions please do report them as precisely as possible when you have them nailed down, so we can reproduce them.

Thanks

S

| -----Original Message-----
| From: cvs-ghc-bounces at haskell.org [mailto:cvs-ghc-bounces at haskell.org]
| On Behalf Of Isaac Dupree
| Sent: 10 August 2007 12:46
| To: BuildBot Collator
| Subject: My current speed-testing validating technique
|
|
| The procedure I am about to describe, should make me notice serious
| performance regressions in the compiled stage1.  Also it tells me when
| my patches don't break validate any *more* than an unpatched HEAD
| (***which is true currently, that validate is slightly broken for me.
| Should I commit? Should I be hopeful that this is less likely to happen
| in future?***)
|
|
| So, this is my technique (using scripts I've made for steps (1) and
| (2),
| summarized at the end of this message):
|
| (1) update the local mirror of ghc
| (2) Check out two copies of the local repo, adding important files like
| mk/validate.mk
| (3) Pull the patches I want to test into the second copy. `time
| ./validate` on the first copy; when that's done, on the second copy.
| Compare times.
|
|
| Notes:
| I'm using zsh's built-in "time", which works for me.
|
| I make sure to have -O to compile stage1 (==> using mk/validate.mk),
| because optimized GHC is the one that really needs not to regress in
| performance, and -O for everything else too, in order to exercise that
| GHC well enough.  This tends to make validate take around 10% longer
| (when I tested, 17 minutes instead of 15 minutes, roughly, on my
| current
| machine).
|
| This doesn't work if my changes are significant enough to affect the
| compilation time of stage1 significantly, which I am assuming I'll
| know.
| (this may be a flawed assumption, when using -O ... more detailed
| timing
| in the validate script would be more helpful here)
|
| I need to be careful to run the tests when my system is fairly idle
| (which it usually is, and I can tell when it's not because then the fan
| runs audibly!). At some time I tested validate on the same repo twice
| in
| a row, and the amount of time was VERY consistent.  If I wanted to be
| even surer, I could set CPUS=0 so it only uses one thread and the rest
| of the system can use the other "core"...
|
| This also tells me whether the HEAD is broken for me even without my
| patches;
|   and does not rely on make distclean
|   (the latter could be a bad thing if I broke distclean, but if I
| thought I'd done something that would break it (including any adding or
| removing of files), I'd test that somehow!)
|
|
| (1)
| chmod +x darcs-all && ./darcs-all pull --all && \
|    ./darcs-all --complete get
| #"get" in case any core libraries have been added
|
| (2)
| darcs get (local-mirror-location) --repo-name="$NAME" && cd "$NAME" &&
| \
| chmod +x darcs-all; ./darcs-all --testsuite --complete get && \
| (copy my compile-settings to mk/build.mk and mk/validate.mk) && \
| chmod +x validate push-all
|
| (Also I put 'ALL no-set-default
| ' in _darcs/prefs/defaults, which seems to be buggy in that I don't
| also put that in the darcs-all repos...)
|
|
|
| Isaac
|
| _______________________________________________
| Cvs-ghc mailing list
| Cvs-ghc at haskell.org
| http://www.haskell.org/mailman/listinfo/cvs-ghc