[Haskell-cafe] [14/16] SBM: Behind the measurements (rationale)

[Peter Firefly Brodersen Lund]

 // I am getting sick and tired of working on this project and it's probably
 // better to get it fired off than polishing it any further.
 //
 // This email could benefit from being rewritten from a rough draft into a
 // well-crafted letter but that would take a couple of hours.
 //
 // So here it is, a lot rougher than I'd like -- but it *IS* :)

why so big input files?
  easiest way to spot non-linearity and bad memory behaviour.  Anyway, files
  should be big enough to overflow caches and kick the gc in.
  (short files interesting, too, but the big ones cause more complex behaviour
  of run-time system and CPU.  If complex behaviour is behaved then simple
  behaviour probably is too -- but can still get its constant factors
  improved.  If complex behaviour bad, then shouldn't that be fixed in any
  case?)

waitpid4() has a struct w/ info about the child program's resource usage.
  unfortunately, the peakrss field is not filled in.  Seems to be a general
  Unix problem.  I've seen complaints on the net that Solaris doesn't fill it
  in, either.  Other solution needed.

pause-at-end, /proc/self/maps + /proc/self/status.  VmmHWM = peak of VmmRSS,
  which is Resident (working) Set Size.  It doesn't say what is shared with
  other processes or the operating system, though.  In our case, we don't
  expect to share anything but some libraries -- which nobody else wants to
  share with us anyway (except for the C library).  We are the only user of
  them.
  Discovered about a week ago that I could probably have used waitid()
  w/ WNOWAIT flag but didn't know.  Was quick to write pause-at-end, anyway.
  It took about 15 minutes from the desire to know the peak memory use to
  having written and tested the first cut of it.  Pause-at-end not completely
  bullet-proof in case of dyn libraries that get unloaded before the end of
  the program has been reached.  On the other hand, plenty good enough for
  these tests + can conceivably allow more intricate poking around than
  waitid() solution.

getting good measurements - eatmem, dd, probably should also dd library.
  good to have a "sacrificial run".  Good to measure how good the measurements
  are (rel. std.dev + user/sys/real check).
  why average -- disturbances are mostly interrupts, daemons that everybody
    have anyway, slightly luckier/unluckier physical pages.  These are real
    effects that nobody can control anyway.  I'm not interested in the best
    possible times on an ideal, undisturbed machine with a helpful kernel.
    I'm interested in clean times under realistic circumstances.  Therefore
    average instead of minimum.
  why I use real and not user/sys -- handling of blocking reads vs. mmap vs.
   madvise/fadvise vs. reading in separate thread in the future.  User+sys
   would probably give me better numbers at the moment and I could change to
   real later.  Still, I choose to stay with real (and the difference is marginal,
   anyway).
   
   Funny that the exact distribution of time between sys and user fluctuates a
   lot.  In space-bslc8-lenfil-2 sys varies between 0.160s and 0.244s.  Real is
   completely stable with 5x 1.396s and 1x 1.397s.

  look at /proc/interrupts, perhaps copy before/after to .intr?  Warn if more
  than 100 (or 1000) Hz + 10%?

write date/time + runlevel to platforminfo and/or sysinfo.

barcharts
  why barcharts.
  should the time/mem barcharts be equal length?  don't think so (hard to
    colour them in a text file.  Would work with less -r and the console but
    not in an email or a text editor.  Visual difference is good).
    But should perhaps not be /that/ different.
  visible markers if measurements bad.
   (5% real/user/sys check, typically within 0.1% on old laptop when doing a
   quick or thorough benchmark.  Occasionally up to 1% - and 3% on c/byte-4k
   because it only takes 56ms in total.)
  prints out how tight the user/sys/real thing is.

microarchitecture -- performance counters.  Would be interesting to look at
    once the obvious performance problems have been handled.  Let's fix the
    memory usage of bytestrings, the performance of lazy bytestrings, and 
    start using registers in the machine code first.
    regularity of input file probably means that branch predictor on all three
    CPUs can remember pattern of spaces vs. non-spaces (or at least part of the
    pattern).  Branch predictors not only use two-bit saturating counter for
    strongly non-taken/weakly non-taken/weakly taken/strongly taken.  They also
    try to remember the pattern of jumps/non-jumps.  A more realistic test
    would have less regular input file.  This effect is very small given the
    current performance limiters, though.

cache -- turned out to be pretty regular (by eyeballing cachegrind reports).
    Go up a factor of 10 in filesize and the number of access also went up a
    factor of 10.  The miss ratios stayed the same.  The miss ratios differed
    a bit between the benchmarks but I don't think it's time to look into that
    yet.  The data are available, though, for those who can't wait to look
    into that.

minor page faults
    we gather that through /usr/bin/time -- and could also get the same info
    from dumping the right file inside /proc/self/.  Probably not important yet.
    Probably will be once all the low-lying fruit has been gathered up from the
    ground.
    More of a factor on slower OS'es than Linux.


C files.  Buffer size.
    reading it all in one go is slower than (re)using a small buffer.  Cache
    effects, both in the operating system when copying (because the destination
    will be cached with a small buffer but non-cached with a big buffer) and in
    the application (everything will be cached with the small buffer, nothing
    with the big buffer).  Note that at least the Core and the Athlon64 have
    automatic prefetchers that tries to fill the cache in advance so we don't
    have to wait for the cache misses.  Doesn't quite seem to work.
    Older caches had a different write behaviour, they were write-through instead
    of the modern (lazy) write-back.  For those caches the writing to the user-space
    buffer should be slow even when a small buffer is reused all the time (because
    we would have to wait for all the writes to be flushed out to main memory).

C files.
    getchar/getchar_unlocked.  A comparison with getchar() is NOT what the
    simple haskell program does [grammar!].  Thread-safe by default (cause of
    libraries).  getchar_unlocked() is what the haskell programs do.  getchar() and
    getchar_unlocked() use a single buffer for stdin.
    getwchar() and getwchar_unlocked() included at the insistence of wli.  Much
    slower, because it is run-time dependent on locale (to choose encoding).
    Therefore, can't be a macro like getchar_unlocked() is.  With an indirect
    jump, should be same speed as getchar() on Core and Athlon64 -- but
    curiously isn't.

C and Haskell integer sizes and other limitations.
    Haskell uses unboxed 32-bit signed integers, except in lazy lenfil tests.

    Most of the C programs are simple and just use an int for the space count.

    One of them (space-megabuf)is more complicated.
    off_t is 64-bit.  ssize_t is 32-bit.  Potential overflow in c/space-megabuf.
    Potential 32-bit wrap-around in all my C tests.  Same problem with all
    Haskell tests, except for the two lenfil tests that use lazy bytestrings,
    because they use a 64-bit int for the length of the intermediate string of
    just the spaces filtered out from stdin.
    Also potentially out of memory.
    In practice, they have almost the same limit because they use about
    107MB for the 143MB input file.  In other words, it will run out
    of virtual address space or RAM or swap at about the same time that the
    others will run out of bits in a 32-bit signed integer.

-Peter