HaBench
Contents |
[edit]
1 Haskell Benchmark Suite
Recent discussions have illustrated the need for an up-to-date Haskell benchmark suite. The goal would be to package a number of representative benchmarks, written in Haskell, which users can run on their machine. Gathering (and publishing) performance numbers for these benchmarks on a number of different machines will hopefully help in guiding future development of both Haskell compilers and programs.
[edit]
1.1 Contributors
- Kenneth Hoste - boegel on #haskell@FreeNode (ideas/advice/Wiki page maintainer)
- Donald Bruce Stewart - dons on #haskell@FreeNode (darcs repo maintainer (probably))
- Andy Georges - Itkovian on #haskell@FreeNode (measurements/analysis/advice)
- John Meacam - JohnMeacam on #haskell@FreeNode (advice/JHC support)
- Arthur van Leeuwen - earthy on #haskell@FreeNode (Computer Language Shootout man-on-the-insde)
- Neil Mitchell - ndm on #haskell@FreeNode (suggestions/Yhc support/user)
- Tom Davie - beelsebob on #haskell@FreeNode (suggestions/debugging support/user)
- Jeremy Shaw - stepcut on #haskell@FreeNode (suggestions)
- < Want to contribute? Add your name here! >
[edit]
1.2 Naming
* HaBench (intented to be the Haskell Benchmark Suite) * nofib/nofib 2.0 (follow-up of well-known nofib)
[edit]
1.3 Ideas
- start from the (out-of-date?) nofib benchmark suite (GHC-only, includes ghcisms, but very usefull programs (gzip in pure Haskell, a whole strictness analysis pass of a compiler, quantum mechanics simulator and other _real_ stuff (JohnMeacam)) (website)
- include Computer Language Shootout submissions (website)
- YHC regressions suite (URL)
- microbenchmarks for comparison with C (for example bytecopying) (dons)
- both a Haskell98 and Haskell'-only (sub)suite (JohnMeacam)
- multiple Haskell compilers (GHC, JHC, ...)
- maintain benchmark according to Hackage (Hackage list)
- Rename this project as nofib, otherwise it is like that people will have to run both this and nofib, because people will ask for the nofib results as well (ndm)
- Libraries and tools in Haskell
- Make sure code is pure Haskell as accepted by standard Haskell 98 (with addendums) or Haskell'
- For each program include a correct version along with several buggy versions (perhaps even bugs that the programmer introduced while writing the benchmark) so that the programs can be used to test debuggers
- For each program, include a clean, simple idiomatic version, and a hand-optimized fast version. This should hopefully give clues as to which programs have the most room for improvement, as well as what types of optimizations are helpful for a given program.
- Divide benchmarks into imaginary, spectral and real (just like nofib did). (SimonPJ)
[edit]
1.4 Notes
Random notes are below.
- (JohnMeacham) Insert non-formatted text here pretty much every compiler has a mode similar to 'ghc --make', so just let the user specify an arbitrary command line for the compiler to use, like 'jhc -v -flint $< -o $@' or 'ghc --make $< -o $@'
- (beelsebob) read Olaf's paper, he has evidence that speed problems in Haskell are usually because people are doing things too strictly
[edit]
1.5 More info?
- #haskell IRC channel at Freenode
- initial thread @ Haskell mailing list
- follow-up thread @ Haskell-Cafe mailing list
[edit]
1.6 Comments
Comments? Feel free to add them! Suggestions? Mail Kenneth at kenneth [dot] hoste [at] elis [dot] ugent [dot] be .