nofib benchmarks for measuring the effects of compiler
marlowsd at gmail.com
Mon Oct 12 04:58:43 EDT 2009
On 09/10/2009 17:25, David Peixotto wrote:
> Thanks for the input. I'll take a look at the nobench suite. I might be
> interested in working to improve nobench. Is there some obvious
> improvements that people have in mind (besides using Criterion), or it
> just generally needs some updated benchmarks? I suppose one question to
> consider is whether it should include both parallel and sequential
> Simon M, can you say what your reliance on nobench-analyse is (output
> format, specific collected stats, etc.)?
Ok, here are the ways I use it:
- to compare the results from two nofib runs, on various metrics:
runtime, GC time, mutator time, allocations, binary sizes, sizes
of modules etc. Pretty much all the stats that nofib-analyse
compares are useful to us. I normally use the text output option,
rather that HTML.
- to generate LaTeX output for papers. This is a bit more ad-hoc,
typically I add more functionality as we need it.
- to generate CSV files for importing into Excel to generate graphs,
for papers and talks. Again, quite ad-hoc and funcionality added
The tool itself is a bit ugly and could do with a rewrite. I wouldn't
object to having to add e.g. LaTeX/CSV support myself to something else,
as long as the replacement is suitably clean and extensible.
> On Oct 9, 2009, at 6:13 AM, Simon Marlow wrote:
>> On 09/10/2009 09:43, Malcolm Wallace wrote:
>>>> Would you say that the nofib benchmarks are the best available for
>>>> measuring the effectiveness of compiler optimizations, or is there a
>>>> better benchmarking suite for that purpose?
>>> I don't know about "better", but the nobench suite (which is largely
>>> based on nofib, but goes further) automates the comparison of two or
>>> more compiler builds, and produces nice HTML tables of the results. It
>>> is currently languishing a bit - could do with some love and attention:
>> We do need a better benchmark suite. In GHC we still use nofib, but
>> it's becoming less useful as time goes by. I have some reliance on the
>> nofib-analyse tool that comes with nofib, which would need updating
>> (or replacing) to work with nobench.
>> Also we should be looking at using Criterion for our stats.
>> Working on nobench would be a really useful project for someone... any
>> Cvs-ghc mailing list
>> Cvs-ghc at haskell.org
More information about the Cvs-ghc